I've been giving this some thought and I'm going to say the wipeout series of games but only at the faster speed levels. I've similar thoughts with the ridge racer series but that's slightly more forgiving although can be frustrating in the endgame as well. Love both games though.
No ones mentioned the GB toughies so I'll go there. Solar Striker, Fortress of Fear, The hunt for red October....so hard they made me cry... And Gargoyle's Quest...not the toughest game in the world but you could get to a certain point and then no further!!! It was more of a tough lesson for later life