It might seem a little obvious, but latency issues make a huge difference in a virtual world, especially one that you're projecting just a few millimetres from a gamer's eye. Over on Valve's official blog, Michael Abrash goes into a lot of detail around latency issues as they affected legacy hardware back in the VGA days, and how they affect virtual worlds. He makes the very sensible point that not all programming breakthroughs come just because you slave away at them — his legacy approach occurred to him while decorating a Christmas tree — but also that the latency needed for proper VR is much more finicky than regular PC gaming.
How much latency is too much? Less than you might think. For reference, games generally have latency from mouse movement to screen update of 50 ms or higher (sometimes much higher), although I’ve seen numbers as low as about 30 ms for graphically simple games running with tearing (that is, with vsync off). In contrast, I can tell you from personal experience that more than 20 ms is too much for VR and especially AR, but research indicates that 15 ms might be the threshold, or even 7 ms.