The Art and Engineering of VR Game Testing
/We have been working hard on a VR game for the super cool people out at Red Oak Studio (check out the preview of the game on Steam ). The game is built with Unity and C#, and all the testing has been purely manual testing. VR, as you know, is a whole new ball game be it Ux, development or testing. And this has been a huge learning curve for us all.
I want to share some of the things I've learnt about VR testing when running the QA/tests on this project today.
0. Be a Gamer and VR proficient first :
I consider this as a pre-requisite before professionally jumping into VR. You must have prior experience and passion for gaming. You also need to possess sound knowledge of the genre you are targeting to make the game for. For instance, if you want to make an action game, you need to know the common aspects of this genre. Don't you expect to open a documentation, start testing and producing great output. It simply won't work no matter what superman tester you are.
Also, try to to feel and adapt the VR world in heart by fiddling with it for at least one month. Make it natural for you. Otherwise, you will miss lots of normal aspects. You can't expect to start making a good game the first day of laying your hand on it.
1. Don't mix "Usability" with "Challenge" and "Challenge" with "Frustration" :
In software testing world, usability/user experience (UX) is a vital aspect. Simply speaking, it's making sure the users can perform everything conveniently and with as less difficulties as possible. But it's not always applicable in gaming world. In games, a difficulty often leads to "Challenge". For instance, you might need to scratch your head for long to find a weapon or find your way to next step. It's a key element of a game that makes it interesting to the gamer.
But some times, too much difficulty leads to "Frustration". For instance, getting dead over and over trying to clear a wave of enemies, could make the gamer bored and ultimately lose interest for it. Also, in training/on-boarding level a new player expect it to be prompt and easy. While making a game, you need to keep a balance among these "Usability", "Challenge" and "Frustration" factors.
2. Transition scenes:
I found scene transits to be the most vulnerable areas for generating bugs. Examples are: level changes, loading game from one part to another, changing game environment, starting game, ending game, reincarnating, loading from checkpoint etc. Things often don't quite work as expected in these areas. You have to spend extra time refining scene transits.
3. Beware of Chaperone bounds :
This was a new thing to me. Do test your game by putting the controllers near or out of the "Chaperone bounds" (the greenish grid that appear whenever you get close to the bounds of playable area). It's very important if you have a small play area in a congested room. we encountered countless issues by it. Every thing started to act crazily for no apparent reasons. We had to take extra cautions dealing with this aspect.
4. Game pause menu :
In-game pause menu is another area that requires extra care. First, pause has to be instantaneous. Second, no work can perform in background. Third, all things should resume properly whenever player un-pauses it. Pause seems a natural thing as a gamer but it's a huge pain for devs and gold mine for testers. We spend half of our time solidifying pause in our game. If your game has options to perform extra actions from within the pause menu e.g. loading checkpoint, restarting etc. then it adds to more trouble. So, keep an extra eye for it.
5. System's pause :
Along with in-game's pause, keep in mind of the system's pause as well. You get it by pressing the 'system' button at the bottom part of your controller. It will take you to Viveport's dashboard usually. The common issue doing so is, game will continue to run on background if you don't handle it manually. So whenever 'system' button is pressed, make sure the game is properly paused in background as well.
6. Check for Controller and Headset's availability:
This is an often overlooked but important aspect. Your game needs to validate the controllers are on and within range of base stations while starting. Otherwise, lots of unwanted issues will generate. Prompt the user visually to turn the controllers on and assert enough wait time to properly get them detected. Deal with it while the game is running as well because one controller might turn off due to low power. Test it rigorously turning on/off single and both controllers during different times of game and note the affect.
Along with controllers, make sure the headset is available at the beginning of the game. It also validates the PC is VR ready and your game can run on it.
7. Beta test with different classes of gamers :
I can't stress for this enough. No matter how foolproof and cozy the game you think is for you, you'll always find issues and surprising new angles by testing with other gamers. Never ever release your game without having tested by several gamers. Their experience and feedback will prove invaluable. It's also important to incorporate several classes of gamers based on expertise levels. We grouped three classes of game testers -
(1) Those who have very limited gaming experience in other platforms, let alone VR
(2) Veteran gamers in other platforms (PC/Console) but have limited to no experience in VR
(3) Proficient gamers in both regular platforms and VR world
You have to take into account every group's reaction. Remember, your game will be played by wide range of audiences so you have to make sure it's convenient for everybody, not just pro-gamers.
8. Make the game equally comfortable for people with low eye power :
Not every gamer will have 6/6 vision and you need to adopt your game thereby. It's far more important than it sounds. We learnt it the hard way from one our beta tester's experience. Believe it or not, we had to completely rewrite one part of our game just for it. The reason is, vive is not that cozy for people who wear glasses. Most of them wear the headset without glasses. They will often find some game objects blurry. In our game, we had a great sniping gameplay with long distance shooting. But thanks to vive's mediocre resolution, glass-people found them hard to detect, let alone shoot. We had to change lots of game elements keeping in mind of low-visioned gamers. You better to.
9. Test compiled build version, don't just rely on Unity :
It's another new lesson we learnt the hard way. We used to test our game by running directly from Unity. Everything went well but when we produced and tested build version (.exe runner), lots of anomalies start to appear to our surprise. Sometimes unity loses reference of game assets in compiled version for no apparent reason. Sometimes, the developer forget to take care of debug elements. Many of such issues you might encounter in build version. So double check everything is working as expected in build version before publishing your game.
10. Avoid 'Nod' gesture inputs if possible :
So, you find the nod gesture as input for affirmation (like in "InMind VR") so cool and possibly thinking to use it in your game, right? Wrong!!! We also initially thought the same and had some nod gesture as affirmation input in our game at first. Turned out it's causing more pains than attraction. First, it doesn't work as smooth as it should. Even veteran VR gamer like me had to try multiple times to make it work every time. Second, VR newbies find it very difficult and get frustrated easily. Ultimately we had to ditch the whole nod thing and rely on normal controller input. I recommend you to get rid of head tracking input for your game as well unless it's absolutely necessary or you are damn sure it works perfectly.
Wohooo, that's a pretty long article (at least for me), and if you've made it up to here you deserve a treat! Here is the teaser video for the game and some game footage from the preview version.
