Beta testing. How?

[InCreator]

I wanted to figure out how beta testing traditionally works with AGS games.
How is it implemented, what is the time frame and form of communication?

What I imagine is create a thread, people PM with their wish to test, they get links to game, they have a time frame of some sort, and they PM or e-mail back with found bugs.

Is this how things should work or is there a better way to handle the testing?
And how much time does it usually need?
Experiences, anyone?

[voh]

I would probably opt for a mailing list like a yahoo/google group, so that if someone reports a bug, everyone on the beta testing team gets it and can check it out. This way you get an organized system automatically (if you've got the right people).

We playtested a couple of roleplaying games (pen and paper) and did it this way. Got a lot of insights from other people's experiences through the mailing list.

[Radiant]

"Traditionally"? I don't think there is any such thing

In my experience:
* Most people are poor beta testers. A poor beta tester will cost you effort and will not give much of a good result; poor beta testers are a net negative to your project, as well as a time sink.
* Some people are enthousiastic about playing your game (earlier than the rest of the world). These fall in the category above.
* If people PM their "wish to test", you must therefore judge their "ability to test".
* You don't need a lot of testers. You should test a bit yourself, and having one or two testers aside from you is sufficient (in comparison, most of the beta testing for ATOTK was done by two people)
* Communication is fine by e-mail back and forth. Alternatively, you can make a private thread for these people.
* Pick at least a month, two if it's a big game.
* Update frequently, so that people don't repeatedly fall into the same bugs, and can verify that things are fixed (and no new things are broken).
* Put a decent debug mode in your game. The standard "teleport + stuff pockets" is nice, add whatever else you can code for it.
* It is a good idea to not tell the testers the solutions (to puzzles) beforehand. That way, you can find out which ones are too easy or too hard.
* Golden rule: PAY ATTENTION to everything they say! I know way too many authors who get defensive towards bug reports, and discount them because "it was not intended that way". This is a POOR excuse, and will reduce the end quality of your game.

[Ghost]

I only did one beta test but would like to show how I did it as an example:
I was send the game, with a walkthrough which I ignored. I played the game. I checked for the following:
* Graphical glitches (like areas where walkbehinds or walkable ares didn't work- things overlapping, frames with artifacts, the standard
* GUI glitches (unnecessary spaces, missing text)
* Puzzle logic.

Then after playing the whole game once, I played it again in "perfect order" to check for story logic, puzzle dead ends etc.

Then, I played it again to check for grammar.

I don't know if this is typical behaviour, but the end result was quite good. I think it is a rather technical approach, but worked nice.

On the whole I agree with Radiant that testers who just wish to get a first hand impression are maybe too interested and involved, though calling them bad testers is a bit too strong in my opinion. They are still good for giving feedback.

[Akatosh]

Also, it might be good to have a beta-tester with a bit of experience in AGS-scripting... they'll most likely know what to look after (at least for "typical" errors) because they've probably made and fixed them themselves already.

[TheJBurger]

* Put a decent debug mode in your game. The standard "teleport + stuff pockets" is nice, add whatever else you can code for it.

Agreed. Use the other CTRL + Letter to trigger certain events and cut back on the waiting time to retest certain scenes.

I really don't know anything about beta-testing, but from the one game I did release, here's how it went.

- Alpha Testing -
After my game was playable from start to finish I contacted someone for music (I wouldn't recommend doing this that late, however my game was short and I thought the music could be completed quickly enough), and during the production of the music I alpha-tested my game probably 10-20 times.

Each time I tested it, I would keep a REAL-LIFE pen and paper next to me to write down every bug, or every little graphical glitch I thought could be improved in anyway. If I didn't write it down, I could only keep three things in my short-term memory before forgetting them. Writing down was a MUST.

After I went through an alpha-testing cycle, I would go back and check off everything on my list I thought could be improved. I re-did this over and over, each time surprisingly finding more and more things and adding to my list. Only once I could play through the game and not notice any small little deficiencies did I start beta-testing.

- Beta Testing -
During beta-testing, I wanted feedback on everything really, not just errors. I first had my two musicians beta-test the game and they gave me some pretty critical feedback on basic things in the game, which I couldn't believe I had overlooked. PMing was a sufficient enough way to communicate.

After that I had two more beta-testers come in and play the updated version. Once again, more bugs were found and more awesome suggestions were made. I re-evaluated the game and fixed it. The testers played through once more and couldn't find any bugs, so, I released it.

Did it work out okay? Yes, I think so. The game would've lacked several features that I never would've thought of on my own, so I owe the beta-testers a huge credit for that.

[paolo]

Beta-testing adventure games is hard because they are event-driven, that is, when something (an event) happens in the game, either because the player made it happen or the game did, something else happens. The tricky bit is that these events can often happen in any of very, very many orders.

So, ideally, beta-testing needs to ensure not only that it has 100% coverage (testing everything that can possibly happen or be done in the game) but also that all possible sequences of events are tested. For example, consider this scenario:

NPC1 says to the PC: "Bring me items X and Y and I'll give you item Z". Collecting each of these items is a subquest, and they can be collected in either order.

The testers will need to test the following:

* Collect item X
* Collect item Y
* Give item X to NPC1
* Give item Y to NPC1
* Ensure that PC receives item Z from NPC1

There are at least four permutations here:

1: Collect item X; collect item Y; give item X to NPC1; give item Y to NPC1
2: Collect item X; collect item Y; give item Y to NPC1; give item X to NPC1
3: Collect item Y; collect item X; give item X to NPC1; give item Y to NPC1
4: Collect item Y; collect item X; give item Y to NPC1; give item X to NPC1

Other things to test here include:

* What happens if you give one item to NPC1 and then speak to them again - does NPC1 still ask for both items (which is a bug) or just the missing item (which is correct)?
* What happens if you give both items to NPC1 and then speak to them again - does NPC1 still ask for an item/the items (which is a bug) or do something like tell the PC that the quest is complete (which is correct)?

The number of permutations can be quite large even in small games, and in very large games, like A Tale of Two Kingdoms, where there are many paths through the game as well as numerous orders in which items can be collected and used, the number of possible playthroughs is immense, and this sort of exhaustive testing is not possible.

Much the same applies to beta-testing of other software, where processes A and B may work fine by themselves, but reveal a bug if B is done before A, for example.

So what approaches can be taken to ensuring that as many as possible of the various paths through the game are tested?

EDIT: For Trance-Pacific, the beta-testers found some bugs when going through the game using the walkthrough, and another bug or two when playing the game without it and doing things that other players would presumably try as well. Attempting exhaustive testing was out of the question because it was a MAGS game and the deadline was only two days away!