ideas for the showstopper

[monsto]

i'm a web developer and have had intermittent javascript etc showstoppers on sites i've built.

what i did may help you guys. i had a friend build me a program that surfed my site like a human. all it did was randomly move the mouse around the screen in a contiguous fashion, move to a random link, click it, and the program logged every action. the key part of this program was that if the browser stopped responding, then it quit the program, marked position of the failure in the log and started it again. then we set it to sail and walked away.

a relatively simple .NET app that ran well. after a weekend where it generated some 400mb of logs, i had enough crashes with enough info that i was able to find the problem and fix it.

i realize that such a bot for exult wouldn't be as simple. in fact, in order to effectively use the game in a manner that would be 'humanlike' you'd need to have a pretty complex expert system that's able to path the player etc. but i'm not sure that's necessary. you may be able to have it do enough things via automation that the number of iterations that the bot is able to produce would give you much needed data. even if it .1% of the sessions return usable data, if you're capable of throwing 100k sessions at it in the space of a couple weeks, well then that's 100 reliable bug reports.

second thing that i wanted to suggest is to try a different compiler. i've worked on mods to retail games where GCC caused items to 'stick' in the game world. you couldn't pick them up but there they were. we went round and round for months trying to figure it out. Changing compilers, without changing a line of code, fixed it.

that's it. hope i've helped.

[Kensu]

What do game development studios do by way of automated testing? Do they just test the engine itself, and none of the game; or do they even test the engine?

[monsto]

the game companies i worked for (after i did modding), besides unit tests, did "complete" testing of the entire product in a what was expected to be a finished state. the testers did this super tedius running around in the game and pretty much just took notes. i was never involved n the testing phase, but i suppose that the logic was that any engine 'bug' would manifest in a way that gameplay testers could detect as they were playing the @!#$ out of it.