In the latest in a series of interviews with speakers from this August's GDC Europe, David Press, technical director at EVE Online
developer CCP Games, discusses the importance of automated testing in
MMOs, and why implementing such a system became a "now-or-never
decision" for the company.
Before joining CCP, Press worked at EA Chicago as a graphics programmer on Def Jam: ICON
and an unreleased fighting title. In 2007, he left for Atlanta, Georgia
to work at CCP as a graphics programmer, and then moved on to become a
systems programmer, and later a technical director.
In anticipation of his GDC Europe talk, "Orchestrator: A Postmortem on an Automated MMO Testing Framework,"
Press explains the various game elements an automated testing system
should or should not test, and how CCP implemented such a system for its
World of Darkness MMO.
What are some of the advantages of automated MMO testing?
Most MMOs are very large projects with numerous interacting game
systems, and it is simply infeasible to have manual testers constantly
testing every nook and cranny of the game. Automated tests allow
developers to get wide test coverage before they even check in their
changes and they can be run on a changelist-by-changelist basis so that
narrowing down bugs is trivial.
At what point did CCP realize that automated testing was necessary? What were the driving actors behind the decision?
For the World of Darkness project, we started unit testing
within the first year of active development, and system testing came in
during the next year. We knew that it is nigh impossible to bolt
automated testing onto code that wasn't written with it in mind, so it
was a now-or-never decision.
What sort of blind spots does automated testing tend to have? Are
there any specific things that are hard to test with an automated
system?
Subjective tests are obviously difficult to automate. Whether a
texture is rendering or a UI element is correctly aligned isn't worth
the effort to test. If you're familiar with the Model-View-Controller
pattern, you want to test the interaction between the controller and the
model; the view should be verified by your testers. For MMOs
specifically, there's definitely a line not worth crossing in terms of
how much testing should be automated. For instance, the emergent
behavior that we witness in our games when large groups of players
interact is often hard to predict and anticipate.
We also decided not to write a test for every single part of
our systems, but rather test the basic functionality of each system.
Especially during development, the details of systems change too often
to make it practical to get complete test coverage from your system
tests.
What is the biggest challenge developers face when writing an automated test for an MMO?
There are so many systems that need to be tested in an MMO, so a
framework that makes it easy to write these tests is a must or
programmers just won't have the time or motivation to do them. Also,
because MMOs are distributed systems, you need to be able to write tests
that can handle the vagaries of asynchronous communication between
multiple, independent processes in a robust, repeatable manner.
How will your GDC Europe talk address automated testing in MMOs, and what do you expect people to take away from it?
My talk will present a framework for writing and executing automated
system tests for the World of Darkness MMO. I will discuss the design
decisions that went into the framework and the rationale for them. I
will also talk about some of the challenges we still face with our
automated tests and how we try to mitigate them.
Additional Info
GDC Europe will take place August 15-17, 2011 at the Cologne
Congress-Centrum Ost, alongside the major gamescom trade show, and will
host lectures and panels with other notable speakers, including keynotes
from Ultima creator Richard Garriott and Wooga founder Jens Begemann, and more.
For more information on GDC Europe, please visit the official GDC Europe website, or subscribe to updates from the new GDC Europe-specific news page via RSS, Twitter, or Facebook. GDC Europe is owned and operated by UBM TechWeb.
