Tufts OpenCourseware
Search
Author: Ming Y Chow

 

Tufts OpenCourseWare
Introduction to Game Development
M. Chow
Spring 2012

 

Game Testing

 

Keywords

  • Quality Assurance (QA) - the professional term for testing
  • Bug - A software problem that exists in the implementation-level (i.e. only exist in source code)
  • Flaw - A software problem at a much deeper level, particularly in the design, and likely in the code level
  • System testing - Test the individual game components; internal to group
  • Play testing - Primarily done among development team, product and marketing managers
  • Alpha - In general, refers to the fact that everything has been implemented, but there are many bugs and gameplay/functionality issues; wide audience among senior personnel in company
  • Beta - In general, refers to the fact that everything has been implemented, and many critical gameplay/functionality issues and bugs have been resolved (but there are small bugs still lingering); even wider audience, nearly public

What I Learned in the Real World

  • Look, no software is perfect, and all software have problems
  • Why is software is not perfect, and why is software getting more difficult to make and manage?
    • Complexity
    • Extensibility
    • Connectivity
  • It would be nice to "find and fix all bugs," but it is largely wishful thinking (many constraints)
  • You don't want to find and fix all bugs, but you do want to find and fix the embarrassing and critical ones (before the CEO or senior management finds them)!
  • Testing often done very late in the integration process (right before rollout)
  • Unfortunately, testing can be a thankless and underappreciated job. The reasons behind the negativity:
  • You will most likely tick-off the engineers and programmers when you find bugs, and tell him/her to fix them
  • Because testing is done so late in the integration process, a lot of known and found bugs will not be resolved in time because the product needs to be shipped (i.e. lack of impact of testers). This is a very poor practice.
  • Many "agree" that testing should be tightly incorporated in the development phase (but few actually do it)
  • Personally, I feel that it is necessary to incorporate testing early in the development, if not, design phase. That is why I decided to talk about testing now rather than near the end of the course to stress the importance of testing.
  • Excellent oral communication - You will interact with the developers! You need to explain things clearly, and use good English.
  • Excellent written communication - Developers must to able to comprehend and recreate the problem
  • Tech and game-savvy - You must be able to play the game, and navigate through the product, largely on your own with minimal instructions
  • Endurance - You must be able to work on a project for a long time. You will likely have to work long hours, especially during crunch time (12 - 15 hours a day is not uncommon). Finally, you will have to do a lot of repetitive and mundane work: like playing an unfinished game over and over again, installing software, adding/removing hardware.
  • Good attitude - Being a game tester is a "foot-in-the-door" to the industry, but it will lead to bigger and better things if you are good and upbeat (e.g. promotion to a lead position very quickly, or move to be a developer)
  • Skills required for the job:
  • QA infrastructure: 1 QA personnel for approximately every 3 developer (many game companies have 15 - 20, or more, personnel on the QA team)

General Defect Tracking

  • Reporting your bugs
  • Many commercial defect tracking systems (databases)
  • Your report must be descriptive and concise for the developers to recreate and fix the problem (that is why your written communication skills are so critical)
  • The types of bugs that you report (severity):
    • Showstopper - These bugs must be fixed immediate or the skies will fall (e.g. glaring problems including audio glitches, game crashes, missing features, etc.). Consequences include: bad press, poor sales, people getting fired, and a black mark to the company's name.
    • Serious - Problems that gamers will likely notice, but does not take away from the gameplay experience (thus, can hold off)
    • Intermediate - Ditto
    • Low - Not so glaring problems that can be fixed in the next revision or patch
    • Suggestion - Changes or features that would be nice to have
  • Other pertinent fields in a bug report:
  • Reported on date
  • Report by (your name)
  • Assigned to
  • Feature
  • Description
  • Build / version
  • Status: {Open, Closed, Resolved, Not a bug, Cannot replicate}
  • Resolved on date
  • Resolution method
  • Assigned to (another developer)
  • Assigned to developer will take a look into the problem, and will add to the report:
  • After bug has been resolved, tester must verify the fix
  • It is possible that some bugs are actually not bugs
  • It is not uncommon to reopen a bug case

Regression Testing

  • If you have a plan to test a set of features, and changes and bugfixes are made, you have to go back and retest everything (the set of features) from the start
  • Important if you have a patch to be released for the game
  • Quite common that new bugfixes and changes introduce new bugs!
  • Very loopy process

Focus (Usability) Testing

  • Very different from QA testing
  • Not necessarily trying to find bugs
  • What you are testing:
    • Ease of use
    • Interface design
    • Can you perform a certain task
    • Fun
  • Can take place in the design phase or in the integration phase
  • In the design phase, use low-tech mockups to see how users would do a certain task, which would aide the design of the task in the actual game
  • In the integration phase, players actually play the game, and perform a list of tasks details on a test plan (checklist)
  • Usability testing should be conducted by someone who is not a developer (no knowledge of the game, thus no influence on testing)
  • Subjects should also have no knowledge about the game
  • Subjects must be carefully monitored, and all actions documented by evaluator/interviewer (i.e. note where the subject was confused, mouse and key presses)
  • Subjects are encouraged to think aloud
  • Very common to videotape the subjects
  • Evaluation

QA Involvement with Other Aspects of the Game Business

  • Usability testing to assist the design the game
  • Create videotapes of gameplay to company to decide whether to even publish the game
  • Make screenshots for the game box and advertisements
  • Presentation: verify and check the game requirements, packaging, manual before the game is shipped and advertised
  • Work directly with customer support

Best Practices

  • Integrate testing into the development process
  • Instead of testing everything at the end, test at certain milestones
  • Outsource QA:
    • Why: Make sense if QA is "seasonal;" less overhead and personnel to deal with, especially if you are a small game studio
    • Why not: Takes time to train new people; more expensive to take in specialized and seasoned QA personnel
    • Make sure that you have a qualified and seasoned pool of QA personnel ready for your use. One or several in-house QA leads is almost necessary.
  • Automation:
  • A lot of QA work is mundane
  • Write scripts to do repeated and mundane tasks
  • Very efficient, and saves a lot of man-hours
  • Define what you are testing
  • Provide a rubric of expectations to developers
  • You do not necessarily want to test everything; you do want to test the critical game features
  • Identify the classes of problems to be tested for a feature
  • Write test plans: