Important Note

Tufts ended funding for its Open Courseware initiative in 2014. We are now planning to retire this site on June 30, 2018. Content will be available for Tufts contributors after that date. If you have any questions about this please write to edtech@tufts.edu.

Tufts OpenCourseware
Search
Author: Ming Y Chow

 

Tufts OpenCourseWare
Introduction to Game Development
M. Chow
Spring 2012

 

Game Testing

 

Keywords

  • Quality Assurance (QA) - the professional term for testing
  • Bug - A software problem that exists in the implementation-level (i.e. only exist in source code)
  • Flaw - A software problem at a much deeper level, particularly in the design, and likely in the code level
  • System testing - Test the individual game components; internal to group
  • Play testing - Primarily done among development team, product and marketing managers
  • Alpha - In general, refers to the fact that everything has been implemented, but there are many bugs and gameplay/functionality issues; wide audience among senior personnel in company
  • Beta - In general, refers to the fact that everything has been implemented, and many critical gameplay/functionality issues and bugs have been resolved (but there are small bugs still lingering); even wider audience, nearly public

What I Learned in the Real World

  • Look, no software is perfect, and all software have problems
  • Why is software is not perfect, and why is software getting more difficult to make and manage?
    • Complexity
    • Extensibility
    • Connectivity
  • It would be nice to "find and fix all bugs," but it is largely wishful thinking (many constraints)
  • You don't want to find and fix all bugs, but you do want to find and fix the embarrassing and critical ones (before the CEO or senior management finds them)!
  • Testing often done very late in the integration process (right before rollout)
  • Unfortunately, testing can be a thankless and underappreciated job. The reasons behind the negativity:
  • You will most likely tick-off the engineers and programmers when you find bugs, and tell him/her to fix them
  • Because testing is done so late in the integration process, a lot of known and found bugs will not be resolved in time because the product needs to be shipped (i.e. lack of impact of testers). This is a very poor practice.
  • Many "agree" that testing should be tightly incorporated in the development phase (but few actually do it)
  • Personally, I feel that it is necessary to incorporate testing early in the development, if not, design phase. That is why I decided to talk about testing now rather than near the end of the course to stress the importance of testing.
  • Excellent oral communication - You will interact with the developers! You need to explain things clearly, and use good English.
  • Excellent written communication - Developers must to able to comprehend and recreate the problem
  • Tech and game-savvy - You must be able to play the game, and navigate through the product, largely on your own with minimal instructions
  • Endurance - You must be able to work on a project for a long time. You will likely have to work long hours, especially during crunch time (12 - 15 hours a day is not uncommon). Finally, you will have to do a lot of repetitive and mundane work: like playing an unfinished game over and over again, installing software, adding/removing hardware.
  • Good attitude - Being a game tester is a "foot-in-the-door" to the industry, but it will lead to bigger and better things if you are good and upbeat (e.g. promotion to a lead position very quickly, or move to be a developer)
  • Skills required for the job:
  • QA infrastructure: 1 QA personnel for approximately every 3 developer (many game companies have 15 - 20, or more, personnel on the QA team)

General Defect Tracking

  • Reporting your bugs
  • Many commercial defect tracking systems (databases)
  • Your report must be descriptive and concise for the developers to recreate and fix the problem (that is why your written communication skills are so critical)
  • The types of bugs that you report (severity):
    • Showstopper - These bugs must be fixed immediate or the skies will fall (e.g. glaring problems including audio glitches, game crashes, missing features, etc.). Consequences include: bad press, poor sales, people getting fired, and a black mark to the company's name.
    • Serious - Problems that gamers will likely notice, but does not take away from the gameplay experience (thus, can hold off)
    • Intermediate - Ditto
    • Low - Not so glaring problems that can be fixed in the next revision or patch
    • Suggestion - Changes or features that would be nice to have
  • Other pertinent fields in a bug report:
  • Reported on date
  • Report by (your name)
  • Assigned to
  • Feature
  • Description
  • Build / version
  • Status: {Open, Closed, Resolved, Not a bug, Cannot replicate}
  • Resolved on date
  • Resolution method
  • Assigned to (another developer)
  • Assigned to developer will take a look into the problem, and will add to the report:
  • After bug has been resolved, tester must verify the fix
  • It is possible that some bugs are actually not bugs
  • It is not uncommon to reopen a bug case

Regression Testing

  • If you have a plan to test a set of features, and changes and bugfixes are made, you have to go back and retest everything (the set of features) from the start
  • Important if you have a patch to be released for the game
  • Quite common that new bugfixes and changes introduce new bugs!
  • Very loopy process

Focus (Usability) Testing

  • Very different from QA testing
  • Not necessarily trying to find bugs
  • What you are testing:
    • Ease of use
    • Interface design
    • Can you perform a certain task
    • Fun
  • Can take place in the design phase or in the integration phase
  • In the design phase, use low-tech mockups to see how users would do a certain task, which would aide the design of the task in the actual game
  • In the integration phase, players actually play the game, and perform a list of tasks details on a test plan (checklist)
  • Usability testing should be conducted by someone who is not a developer (no knowledge of the game, thus no influence on testing)
  • Subjects should also have no knowledge about the game
  • Subjects must be carefully monitored, and all actions documented by evaluator/interviewer (i.e. note where the subject was confused, mouse and key presses)
  • Subjects are encouraged to think aloud
  • Very common to videotape the subjects
  • Evaluation

QA Involvement with Other Aspects of the Game Business

  • Usability testing to assist the design the game
  • Create videotapes of gameplay to company to decide whether to even publish the game
  • Make screenshots for the game box and advertisements
  • Presentation: verify and check the game requirements, packaging, manual before the game is shipped and advertised
  • Work directly with customer support

Best Practices

  • Integrate testing into the development process
  • Instead of testing everything at the end, test at certain milestones
  • Outsource QA:
    • Why: Make sense if QA is "seasonal;" less overhead and personnel to deal with, especially if you are a small game studio
    • Why not: Takes time to train new people; more expensive to take in specialized and seasoned QA personnel
    • Make sure that you have a qualified and seasoned pool of QA personnel ready for your use. One or several in-house QA leads is almost necessary.
  • Automation:
  • A lot of QA work is mundane
  • Write scripts to do repeated and mundane tasks
  • Very efficient, and saves a lot of man-hours
  • Define what you are testing
  • Provide a rubric of expectations to developers
  • You do not necessarily want to test everything; you do want to test the critical game features
  • Identify the classes of problems to be tested for a feature
  • Write test plans: