A Game Developer's Wish List for Researchers

Revision as of 02:07, 20 February 2011 by Checker (talk | contribs)

"If we aren't just about to fail, but not failing, we could have made the game cooler."

Today I gave a short talk at the 2011 ACM Symposium on Interactive 3D Graphics and Games entitled A Game Developer's Wish List for Researchers.

Here's the abstract:

Titleslide.jpg

Game developers have always turned to academic research for inspiration and to find approaches to technical problems. In the past, games were persona non grata in the research community, and so game developers reading research papers had to be content with merely consuming the available information without having much influence on the direction of research, or on its presentation. That has changed in recent years, with games growing in importance as a target and even a vehicle for academic research. Now that we have your ear, this lecture discusses a variety of ways researchers could make their work more useful to game developers. Some of these will be simple formatting suggestions, and some will be completely different ways of thinking about research topics, with various stops and detours in between.

The talk was part of the Industry Session at the conference, where two invited speakers, myself and Dan Baker from Firaxis, spoke about how academic research gets used (or doesn't get used, as the case may be) in the game industry.

My talk focused on a few different characteristics researchers need to think about if they want game developers to use their research.

First, I pointed out that contrary to the belief that performance is the single overriding factor in the evaluation of research by game developers, the most important factors are, in this order:

  1. robustness
  2. simplicity
  3. performance

Then I went into each of these in detail, talking about the various ways in which these characteristics are important.

Simplicityslide.jpg
If you're doing art and entertainment, you want to—as a rule—be right up against the systemic complexity that breaks the camel's back. So, taking on a new piece of straw is dangerous and has to be clearly worth the risk.

I also talked about why researchers releasing source code for their algorithms would be a huge boon to the game development community in evaluating research, and how that's not because we're lazy and we don't want to write the code ourselves. I make a case that the source code is actually more rigorous than the paper in a lot of important ways.

I then talked a little bit about what kinds of things to research, although not about anything specific in this area; I mostly just provided some constraints to keep in mind for various areas like graphics, AI, and animation. I encouraged researchers to look into perceptual models and metrics more, because I see these as a way of formalizing and understanding some of the hacks we all do to get images on the screen.

Finally, I talk about a laundry list of small topics, including:

  • Researchers should not patent their work if they want game developers to use it, and authors should be required to disclose any IP encumbrances at the top of the paper so readers can avoid tainting themselves.
  • I rail against the plethora of academic paywalls, and urge researchers to put both their new papers and their older papers on their websites. I will write more on this topic in the future, believe you me.
  • I discuss the importance of reporting negative results, and tell researchers to follow in the footsteps of medical researchers, who recently launched the Journal of Negative Results in BioMedicine, saying if existing publishing channels won't publish them, then put them on your websites.
  • I talk about how the peer review process for our SIGGRAPH paper forced me to cut a lot of the negative results and caveats, and how this shows the existing peer review system is broken.
  • One topic I touched on, and then came up again during Q&A, and then again at lunch was the difficulty of attaining real data from games. I told people the problem they're running into is asking for the developers to provide the data to them, rather than simply using the millions of PC game modding tools to rip the data themselves, and then simply asking for permission to use it in their presentations. Game developers don't have time to package the data up, but I think most would be happy for researchers to use their levels as "real-world" datasets. As usual, Valve is way ahead of everybody here, and occasionally provides L4D maps to researchers (including two of the three shadow talks right after our session!). EA provided the Sims 2 assets to CMU's Alice program, as well. So, it is possible to get real data from games, and I hope more researchers do this.

And more. Here is the synced slideshow. I start by commenting on a question from the end of the previous session, so it sounds like it's starting in the wrong place, but it's not.

Welp, the web service I was using for the slide syncing died, so I'll have to redo it on youtube I guess.

And here's the ppt and mp3 if you can't view the fancy flash presentation above.

Acknowledgements

I'd like to thank Jeff Roberts, Jay Stelly, Sean Barrett, John Carmack, Jonathan Blow, and Charles Bloom for helping me brainstorm for this talk (several of the ideas in the talk are stolen directly from emails they sent me), and also these fine people on Twitter, who kindly responded to my request for ideas:

This page was last edited on 21 September 2021, at 20:12.