A Game Developer's Wish List for Researchers

"If we aren't just about to fail, but not failing, we could have made the game cooler."

Today I gave a short talk at the 2011 ACM Symposium on Interactive 3D Graphics and Games titled A Game Developer's Wish List for Researchers.

Here's the abstract:

Game developers have always turned to academic research for inspiration and to find approaches to technical problems. In the past, games were persona non grata in the research community, and so game developers reading research papers had to be content with merely consuming the available information without having much influence on the direction of research, or on its presentation. That has changed in recent years, with games growing in importance as a target and even a vehicle for academic research. Now that we have your ear, this lecture discusses a variety of ways researchers could make their work more useful to game developers. Some of these will be simple formatting suggestions, and some will be completely different ways of thinking about research topics, with various stops and detours in between.

The talk was part of the Industry Session at the conference, where two invited speakers, myself and Dan Baker from Firaxis, spoke about how academic research gets used (or doesn't get used, as the case may be) in the game industry.

My talk focused on a few different characteristics researchers need to think about if they want game developers to use their research.

First, I pointed out that contrary to the belief that performance is the single overriding factor in the evaluation of research by game developers, the most important factors are, in this order:

robustness
simplicity
performance

Then I went into each of these in detail, talking about the various ways in which these characteristics are important.

If you're doing art and entertainment, you want—as a rule—to be right up against the systemic complexity that breaks the camel's back. So, taking on a new piece of straw is dangerous and has to be clearly worth the risk.

I also talked about why researchers releasing source code for their algorithms would be a huge boon to the game development community in evaluating research, and how that's not because we're lazy and we don't want to write the code ourselves. I make a case that the source code is actually more rigorous than the paper in a lot of important ways.

I then talked a little bit about what kinds of things to research, although not about anything specific in this area; I mostly just provided some constraints to keep in mind for various areas like graphics, AI, and animation. I encouraged researchers to look into perceptual metrics and models more, because I see these as a way of formalizing and understanding some of the hacks we all do to get images on the screen. I think it's actually risky for researchers to look to game developers to suggest research topics, since we are often necessarily short-sighted.^[1] I think it may be better for researchers to actually understand and play games, and then come up with their own topics, although that's not a hard and fast rule, obviously.

Finally, I talk about a laundry list of small topics, including:

Researchers should not patent their work if they want game developers to use it, and authors should be required to disclose any IP encumbrances at the top of the paper so readers can avoid tainting themselves.
I rail against the plethora of academic paywalls, and urge researchers to put both their new papers and their older papers on their websites. I will write more on this topic in the future, believe you me.
I discuss the importance of reporting negative results, and tell researchers to follow in the footsteps of medical researchers, who recently launched the Journal of Negative Results in BioMedicine, saying if existing publishing channels won't publish them, then put them on your websites.
I talk about how the peer review process for our SIGGRAPH paper forced me to cut a lot of the negative results and caveats, and how this shows the existing peer review system is broken.
One topic I touched on, and then came up again during Q&A, and then again at lunch was the difficulty of attaining real data from games. I told people the problem they're running into is asking for the developers to provide the data to them, rather than simply using the millions of PC game modding tools to rip the data themselves, and then simply asking for permission to use it in their presentations. Game developers don't have time to package the data up, but I think most would be happy for researchers to use their levels as "real-world" datasets. As usual, Valve is way ahead of everybody here, and occasionally provides L4D maps to researchers (including two of the three shadow talks right after our session!). EA provided the Sims 2 assets to CMU's Alice program, as well. So, it is possible to get real data from games, and I hope more researchers do this.

And more. Here is the synced slideshow. I start by commenting on a question from the end of the previous session, so it sounds like it's starting in the wrong place, but it's not.

Welp, the web service I was using for the slide syncing died, so I'll have to redo it on youtube I guess.

And here's the ppt and mp3 if you can't view the fancy flash presentation above.

This talk is a sibling to My AIIDE 2010 Lecture on Game AI.

Acknowledgements

I'd like to thank Ocean Quigley, Jeff Roberts, Jay Stelly, Sean Barrett, John Carmack, Jonathan Blow, and Charles Bloom for helping me brainstorm for this talk (several of the ideas in the talk are stolen directly from emails they sent me), and also these fine people on Twitter, who kindly responded to my request for ideas:

↑ I'm reminded of the time when even John Carmack was telling 3D hardware manufacturers to not work on Z-buffers because they were too expensive. Luckly, Gary Tarolli ignored him. Also, memory prices dropped through the floor. That helped too.

[1] I'm reminded of the time when even John Carmack was telling 3D hardware manufacturers to not work on Z-buffers because they were too expensive. Luckly, Gary Tarolli ignored him. Also, memory prices dropped through the floor. That helped too.

[1]