Metrics Fetishism - Revision history

Checker at 21:41, 24 August 2010

2010-08-24T21:41:55Z

← Older revision		Revision as of 21:41, 24 August 2010
Line 5:		Line 5:
	In the old days, people designed games by their instinct and '''intuition'''. Notice I didn't say the "good old days", it was just how things were done. If you wanted feedback, you'd playtest the game, and watch the players struggle with the controls, or waltz through some section you thought was hard, or whatever. Then you'd iterate.		In the old days, people designed games by their instinct and '''intuition'''. Notice I didn't say the "good old days", it was just how things were done. If you wanted feedback, you'd playtest the game, and watch the players struggle with the controls, or waltz through some section you thought was hard, or whatever. Then you'd iterate.

	~~Lately~~, as computers got networked and more powerful, we started recording '''metrics''' about how players played our games, and feeding that back into the design process. We dump a lot of information to a database, and then we mine that information for clues about how to optimize certain aspects of our games, whether it's "fun" or "ARPU" or whatever. This happens during development, and now that a lot of games are online, it happens after we ship, with the design being constantly iterated based on metrics gathered live after launch.		More recently, as computers got networked and more powerful, we started recording '''metrics''' about how players played our games, and feeding that back into the design process. We dump a lot of information to a database, and then we mine that information for clues about how to optimize certain aspects of our games, whether it's "fun" or "ARPU" or whatever. This happens during development, and now that a lot of games are online, it happens after we ship, with the design being constantly iterated based on metrics gathered live after launch.

	Now, metrics and closed feedback loops are great in the abstract, but like any tool, they can be misused. Just as it's usually a bad idea to design a game based totally on your intuition without grounding that intuition in the cold hard reality of another human being touching the controls, it's also bad to blindly gather and follow metrics data.		Now, metrics and closed feedback loops are great in the abstract, but like any tool, they can be misused. Just as it's usually a bad idea to design a game based totally on your intuition without grounding that intuition in the cold hard reality of another human being touching the controls, it's also bad to blindly gather and follow metrics data.

Checker at 05:52, 4 August 2010

2010-08-04T05:52:49Z

← Older revision		Revision as of 05:52, 4 August 2010
Line 15:		Line 15:
	But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed. The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space. If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.		But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed. The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space. If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.

	'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. ~~'''The metrics are the craft, and the intuition is the art.'''~~		'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that.

	<center>[[Image:Local_maximum.png]]</center>		<center>
			'''The metrics are the craft, and the intuition is the art.'''
			<br/>
			[[Image:Local_maximum.png]]</center>
	<br/>		<br/>

	<references/>		<references/>

	[[Category:Lectures]]		[[Category:Lectures]]
	[[Category:Opinions]]		[[Category:Opinions]]

Checker at 05:41, 4 August 2010

2010-08-04T05:41:36Z

← Older revision		Revision as of 05:41, 4 August 2010
Line 15:		Line 15:
	But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed. The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space. If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.		But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed. The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space. If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.

	'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. The metrics are the craft, and the intuition is the art.		'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. '''The metrics are the craft, and the intuition is the art.'''

	<center>[[Image:Local_maximum.png]]</center>		<center>[[Image:Local_maximum.png]]</center>

Checker: major edit

2010-08-04T05:37:10Z

major edit

← Older revision		Revision as of 05:37, 4 August 2010
Line 16:		Line 16:

	'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. The metrics are the craft, and the intuition is the art.		'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. The metrics are the craft, and the intuition is the art.

			<center>[[Image:Local_maximum.png]]</center>
			<br/>

	<references/>		<references/>

Checker: Created page with "In my recent lecture, ''Achievements Considered Harmful?'', I ranted briefly about what I call ''metrics fetishism'', and how's it's taking over game design and how we should…"

2010-08-04T05:26:27Z

Created page with "In my recent lecture, ''Achievements Considered Harmful?'', I ranted briefly about what I call ''metrics fetishism'', and how's it's taking over game design and how we should…"

New page

In my recent lecture, ''[[Achievements Considered Harmful?]]'', I ranted briefly about what I call ''metrics fetishism'', and how's it's taking over game design and how we should be careful.

<center>[[Image:foothills.jpg]]</center>

In the old days, people designed games by their instinct and '''intuition'''. Notice I didn't say the "good old days", it was just how things were done. If you wanted feedback, you'd playtest the game, and watch the players struggle with the controls, or waltz through some section you thought was hard, or whatever. Then you'd iterate.

Lately, as computers got networked and more powerful, we started recording '''metrics''' about how players played our games, and feeding that back into the design process. We dump a lot of information to a database, and then we mine that information for clues about how to optimize certain aspects of our games, whether it's "fun" or "ARPU" or whatever. This happens during development, and now that a lot of games are online, it happens after we ship, with the design being constantly iterated based on metrics gathered live after launch.

Now, metrics and closed feedback loops are great in the abstract, but like any tool, they can be misused. Just as it's usually a bad idea to design a game based totally on your intuition without grounding that intuition in the cold hard reality of another human being touching the controls, it's also bad to blindly gather and follow metrics data.

In the old days, we erred too far on the intuition side of the equation, but now I fear we're erring too far on the metrics side of the equation. This is what I call ''metrics fetishism''.

The problem is not that the data is wrong<ref>Assuming there aren't bugs in the metrics code instrumentating the game code, of course, which is probably not a great assumption, but anyway...</ref>, the problem is that we tend to gather the data that is convenient to gather, '''we worship that data because it is at least some concrete port in the storm of game design and player behavior''', and then we assume we can take a reasonable derivative from that data and hill-climb to a better place in design-space.

But, as anybody who knows any math can tell you, hill-climbing is not a very good algorithm for optimizing complex functions, and game design is a very complex function indeed. The problem with hill-climbing is it can only see ''local maxima'', it doesn't have any visibility into where the ''global maxima'' lie in the space. If you blindly hill-climb, you will end up at the top of the nearest hill, terrified to move because all derivatives point down, while the giant mountain of game design awesomeness is sitting right over there.

'''We really want a mix of intuition and metrics.''' The intuition gets you out of local maxima, and yes, it sometimes makes things worse for a bit, but it's our only hope for finding higher hills. In a globally optimizing algorithm like [http://en.wikipedia.org/wiki/Simulated_annealing simulated annealing], the intuition is the equivalent of turning up the temperature so you jump around in state space. The metrics allow you to polish those new points to perfection, but metrics alone won't find points worth polishing...you need to trust your intuition for that. The metrics are the craft, and the intuition is the art.

<references/>

[[Category:Lectures]]
[[Category:Opinions]]