Game Analytics: Making Sense of Data

These days most game companies adopt some form of analytics: a process whereby data is collected on how well the game performs as well as how players behave within it. These data are processed and analyzed, and results serve to inform decision making and business processes across companies.

Game Analytics is what allows major commercial titles such as Fortnite to react to community behavior, rolling out frequent updates addressing gameplay balancing for example. Many games, notably those adopting a freemium monetization model, require analytics for the purpose of recognising levels of engagement in an effort to drive revenue from the players.

Game development is a wonderful combination of art, craft, and science. Game Analytics is an example of the science component. It is a professional discipline focused on collecting and analysing data to support decisions about designing and producing games, improving user experiences, prioritizing features, and much more, across all parts of an organization (design, marketing, user research, management, etc.), in operational, tactical, and strategic contexts. It sounds like quite a mouthful because Game Analytics is so broad in scope, but as a discipline it shares DNA with GUR in that both try to supplement the personal experience and intuition that is key to much of game development with empirical methods and evidence.

An example of a simple analysis based on behavioral telemetry from the MMOG Tera Online. The left diagram shows a financial index for a group of classes from an early version of the MMORPG Tera. It indicates some differences. Looking at the maximal difference between the highest earning class and the lowest across levels, we see a consistent 10-30% difference. This can in turn lead to player frustration so a class balancing is needed. In this way Game Analytics can directly inform design about potential problems, and how to solve them. 

Game Analytics has a heavy focus on how users (i.e. players) interact with the game and has become a cornerstone of game development within a short decade. This rapid introduction stems from a confluence of factors, notably:

1. A general requirement for business intelligence and analytics in the game sector, as detailed data from platforms, players, advertisers, and payment processors became available.
2. Within the AAA context, the need to enrich traditional kinds of live user testing data such as screen capture and surveys with details about player choices and actions.
3. The emergence of online social games embedded in pre-existing, data-rich networks such as Facebook.
4. The rise of the mobile phone and the freemium business model which, like an online storefront, relies on analysis of user behaviour to optimize revenue.
5. The advent of new, better, and more readily accessible tools for collecting and searching large volumes of information (i.e. ‘big data’ technologies). The games industry today serves over 2.4 billion customers and comprises a US $130 billion market, with high rates of innovation across design, technology, and business models. With exponential increases in data processing added to the mix, it is no wonder that data-driven analytics have seen as rapid growth in the games industry as in many others in recent years.

What is Game Analytics?

Game Analytics is formally the process of discovering and communicating patterns in data, as applied to game development and research. Game Analytics is a source of business intelligence in game development, not only about games and the people who play them, but also the process of developing games and the process of running a game company. Despite some trepidation to the contrary, Game Analytics emphatically does not mean eliminating personal experience or common sense in game design, but rather empowering these through empirical insights. Analytics combines statistics, data mining, machine learning, engineering, programming, and operations research, as well as data visualization, communication, and systems thinking. It is a broad field covering virtually all aspects of collecting, storing, analysing, and communicating data and information. Despite being a relatively new element in game development, Game Analytics has grown to be virtually ubiquitous in the industry because it improves our understanding of the players. Game Analytics’ focus on user behaviour often considers users from two perspectives: as customers and as players. In the former case, we focus on the value the user provides to the business, and typically investigate patterns in purchasing/spending behaviour, churn prediction, finding average revenue per user, etc. In the latter, the focus is the value we are providing to the user, as reflected by their behaviour and the user experience. The two are sometimes hard to separate, especially in free-to-play games, but remembering this distinction is important.

What is game analytics for?

Analytical methods can generate answers to a wide variety of questions. The most complex of these benefit from or even require direct collaboration with other disciplines (design, marketing, games user research, etc.) to get past the raw behaviours of the players to an understanding of player motivation, personality, and the context of play in order to provide actionable insight. Requirements on the data science expertise of the analyst vary. Basic questions, such as finding out the number of daily active users, tend to be close to or entirely automated these days, and require minimal in-house data science resources. Problems like predicting player churn, on the other hand, require more advanced expertise and resources, and thus are commonly outside the capacity of a small developer. Middleware companies and consultants are readily available to fill this gap, but small organizations especially need to be careful that the optimizations these groups can provide are worth the fees.

In essence, the more complex the kinds of analytics you want to run, the higher the organizational maturity in GA needs to be. A full-blown discussion of maturity and capacity-building is outside the scope of this chapter, but there are four key dimensions to consider:

1. Volume: How much data do you have? This is mostly a function of the scale of your game (e.g., the number of players) and the level of detail you log. Equally important is how much data you can effectively use based on your infrastructure.

2. Variety: How many different types of data do you have? Which different behaviours are recorded from the game itself, and what additional data sources (advertising, revenue, operational logs, etc.) do you have access to?

3. Capability: What types of skills and tools can your analysts bring to bear to answer your questions and solve your problems? Data wrangling, data visualization, data science, and technical communication skills are all important here.

4. Acceptance: How is your organization acting on the insights analytics provides? You can expect product owners to consider key performance indicators (KPIs) when making business decisions, but acceptance can range from there through to informing the design process with detailed behavioural analysis to incorporating machine-learning tools (or the outputs thereof) directly into the user experience.

These dimensions push and pull on one another to determine what an organization can really do with GA.

The kinds of questions user-focused game analytics can answer, in ascending order of complexity and difficulty. The further down on the scale, the harder it is for analytics to answer the question in general, and the more substantial the benefits of integrating with user research. 

Case Study: Building a Player Recommendation System for Destiny

Game Analytics represents a unique domain through which to apply machine learning. In the big data context, gaming data is high frequency and longitudinal. A game server may carefully track every single action of a player in their lifetime. All this data gives rise to a rich diversity of analyses from funnel analysis, churn prediction to engagement graphing and profiling. However, in Game Analytics, recommender systems, i.e. systems that try to predict a useful further action, product or similar, remain relatively unexplored (for a great introduction see here).

Recommender systems for gaming form a unique challenge. We are not talking about recommending which games to play; rather, recommending certain behaviors to increase performance in a game. MMOGs like Destiny are complex and require so much decision making that players often don’t even know how to improve. Should I use this skill tree or equip this weapon? The obvious answer to improve is to play more. Someone who has played 10,000 hours will be significantly better than someone who has played 10. Recommender systems provide a means for systematically helping players improve through data. The thing that makes recommenders hard for gaming is that that different players will have different preferences over how they play a game. We would not want to suggest to a shotgun player that they should use a sniper rifle.

In this example, supported by game developer and publisher Bungie, we built a recommender system for the online First-Person Shooter title Destiny, aiming to help players improve at the game by suggesting different weapon loadouts and stat allocations. The key here is that we suggest actions that are still true to the player’s preferences. To do this, we first cluster the players based on their unique playstyle and stat preferences. Our hybrid recommender system involves taking different intersections of profiles and then recommending players that are similar, but with a higher combat rating. Our online evaluation on Reddit showed that the results were of great interest to players and that they would act on the recommendations if the system was a live service. Analytical systems designed to help players improve can have a huge impact on not only games like Destiny but also eSports in the future. To read more about this project, check out this Gamasutra post.

The Destiny recommender system utilizes a hybrid approach. hybrid means a system that utilizes multiple approaches to build recommendations. In Destiny, players control a character that has access to hundreds of guns and special skills in order to defeat monsters and other players. Building multiple profiles across different categories of game features permits recommendations to be made against a flexible number of these profiles. 

Example report from the Destiny player recommender system. The report recommends weapons to try, specific loadouts, playstyles, and additional information guiding the player in developing their skills, based on players with similar playstyle but better performance. 

See below for a range of case studies in Game Analytics

How Players Lose Interest in Playing a Game: An Empirical Study Based on Distributions of Total Playing Time

An important problem in game analytics is to characterize how player engagement evolves over time. Reliable models allow for assessing the long-term success of game products and can provide estimates of how long players may be expected to keep actively playing a game. We introduce methods from random process theory into game data mining in order to draw inferences about player engagement. In this paper, we find that the Weibull distribution gives a good account of the statistics of total playing times. This implies that an average player’s interest in playing one of the games considered evolves according to a non-homogeneous Poisson process. Therefore, given data on the initial playtime behavior of the players of a game, it becomes possible to predict when they stop playing.

Modelling Early User-Game Interactions for Joint Estimation of Survival Time and Churn Probability