Game Analytics: Making Sense of Data

These days most game companies adopt some form of analytics: a process whereby data is collected on how well the game performs as well as how players behave within it. Game Analytics is what allows major commercial titles such as Fortnite to react to community behavior, rolling out frquent updates addressing gameplay balancing for example. Many games, notably those adopting a freemium monetization model, require analytics for the purpose of recognising levels of engagement in an effort to drive revenue from the players.

Game development is a wonderful combination of art, craft, and science. Game Analytics is an example of the science component. It is a professional discipline focused on collecting and analysing data to support decisions about designing and producing games, improving user experiences, prioritizing features, and much more, across all parts of an organization (design, marketing, user research, management, etc.), in operational, tactical, and strategic contexts. It sounds like quite a mouthful because Game Analytics is so broad in scope, but as a discipline it shares DNA with GUR in that both try to supplement the personal experience and intuition that is key to much of game development with empirical methods and evidence.

An example of a simple analysis based on behavioral telemetry from the MMOG Tera Online. The left diagram shows a financial index for a group of classes from an early version of the MMORPG Tera. It indicates some differences. Looking at the maximal difference between the highest earning class and the lowest across levels, we see a consistent 10-30% difference. This can in turn lead to player frustration so a class balancing is needed. In this way Game Analytics can directly inform design about potential problems, and how to solve them. 

Game Analytics has a heavy focus on how users (i.e. players) interact with the game and has become a cornerstone of game development within a short decade. This rapid introduction stems from a confluence of factors, notably:

1. A general requirement for business intelligence and analytics in the game sector, as detailed data from platforms, players, advertisers, and payment processors became available.
2. Within the AAA context, the need to enrich traditional kinds of live user testing data such as screen capture and surveys with details about player choices and actions.
3. The emergence of online social games embedded in pre-existing, data-rich networks such as Facebook.
4. The rise of the mobile phone and the freemium business model which, like an online storefront, relies on analysis of user behaviour to optimize revenue.
5. The advent of new, better, and more readily accessible tools for collecting and searching large volumes of information (i.e. ‘big data’ technologies). The games industry today serves over 2.4 billion customers and comprises a US $130 billion market, with high rates of innovation across design, technology, and business models. With exponential increases in data processing added to the mix, it is no wonder that data-driven analytics have seen as rapid growth in the games industry as in many others in recent years.

What is Game Analytics?

Game Analytics is formally the process of discovering and communicating patterns in data, as applied to game development and research. Game Analytics is a source of business intelligence in game development, not only about games and the people who play them, but also the process of developing games and the process of running a game company. Despite some trepidation to the contrary, Game Analytics emphatically does not mean eliminating personal experience or common sense in game design, but rather empowering these through empirical insights. Analytics combines statistics, data mining, machine learning, engineering, programming, and operations research, as well as data visualization, communication, and systems thinking. It is a broad field covering virtually all aspects of collecting, storing, analysing, and communicating data and information. Despite being a relatively new element in game development, Game Analytics has grown to be virtually ubiquitous in the industry because it improves our understanding of the players. Game Analytics’ focus on user behaviour often considers users from two perspectives: as customers and as players. In the former case, we focus on the value the user provides to the business, and typically investigate patterns in purchasing/spending behaviour, churn prediction, finding average revenue per user, etc. In the latter, the focus is the value we are providing to the user, as reflected by their behaviour and the user experience. The two are sometimes hard to separate, especially in free-to-play games, but remembering this distinction is important.

What is game analytics for?

Analytical methods can generate answers to a wide variety of questions. The most complex of these benefit from or even require direct collaboration with other disciplines (design, marketing, games user research, etc.) to get past the raw behaviours of the players to an understanding of player motivation, personality, and the context of play in order to provide actionable insight. Requirements on the data science expertise of the analyst vary. Basic questions, such as finding out the number of daily active users, tend to be close to or entirely automated these days, and require minimal in-house data science resources. Problems like predicting player churn, on the other hand, require more advanced expertise and resources, and thus are commonly outside the capacity of a small developer. Middleware companies and consultants are readily available to fill this gap, but small organizations especially need to be careful that the optimizations these groups can provide are worth the fees.

In essence, the more complex the kinds of analytics you want to run, the higher the organizational maturity in GA needs to be. A full-blown discussion of maturity and capacity-building is outside the scope of this chapter, but there are four key dimensions to consider:

1. Volume: How much data do you have? This is mostly a function of the scale of your game (e.g., the number of players) and the level of detail you log. Equally important is how much data you can effectively use based on your infrastructure.

2. Variety: How many different types of data do you have? Which different behaviours are recorded from the game itself, and what additional data sources (advertising, revenue, operational logs, etc.) do you have access to?

3. Capability: What types of skills and tools can your analysts bring to bear to answer your questions and solve your problems? Data wrangling, data visualization, data science, and technical communication skills are all important here.

4. Acceptance: How is your organization acting on the insights analytics provides? You can expect product owners to consider key performance indicators (KPIs) when making business decisions, but acceptance can range from there through to informing the design process with detailed behavioural analysis to incorporating machine-learning tools (or the outputs thereof) directly into the user experience.

These dimensions push and pull on one another to determine what an organization can really do with GA.

The kinds of questions user-focused game analytics can answer, in ascending order of complexity and difficulty. The further down on the scale, the harder it is for analytics to answer the question in general, and the more substantial the benefits of integrating with user research. 

Case Study: Building a Player Recommendation System for Destiny

Game Analytics represents a unique domain through which to apply machine learning. In the big data context, gaming data is high frequency and longitudinal. A game server may carefully track every single action of a player in their lifetime. All this data gives rise to a rich diversity of analyses from funnel analysis, churn prediction to engagement graphing and profiling. However, in Game Analytics, recommender systems, i.e. systems that try to predict a useful further action, product or similar, remain relatively unexplored (for a great introduction see here).

Recommender systems for gaming form a unique challenge. We are not talking about recommending which games to play; rather, recommending certain behaviors to increase performance in a game. MMOGs like Destiny are complex and require so much decision making that players often don’t even know how to improve. Should I use this skill tree or equip this weapon? The obvious answer to improve is to play more. Someone who has played 10,000 hours will be significantly better than someone who has played 10. Recommender systems provide a means for systematically helping players improve through data. The thing that makes recommenders hard for gaming is that that different players will have different preferences over how they play a game. We would not want to suggest to a shotgun player that they should use a sniper rifle.

In this example, supported by game developer and publisher Bungie, we built a recommender system for the online First-Person Shooter title Destiny, aiming to help players improve at the game by suggesting different weapon loadouts and stat allocations. The key here is that we suggest actions that are still true to the player’s preferences. To do this, we first cluster the players based on their unique playstyle and stat preferences. Our hybrid recommender system involves taking different intersections of profiles and then recommending players that are similar, but with a higher combat rating. Our online evaluation on Reddit showed that the results were of great interest to players and that they would act on the recommendations if the system was a live service. Analytical systems designed to help players improve can have a huge impact on not only games like Destiny but also eSports in the future. To read more about this project, check out this Gamasutra post.

The Destiny recommender system utilizes a hybrid approach. hybrid means a system that utilizes multiple approaches to build recommendations. In Destiny, players control a character that has access to hundreds of guns and special skills in order to defeat monsters and other players. Building multiple profiles across different categories of game features permits recommendations to be made against a flexible number of these profiles. 

Example report from the Destiny player recommender system. The report recommends weapons to try, specific loadouts, playstyles, and additional information guiding the player in developing their skills, based on players with similar playstyle but better performance. 

See below for a range of case studies in Game Analytics

How Players Lose Interest in Playing a Game: An Empirical Study Based on Distributions of Total Playing Time

An important problem in game analytics is to characterize how player engagement evolves over time. Reliable models allow for assessing the long-term success of game products and can provide estimates of how long players may be expected to keep actively playing a game. We introduce methods from random process theory into game data mining in order to draw inferences about player engagement. In this paper, we find that the Weibull distribution gives a good account of the statistics of total playing times. This implies that an average player’s interest in playing one of the games considered evolves according to a non-homogeneous Poisson process. Therefore, given data on the initial playtime behavior of the players of a game, it becomes possible to predict when they stop playing.

Modelling Early User-Game Interactions for Joint Estimation of Survival Time and Churn Probability

Data-driven approaches which aim to identify and predict player engagement are becoming increasingly popular in games industry contexts. This is due to the growing practice of tracking and storing large volumes of in-game telemetries coupled with a desire to tailor the gaming experience to the end-user’s needs. These approaches are particularly useful not just for companies adopting Game-as-a-Service (GaaS) models(e.g. for re-engagement strategies) but also for those working under persistent content-delivery regimes (e.g. for better audience targeting). A major challenge for the latter is to build engagement models of the user which are data-efficient, holistic and can generalize across multiple game titles and genres with minimal adjustments.This work leverages a theoretical framework rooted in engagement and behavioural science research for building a model able to estimate engagement-related behaviours employing only a minimal set of game-agnostic metrics. Through a series of experiments we show how, by modelling early user-game interactions,this approach can make joint estimates of long-term survival time and churn probability across several single-player games in a range of genres. The model proposed is very suitable for industry applications since it relies on a minimal set of metrics and observations, scales well with the number of users and is explicitly designed to work across a diverse range of titles.

Predicting skill learning outcomes in a large, longitudinal MOBA dataset

The exploration of the relationships between behavior and cognitive psychology of game players has gained impetus in recent years because such links provides an opportunity for improving user experiences and optimizing products in the games industry. At the same time, the volume and global scope of digital game telemetry data has opened up new experimental opportunities for studying human behavior at large scales. Prior research has demonstrated that a relation exists between learning rates and performance. Although many factors might contribute to this correlation at least one may be the presence of innate cognitive resources, as demonstrated in recent work relating IQ and performance in a Multi-player Online Battle Arena game. In this paper, we extend this work by examining the relationship between early learning rate and long term performance using a 400,000 player longitudinal dataset generated by new players of the widely-played MOBA League of Legends. We observed that the learning rate of new players in a competitive season explains a significant amount of variance in the performance at the end of the year. This analysis was then extended by training two multivariate classifiers (Logistic Regression, Random Forest) for predicting players who by the end of the season would be considered masters (top 0.05%), based on their performance in the first 10 matches of the same season. Both classifiers performed similarly (ROC AUC 0.888 for Logistic Regression, 0.878 for Random Forest), extending the time frame for skill prediction in games based on a relatively sparse sample of early data. We discuss the implications for these findings based on preexisting psychological studies of learning and intelligence, and close with challenges and direction for future research.

Win Prediction in Multi-Player Esports: Live Professional Match Prediction

Esports are competitive videogames watched by audiences. Most esports generate detailed data for each match that are publicly available. Esports analytics research is focused on predicting match outcomes. Previous research has emphasised pre-match prediction and used data from amateur games, whichare more easily available than professional level. However, the commercial value of win prediction exists at the professional level. Furthermore, predicting real-time data is unexplored, as isits potential for informing audiences. Here we present the first comprehensive case study on live win prediction in a professional esport. We provide a literature review for win prediction in a multi-player online battle arena (MOBA) esport. The paperevaluates the first professional-level prediction models for liveDotA 2 matches, one of the most popular MOBA games and trials it at a major international esports tournament. Using standardmachine learning models, feature engineering and optimization,our model is 85% accurate after 5 minutes of gameplay. Our analyses highlight the need for algorithm evaluation and optimization and how professional data differs from amateur data.Finally, we present implications for the esports/game analytics domains, describe commercial opportunities, practical challenges,and propose a set of evaluation criteria for research on esports win prediction.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s