The futility of predictions (and a path for seizing success)

I’ve now worked in the games industry for well over a decade. And with the accumulation of experience (and hopefully some wisdom), I’ve formed a few convictions. One thing I feel strongly about is that “hits” (massive successes) are fundamentally unpredictable.

What do I mean by this? The industry’s biggest hits (in terms of engagement or revenue, and regardless of platform) are often surprises out of left field. Helldivers 2 and Palworld are two examples just in the past two months; from the past 5 years, Monopoly Go!, TFT China (Fight for the Golden Spatula), Among Us and Fall Guys come to mind; and if we go back to when I was a newbie to the industry, Minecraft and League of Legends both seemed to come from nowhere. After the fact, people can come up with plenty of rationale to backwards justify each of these cases, either trying to look smart (”it wasn’t a surprise to me!”) or being driven by an innate desire for attributing causality. But before the fact, if one were to analyze any of these games and try to assess a business case – I’d argue that the more you thoroughly analyzed the less you’d be able to confidently present a bullish case. Each of these cases was a low probability event.

Let’s take Helldivers 2 as a concrete example (I’m lovin’ it, and just want to talk about it). Its sales performance has blown by internal expectations. But pre-launch, there seemed to be little to be bullish about this title:

  • It’s competing in a crowded 3rd-person shooter market, and this is the first game from developer Arrowhead in this space (lots of execution risk on “table stakes” features);
  • To make matters worse, it’s made on a discontinued engine, Autodesk Stingray, further raising concerns of execution risk;
  • As a sequel title, the brand has little recognition; the sci-fi setting feels generic, and again, the competition is fierce (Halo, Destiny);
  • The co-op gameplay is punishing (mandatory friendly-fire), which suggests a niche audience and presents a challenge for onboarding and retention.

Post-launch, it’s clear that not only did Arrowhead nail the execution, but the gameplay also managed to capture the zeitgeist as a bit of counter-programming to modern shooter design conventions. And if you trace back to the original Helldivers, it’s “obvious” in hindsight that the team had already “found the fun” of the core game mode (albeit in a top-down camera-angle; and thus the design risk is not as big as it first seems). Furthermore, when faced with the unexpected success, Arrowhead managed to quickly overcome the server capacity issues, and thus sustain the momentum – this is an important feat that I’ll come back to later.

It’s not just that the “unknown” hits are unpredictable. Even for “sure-bets”, sensible analysts would likely be laughed out of the room if they made forecasts that actually lined up with the outlier results. Success is nonlinear, while the burden of proof in justifying a nonlinear forecast is insurmountable. One recent example is Elden Ring – FromSoftware had built up an impeccable record and a loyal fanbase prior to its release, so there was plenty of ammo to be bullish, yet the actual sales performance still shattered expectations (at least 2-3x internal forecasts). Going further back, it’s fun to look at Wall Street analysts’ forecasts for *GTAV* (probably the biggest “sure-bet” ever), and compare to the results.1

This is not a rant against analytics, or business forecasting. “All models are wrong, some are useful.” We need to remember that these are just tools to assist decision-making – “what needs to be true, to justify this amount of marketing spend?” The danger is to blindly follow the outputs of the models.

What I’ve found more interesting – and to make this post a bit more constructive – is to think about whether there is a better way to develop games, given that we know we have very little ability to make good predictions about outcomes. For starters, given the above discussion, it seems that we must allow the game to see the full light of day – no amount of confined player research is a substitute for a real launch and the potential surprises. I’d go as far as to argue, even in the cases where there is little organizational faith in a product, it should be allowed to launch – TFT China is one such example where Tencent launched it with minimal commitment and have been completely floored by its performance.2 This also means that the team needs to be able to cut their losses early, if needed; and that there needs to be a delicate balance between controlling the marketing budget and not starving the game of the minimal spend needed for a viable launch.

The next part is tricky. One thing that separates the outlier successes versus their peers is the developers’ readiness to solve the myriad emerging challenges associated with unexpected success. For example – Arrowhead’s performance so-far in tackling Helldivers 2 capacity scaling has been impressive, given this is the first time they’ve ever encountered such scale.3 And I’ve always felt that a big reason Epic Games leapfrogged PUBG with Fortnite was because they were perfectly positioned with the engineering know-how to quickly iterate and deliver a more polished and stable service. So ideally, the dev team has either the previous experience and/or has done some scenario planning in advance, so that they can better react to situations in real-time and not lose the momentum. But too much pre-planning can derail the development momentum as well, and the team could get paralyzed pre-solving problems that may never arrive. In hindsight everything is obvious – before then the team just has to make their bets and live with them.

In summary, I think I’m advocating for:

  • Don’t let forecasts dictate launch decisions;
  • Take real shots by launching games;
  • Build teams such that they have the problem-solving experience / skills to be responsive to the challenges of unexpected success.

This is by no means rocket science, nor is it the only way. But even when this seems to be the desired approach by an org, it’s very hard to actually practice it. There are a lot of needle-threading and judgment calls, which in hindsight looks either foolish or ingenious – “history is written by the victors.” But I find this ambiguity to also be the appeal of the craft.

  1. Even though many of the game examples I’ve used are premium games / AAA, I’ve seen the same phenomena in free-to-play.
  2. One typical push-back against this approach is the potential damage to the brand (in the case of a poor product). This is a real concern – Google now has an uphill battle with any new consumer product/service launch because they’ve launched and killed off so many (rather haphazardly). But this surely can be mitigated, by carefully managing the player expectations and the launch spend.
  3. I still remember that when I first joined Riot Games in 2011, the all-hands-on-deck situation was that the server tech at the time couldn’t keep up with the game’s growth, and this got to the point where the company had no choice but to make the painful decision to split the EU server into 2 servers to buy some time. (This migration itself took many months, and was painful and controversial in the community.)

1 thought on “The futility of predictions (and a path for seizing success)”

  1. Well written. Prediction is merely maximum likelihood estimation, and outliers are not gonna be part of it (by definition I guess). In an industry pushed forward by outliers, letting prediction take over decision making is the road to mediocrity.

    It’s still a good tool. Prediction serves the best as a basis for thesis/hypotheses validation. Many teams suffer from the lack of prediction (or the frequent/scientific validation thereof) where they just roll with whatever hypotheses they had to begin with until the market strikes with a does of reality.

    I guess the key is to predict small, validate fast. It’s good for iterations. But for a whole product? Greatness can’t be really predicted. We just know it when it happens.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.