Niantic Labs deserves a ton of credit for taking a license of a very popular piece of IP and blending it with a novel platform – Augmented Reality – and turning it into Pokemon Go, one of the most successful launches of all time. They ran into a few problems due to insane load that was probably at least a single order of magnitude higher than they were expecting, but honestly got it under control fairly quickly. I know some of you probably disagree, but having worked for large companies that supported out huge game launches (I was at Microsoft/Xbox when they launched a number of big titles) and startups that suddenly gain a huge amount of traction they weren’t expecting, I promise you that Niantic handled the technical side of expansion about as well as you can ask for.
The problem is that they’ve basically done everything wrong since.
Zero Transparency, No Post-Mortems, No Updates
As of this blog post (August 2nd, 2016), here are Niantic’s channels’ most recent updates:
Facebook: August 2nd, 2016 – Subject: Recent Changes to Pokemon Go
Twitter: July 17th, 2016 – Subject: Pokemon Go launches in Canada
Corporate Blog: July 6th, 2016 – Subject: Pokemon Go launches in AUS/NZ/USA
The Facebook “press release” goes into zero detail on why features are being removed. This is completely unacceptable. By contrast, here is an example of what a company should do when servers go down or there are technical problems – GitHub Incident Report, Jan 28th, 2016. Note how GitHub’s release follows this flow:
- The Event (What Happened?)
- Diagnosis of the Problem (How Did We Fix it?)
- Future Work (How Will We Make Strides to Ensure This Never Happens Again?)
- Conclusion (Sincere Apology)
This is how every single technical outage should be documented. Furthermore, when updates are issued or technical problems cause the removal of previously-expected features (three-footstep bug turning into no tracking at all), a similar release should be issued.
I can only guess why Pokemon Go can’t track local pokemon; this seems like an embarrassing bug that should have prevented the app from shipping entirely. From what I’ve seen (I’ve done a fair bit of reverse engineering of Pokemon Go and have contributed to patches of third-party APIs and bots), Pokemon Go local tracking should work like this:
- Phone sends server GPS coordinates
- Server responds with ordered list of lat/long pairs with pokemon in X kilometers radius
- Phone runs local calculation on distances and displays pokemon on the screen with distances to each pokemon using footprints or other method
Note that steps #1 and #2 must happen for the game to work anyway, so adding step #3 makes no sense that it would “increase server load” as some have claimed.
This leads us to…
Inability to Talk Like a Human
Niantic’s second problem is that they have zero capacity or desire to talk to their users like they are people. This was a problem with Ingress and I’m not surprised to see it continue with Pokemon Go. Niantic has long been considered a company with extremely poor customer service and zero transparency into their actions. Nintendo/The Pokemon Company should have seen this coming and allocated resources to protect their IP, but it’s not like Nintendo has a very good reputation in these areas, either.
Niantic’s last press release – on Facebook and not copied to their corporate blog, which is another huge mistake – reads exactly how disconnected politicians speak to their constituents:
As many of you know, we recently made some changes to Pokémon GO.
– We have removed the ‘3-step’ display in order to improve upon the underlying design. The original feature, although enjoyed by many, was also confusing and did not meet our underlying product goals. We will keep you posted as we strive to improve this feature.
– We have limited access by third-party services which were interfering with our ability to maintain quality of service for our users and to bring Pokémon GO to users around the world. The large number of users has made the roll-out of Pokémon GO around the world an… interesting… challenge. And we aren’t done yet! Yes, Brazil, we want to bring the game to you (and many other countries where it is not yet available).
The first point about the ‘3-step display’ is at best a half-truth because while it was working, it was not confusing at all! It worked very well. Then it was permanently broken – on purpose – and that was confusing. But the feature as it was designed was something that gamers enjoyed and expected. It is clear that the display was a technical problem, and as stated above, all technical problems should be documented and explained. Niantic should have issued a press release that contained language similar to what I’ve drafted:
We have removed ‘3-step tracking’ temporarily from Pokemon Go. We know this is a disappointing piece of news, and we’d like to take this opportunity to explain what went wrong and what we’re doing to fix it.
What Went Wrong?
Our code that we wrote to track pokemon globally and locally scaled poorly. Our inability to plan for scale this wide came back to bite us. In short, we were making too many server calls when we should have been pushing more of the decisions locally and figuring out a better caching plan. Unfortunately, a short-term fix is not in the cards at the moment, so we’ve decided to disable tracking instead of continuing to display a broken feature.
How are We Fixing it?
Right now the client-side app makes too many calls to the server, which artificially increases server load and makes the game unstable for all users. A fix involves decreasing unnecessary communication between server and client and potentially reworking exactly how pokemon spawn. We’ve brought in some expert network consultants to help us rewrite this code from scratch, and we have an alpha version in testing in certain areas right now.
When Will It Be Fixed?
While we do not have a hard ETA, you can expect weekly updates with technical writeups on where we are at – found on our blog and Facebook accounts.
We know that we’ve failed the gaming community at large, and all of us at Niantic would like to apologize for the impact of temporarily disabling this feature. We will continue to work on this issue and will keep the community in the loop at all times with technical progress. Thank you for your continued loyalty, and we continue to work hard to earn your trust.
Refusal to Engage a Motivated Community
Niantic’s desire to shut down services like Pokevision and other third-party tracking sites is exactly what you would expect from a company that is governed by fear and very short-sighted MBA-type empty suits with zero desire to connect with their users. Niantic sees these users as disruptive hackers, when in reality, all successful companies that have a semi-adversarial relationship with users (IT security, for example) do their best to ally themselves with the hackers. When it comes to cybersecurity, smart companies pay out Bug Bounties for hackers who responsibly disclose security holes in their products and services; dumb companies contact law enforcement and try to crack down. The latter approach simply drives all the bugs underground and ensures they will be exploited for black market reasons (and it also generates an enormous amount of bad will), the former approach rewards freelancers for improving their product and taking an interest in improving it.
When Niantic discovered Pokevision (a website that tracked Pokemon Go pokemon spawning at any location in the world), they should have contacted the authors of the tool and offered to hire them in a capacity that helped them solve their ‘3-step tracking’ issues as well as develop future features for the game. If Niantic was worried about server load from these tools, they should have reached out with a rate limiting plan and worked with the website to cooperate together. From the outset, Pokevision repeatedly stated they would take the website down if Niantic or Nintendo requested, showing that they were good actors with the best of intentions. This is an ideal scenario where you engage them and the community at large to fix a problem you have in a collaborative effort. Not only do you end up with a better service, but you generate goodwill in the community.
Niantic needs to admit their product is broken and that they didn’t take security all that seriously (MITM attacks continue to succeed despite the fact that Niantic just installed certificate pinning in their app in an attempt to block HTTPS sniffing). It’s not a war they can afford to fight when they already have a ton of other fires they’re stoking with their heavy-handed approach, and attempting to control what users do with a shipped binary on user-controlled hardware… well, let’s just say that it’s not a battle that can really be won.
Niantic Labs is a company just like the majority of the new American businesses that come to power – full of MBA-type assholes who do everything to limit legal exposure and see customer support as a sinkhole. It’s a perfect parallel to our current political climate, where the majority of our representatives provide little to no transparency on why they vote the way they do and do not document their processes for their constituents – yet we blame American voting turnout on the individuals, not the people above us who fail to connect with your every day person. Corporate strategy follows that exact same model, and there’s no surprise that there’s a massive backlash with users filing for refunds, complaining to Apple/Google in droves, and hacking their software left and right to provide the features Niantic should have had on lockdown from the beginning.
If you futilely swim upstream long enough, don’t be surprised when the current eventually overtakes you. It might require a company (or politician) to swallow a bit of ego and allocate resources to supporting customers (gasp!), but the effort is well-worth the investment.