Facebook's Miscalculated Metrics: What Marketers Need to Know

Over the past couple of months, you may have heard some things about Facebook's metrics.

There was talk of numbers -- lots of them. Things were overestimated. Others were underestimated. People were kind of upset. But mostly, they were confused. What the heck happened? How was Facebook going to respond? And at the end of the day, what did it mean for marketers? Breathe, and don't panic -- we're here to answer all of that. But before we dive in, let's make one thing clear -- none of it is the end of the world.

In fact, most of the issues have already been addressed and repaired; at this point, the most important item on our agenda is to clarify what's actually going on.

What Happened?

It started with video

The drama began in September 2016, when Facebook revealed that there was a problem with its video viewership metrics -- the average time that users spent watching videos was being largely overestimated.

Mathematically, Facebook wrote in a statement, that metric should have been the resulting figure from dividing the total time spent watching a video by the total number of people who played it. Instead, the total viewing time was divided by the number of times the video was watched for three seconds or more.

So, let’s say a video received a total viewing time of five hours, or 300 minutes, and it was watched by a total of 1,000 people, 700 of whom watched it for at least three seconds. The viewership metric should be 30%. Instead, Facebook was dividing those 300 minutes by 700, resulting in a larger metric of nearly 43%. And, says the Wall Street Journal [WSJ], that went on for nearly two years.

For a social media platform that boasts how effective its video tools are for marketers, the announcement was an embarrassment. The advertising world was especially unhappy about it -- Publicis Media, an ad-buying agency, told its clients that Facebook indicated viewing time overestimates of up to 80%. There were calls for third-party metric verification protocols to be put in place, and while Facebook said that it fixed the error and would be looking into such improvements, the metric misfortune didn’t end there.

A bit of a bug

In fact, just yesterday, Facebook announced that it discovered a bug in its Pages Insights that’s been lurking since May. The summary displaying seven- or 28-day organic page reach was incorrectly added up as the sum of daily reach over that period. That means duplicate visitors were being counted in every instance, leading to a number that was 33% higher than it should have been for seven-day summaries, and 55% for the 28-day ones. Facebook clarified that this error would not impact paid ads.

Here’s how Facebook visually represented the error -- the red circle indicates where the duplicate viewership would have appeared.

Source: Facebook

But you’ll notice that there are green circles in that image, too. Those indicate the insights that were unaffected by the bug -- which was the “vast majority” of them -- and includes the following measurements:

• All graphs
• Daily and historical reach
• Per-post reach
• Exported and API reach data
• All data on the Reach tab

What else was impacted?

In addition to the Page Insights, the bug really only impacted a total of four out of Facebook’s 220 measured metrics, according to WSJ. The remainder included:

More video miscalculations.

This time, the “video views at 100%” -- which has been renamed to “video watches at 100%” -- metric was impacted, thanks to a glitch that sometimes causes a video’s audio and visual components to be unsynced.

That means that even though the visual is played to completion, the audio may continue after the visual stops. But since about 85% of Facebook video is consumed without sound, viewers are likely to stop watching the video before this latent audio completes. As a result, “video watches at 100%” metrics might now increase by an estimated 35%.

Instant articles.

Here’s another case of Facebook’s overestimations. The average time spent reading Instant Articles -- a method by which Facebook displays news articles at a rate 10X faster than a typical mobile web browser -- was reported to be 7-8% higher than the actual length of time per article.

Referrals.

In Facebook’s Analytics for Apps dashboard, “referrals” are intended to measure the number of clicks on a post that were directed to an app or website. But it turns out that the “referrals” metric was counting more than that, and inaccurately also included clicks on the same post to view media, like photo or video. That led to an overestimate of referrals by about 6%.

Facebook’s Response

In Facebook’s defense, significant measures have been taken to resolve all of the above issues.

For some, the errors pertaining to ads seem to be the most pressing, which could be why the social media platform has dedicated an entire page to the updates around ads reporting alone. Most of those changes are intended to provide clarification over what exactly is being measured and how -- mostly in the interest of “fairness and transparency,” Mark Rabkin, Facebook’s VP of core ads, told WSJ.

Plus, Facebook claims to be taking the feedback to implement third-party measuring protocols seriously, and aims to further clarify how it’s going to calculate ad viewership, as well as the source of that data. Some of it will be coming from Moat and Integral Ad Science -- platforms that are used to measure ad and content engagement -- which will be used to measure display ad campaigns (previously, those platforms were only available to measure video campaigns).

But Facebook is also enlisting the help of a true viewership pioneer: Nielsen.

Nielsen has its own Digital Content Ratings metric, which Facebook will be implementing to count video viewership -- both on-demand and live. That comes with Nielsen’s Total Audience Measurement, which helps marketers compare digital metrics to those from TV.

There’s also a new blogging property launching -- Facebook’s Metrics FYI -- which will contain regular updates about any and all changes to the platform’s metrics henceforth.

These efforts are all compounded by the formation of a Measurement Council -- or, as we like to call it, Facebook’s jury of peers. The Council will be comprised of “business and measurement executives,” and is a bit of an extension of Facebook’s existing Client Council, which helped to develop the tools that help businesses measure ROI.

What It All Means for Marketers

So just how seriously should we be taking it? Well, in short, marketers have reason to be happy about the improvements that Facebook is making, but shouldn’t freak out over the miscalculations.

Why is that? According to Daria Marmer, HubSpot’s social product manager, “Most of the metrics in question are what we’d call vanity metrics. Views and impressions are important, but don’t have a huge impact on your business at the end of the day.”

And while Marmer echoes the benefits of Facebook’s measures to fix these discrepancies, “We really encourage marketers to tie their social efforts to more concrete metrics,” she said, “such as website visits, downloads, new leads.”

She adds, “The social data from Facebook in HubSpot customers’ portals won’t change based on these updates.”

We’ve got you covered. And, we’ll continue to bring you updates to all things social as they emerge.

Editor's update: As of December 9, 2016, we have learned that Facebook is correcting three additional miscalculated metrics. The first is an improvement to how a campaign's daily and total potential reach are estimated. That reach now takes into account new factors, like audience size. The second is a correction to Live Video insights, as Insights reported higher engagement due to misallocated reactions. Finally, the Facebook Graph API was reportedly showing potentially incorrect counts for Like and Share Buttons. Those totals are currently showing to be either higher or lower than those shown when a URL is entered into the Facebook mobile app's search bar. For more information, check out Facebook's official announcement here.