4 Marketing Data Analysis Mistakes to Avoid
Marketers need to be data-driven to deliver business results. Yet outside of a statistics class you may have taken in high school or college, chances are you haven’t thought much about the biases you may be bringing to your data analysis and interpretation.
It’s easy to get caught up in jargon and business goals, which can actually obscure what you’re measuring—and the actual trends that your data are painting. With an awareness of common pitfalls, you can be on the lookout for missteps in your data analysis that may be clouding your vision.
Here are the top 4 mistakes you should have on your radar when looking at data so you can tell the real story about your KPIs.
1. Having Vague Objectives
With all of the data available to marketers (how many dashboards do you have bookmarked?), it may be tempting to just dive right in. But before you do, you want to make sure that you have a clear goal or objective outlined.
First, identify the research question you are investigating. Be as specific as possible here to ensure that you start your data collection process in the right place. Some questions to ask yourself:
- What business problem am I trying to solve?
- Which channels do I need to include in my analysis?
- What are the relevant KPIs?
- What time frame am I investigating?
- Where can I find the data I need? If it isn’t readily available, how do I collect it?
A clear articulation of your objective will unlock the right steps you need to take in your data collection and analysis. This step is fundamental—taking a little bit of extra time at the start to think about the big picture will ensure that you are measuring and analyzing the right metrics.
2. Misunderstanding How Metrics are Calculated
With deadlines and pressing business goals, many marketers rush to draw conclusions from the data available. However, knowing what your metrics actually represent is pivotal.
There are many things to keep in mind when looking at any single metric:
- What’s the literal formula that computes the data point?
- For aggregated metrics, such as CPA: How large is the sample size? If you’ve only had 10 acquisitions in the time period analyzed, for example, your sample size is likely far too small to draw generalized insights.
- What is the time frame being measured?
Marketers throw a lot of acronyms around, so make sure that you clearly understand the meaning of every single metric. This is key if your research includes multiple channels—the same metric may be computed with different formulas across various channels.
Take Engagement Rate, as an example. It’s calculated differently on Facebook than it is on Twitter. Because of platform-specific differences, you can’t necessarily neatly compare Engagement metrics between Facebook and Twitter without delving into the specific formulas.
If you wanted to aggregate Engagement Rate across platforms, you might consider recalculating the metric universally. For QuickFrame Video Intelligence, for instance, we use the following formula across all of the platforms we analyze to run comparison analyses seamlessly:
Engagement Rate = (Likes + Comments) / Views
3. Allowing Bias to Affect the Data
You must approach any data set with an open mind. You may have a strong hypothesis or instinct as to where the data will fall, but objectivity is key to uncovering the true patterns. Let the data tell the story rather than imposing your preconceived assumptions onto the spread.
There are a number of different biases that may subtly falsify your narrative.
Sampling bias occurs when you draw conclusions from data that is not representative of the entire population you are trying to understand. In the marketing world, this happens most often when you try to use a narrow set of data to draw generalized conclusions.
A simple example: if you are trying to assess the performance of a multi-channel campaign, you must collect and analyze the same metrics across all of the channels on which you are advertising. If you look at performance on only a few of these channels to draw overarching insights, your conclusions will be biased.
A related mistake is overfitting data. This bias occurs when a limited set of data points is forced to tell a larger governing story. If you’re running a month-long campaign, you cannot simply look at a single week’s performance and generalize for the entire month. You may have had great results in Week 2, but declining results in Weeks 3 and 4 may be pointing to critical strategy errors, such as not addressing creative fatigue.
Everyone has a hypothesis for why they are seeing a certain result. Confirmation bias occurs when you seek out (consciously or not)—or assign more weight to—data that confirms your hunch, and in that process, ignore data that could disconfirm your hypothesis.
Seeing a new downward trend after months of growth? It can be tempting to brush it off as a temporary blip, but you need to stay open to seeing the whole picture, even if it might be pointing to an uncomfortable conclusion.
A sort of flipside to confirmation bias is outlier bias: giving too much weight to an outlier, which is an abnormal or extreme data point.
If you’re looking at ad performance and see a major increase in performance on a single day, there may be factors outside of your testing variables that influence that abnormality. Maybe your ad went viral one day. While that may have been amazing for your performance for a day or two, you shouldn’t generalize the narrative for the entire length of the campaign, especially if performance on the other days falls in line with your benchmarks.
Outlier bias can seriously handicap your future judgment. Taking the example above, you may decide to go all-in on similar creative variables to chase another viral hit while ignoring that your campaign didn’t exceed benchmarks for most of the time it was active.
Don’t use outliers to build an overarching narrative. Instead, notice them, and then investigate the underlying causes.
Correlation Does Not Equal Causation
Almost all data analyses that marketers work with are correlations. A correlation suggests that two variables are related in some way. The biggest mistake marketers make all the time is assuming that a correlation suggests a causal relationship.
This is simply never the case unless you are running a highly controlled experiment where you are manipulating only a single variable at a time in a stable setting. When it comes to marketing, there are just too many variables to control. So, almost any relationship you observe in your data is a correlation.
There can be any number of variables contributing to the relationship between A and B:
- A and B could both be caused by an external variable, C
- A could be causing C, which is the cause of B
- A could be causing B, or B could be causing A
- A could be causing B, which in turn also causes A (this is called cyclical causation)
You’ve undoubtedly heard causal vocabulary thrown around in countless meetings (“this is happening because,” “X drives Y”, etc.). However, until you’ve eliminated all of the variables that are possibly contributing to the relationship and definitively pinpointed a causal relationship between a set of variables, assuming causality can result in critical missteps in strategy.
Marketing Data Analysis Mistakes: The Takeaway
The right data can optimize your marketing strategy, pointing to efficiencies that reduce costs and increase returns. As a marketer, knowing how to properly digest and interpret data is key to spotting trends. With these common mistakes in mind, you can sharpen your data analysis skills to uncover patterns that lead to actionable insights.
KEEP ROLLING: Why You Must Optimize Your Video Marketing Strategy