Four Misassumptions That Cause Analytic Projects to Fail (Before They Even Start)

Businesses do not get enough value from analytic work. Indeed, fewer than 20% of analytic projects provide business value! However, as we have covered before, we cannot blame a lack of talent or a lack of technical resources for the common failures. The problem is how analytic and business professionals talk (or don’t talk) to each other.

Analysts get requests every day. Often, these are quick questions with very little context. Such as these::

1. Can you tell me how sales are going for our new product?
2. Can you determine if the new marketing campaign is making a difference?
3. Can you predict the likelihood an employee will quit in the next year?

Upon hearing such requests, analysts spring to action. Their personalities and training encourage them to problem-solve, provide answers, and get to work! They “hear” the request through their own understanding of available data, tools, and statistical methods. Most likely, the requestor will get results in a timely fashion.

And most likely, it won’t answer their question.

Improving the value of analytic work means investing more time and effort up front. Because the request seems straightforward in the analysts' minds, they presume they understand what the requestor means. But often they don’t.

Part of the reason analysts misunderstand requests is the fast pace of work we all experience. It’s not uncommon to rush and skip ahead without clarification. It happens to everyone. 

But in the case of analytics, there are four other critical, mistaken assumptions that analysts make routinely. Being aware of these will encourage us to slow down and get clear.

 Misassumption 1:  The requestor knows—and has accurately stated—exactly how the answers will be used.

While the request certainly has something to do with their needs, it’s rare that a simple request covers the details required to answer their specific question. They likely haven’t had time to think through all aspects of their question. In the questions above, the requestors probably have an idea what “how sales are going” or “making a difference” mean. But analysts don’t.

In business, almost all analytic questions entail something the person wants to know BECAUSE they want to 1) make a decision or 2) take an action. What are those decisions/actions? For example:

Question 1 could imply they want to compare sales of the new product to sales of existing products to decide its future viability. Or it could imply that they want to compare the effectiveness of different salespeople in selling the new product to decide whether to fire one of them. 

Question 3 could be a request about what factors predict turnover so they can decide to do manager training. Or it could be an actual prediction of which individuals are likely to quit so they can intervene to encourage them to stay.

In my experience, other than repeat reports, previously defined by the requestor and the analyst, new requests will not be clear the first time.

 Misassumption 2:  The requestor knows HOW to ask for what they need.

Non-analysts often use similar terminology as analysts, but do not mean the same thing. When a businessperson wants to know if something is “better,” “more favorable,” “delivers an ROI,” or “is predictive.” they likely mean something less technical than would be interpreted by an analyst. 

Given those words, analysts may apply tests of statistical significance or design a complex research study that far exceeds the rigor required for the business decision. That’s because analysts’ training specifies the conditions required to make definitive statements. So, they may over-complicate a simple question because certain words imply a level of scientific accuracy in the data world.

Businesspeople will articulate their questions based on their current priorities. For example: 

Question 2 asks whether a new marketing effort is “making a difference.” Imagine that the business wants to fail fast, the modern practice of abandoning bad ideas quickly to minimize money spent unnecessarily. On the other hand, analysts will focus on making a valid conclusion—which requires a sufficient duration and sample size. They will be reluctant to conclude with certainty (p<.05!) that the marketing campaign is underperforming until enough time has passed.  

Without a conversation about what “making a difference” means, each side will revert to their own meaning. In this case, the analyst’s natural bias will prevent the business from achieving it’s fail-fast goal. Although both sides are trying to do their jobs well, they can unwittingly end up working in opposition.

Misassumption 3:  the requestor has known, defined criteria for making their decision.

One often-overlooked aspect of analytic projects is understanding the criteria the business will use to make their decision. If they want to know if a new product is selling better than older products to decide whether to continue offering the product, how will they know it is better enough?

This is an important question because the analyst may choose a different method for a different criterion. If the business only wants to know it is at least as good, the analyst can choose a method that assesses a yes/no of whether it is selling as well across salespeople or regions. If the business needs it to be 27% better to justify higher production costs, that will require a different approach. 

Quite often, the requestor has not thought about it, assuming that they will “know it when they see it.” By discussing it in advance, the analyst learns more about how the information will be used. Plus, it can help identify—ahead of time—if the business requirements are unrealistic given the available data.

 Misassumption 4: the requestor is the primary audience for the answer.

The reason for an analysis, and the criteria for a decision, are not the only factors that will shape it. How an analysis is performed and how results are packaged will also be affected by the audience for whom it is created.

A CEO will want something different from entry-level employees. A marketing group will respond differently to results, and have vastly different questions, than an academic group. The engineers evaluating a product they created will want different information than the sales team that sells it.

It is not uncommon for analysts to find out—after the fact—that their results were used in an entirely different way, by an entirely different audience than for whom they intended. (e.g. Early in my career, much to my dismay, a quick and dirty correlation I shared in an internal memo found its way to a conference and then to a reporter for the New York Times).  

It’s worth asking how, and by whom, the results are likely to be used.

 Listen well and ask questions!

Learning how to ask a sequence of generative questions to help the requestor clarify their thinking (for themselves and analysts) dramatically improves the chances of answering the right question, in the right way, for the right audience.

Unsure if you need an analytic translator or want to know more about what analytic translation is? Don't hesitate to get in touch with me.