Our team at Machine Intelligence has guided clients building a variety of machine learning applications and forming new analytic capabilities. We’ve learned lessons to keep projects on track for business results. The top lessons:

1.     Precisely define the question to answer.

Statistical models answer tightly defined questions. Small changes to business goals can require different data and types of algorithms. Models are like a precision rifle with telescopic sights set for a specific range and wind. If conditions change, the cross hairs no longer predict the point of impact.

Consider customer loyalty, which summarizes a range of attitudes and behaviors. Loyalty may be highly correlated with churn or future sales, but propensity to buy is a different and more specific goal. Other examples include cross sell vs. up sell. We’ve seen senior leaders can want insight on a bigger picture, e.g. consumer behaviors, and so broadly frame questions. However, a project team needs a precisely defined question.

One core concept should be understood by everyone including non-technical people. Correlation and causation are completely different. To illustrate, if a hotel is always full when prices are highest, should a hotel always raise prices? Price correlates with occupancy, but price does not cause occupancy.

Defining the question should include context such as cost to operate, operational risk, team capacity and skills, and time required for a model to respond (latency). Watch for a tendency toward machine learning, and particularly deep learning, when simpler methods can suffice. Often, well-designed business intelligence reporting or classical statistical models are enough. Finally, while machine learning models can continually improve, they don’t learn automatically. Leaders should understand that retraining models requires a time commitment.

2.     Understand the “shape” of the data.

Artificial intelligence generates excitement, and project teams often want to jump into learning how algorithms work. Instead, methodically learn the contours of data and how it is generated.

Project teams need to really understand the customer behaviors, operating conditions, or other sources that produce data. Teams need to include customer service, field operations leaders, product managers and others who know how the data was produced. Check that simplifying assumptions like categories and time buckets for organizing data reflect the real world.

A good approach starts by:

1.     Visualizing distributions of all variables. Understand the business conditions that affect distributions.

2.     Identifying extreme values, anomalies, and clusters. Confirm the conditions that produced outliers or whether they indicate data quality issues.

3.     Seeing how changes in an independent variable affect the dependent variable.

Be professionally skeptical about data and prepared to discard sources if necessary. In an engagement to optimize placement of video ads, we found that historical data showed a near doubling several years ago in viewers from a particular ethnic group. The client was quick to confirm that changes to content produced the jump. However, we found that the real cause was a change to how Nielsen measured consumer demographics.

On a different engagement, we analyzed media sentiment about an insurance company. A UK insurer shares the same name, the company had named a concert/sports arena that appeared in many tweets, and people often refer to the company using a shortened version of its name that is also an adjective. Even with elaborate parsing rules, we couldn’t confidently use Twitter data.

3.     Plan most time and investment to build training data.

Training models may be the most fun part, but plan to spend the most time assembling or creating training data. Set expectations that half or more of project time will be applied carefully confirming, stitching together, and iterating data.

There is nothing magical about artificial intelligence. It’s simply code that trains a model to reproduce results similar to training data. If the training data is solid, then leaders should have confidence that models will produce perhaps 70-90% of the same result as the training data. The corollary is that leaders need confidence in the training data.

Drill into assumptions and “urban legends” about data. For example, one client told us that results from new deep learning models needed to be completely congruous with a prior approach because their clients’ compensation depended on the results. When tactfully inquiring, we found this happened only once several years ago. The real message was reservations about change.

4.     Train the whole business.

As analytics become increasingly important, data and models need to be managed as business assets rather than IT or marketing tools. Product managers, finance, marketing, and client service leaders should understand the data science development process.

If a project requires building new training data, then involve the whole business. Training data shouldn’t come from IT. Data labeling services such as Amazon Mechanical Turk work well for basic needs like identifying images, but labeling business events generally requires experience with a specific business’ customers and processes.

When categorizing customer behaviors, our experience is that often only a handful of key people are trusted. Consider categorizing messages into a customer contact center for mortgages. Some messages should be prioritized for immediate buying behaviors while others may be escalated for possible regulatory risk-and all categorization needs to be free from any form of bias. Project leaders need to set expectations early that key people should devote considerable time to labeling training data.

Product managers, not IT, should proactively cultivate training data sets. Product managers should understand the strengths of different types of algorithms and discuss these with data scientists and developers. Finance should understand the process and necessary investments. Some key points for business leaders:

• Machine learning models can reproduce about 70-90% of a training data set. That means 10-30% of results will be incorrect or at least differ from training data.
• Results should be evaluated in aggregate, not by cherry picking examples. If a business process requires correctly scoring every instance of incoming data, then take a more rules-driven approach that may lose precision but always returns directionally accurate results.
• Determine early if a model needs to be explainable not just for compliance, but to influence change within a company. This affects the choice of algorithm. For example, neural nets (deep learning) will capture nuances in data but are hard to explain compared to decision trees or regression.
• While business leaders generally understand that machine learning continually improves with more data, this doesn’t happen automatically. Set expectations that teams need to periodically apply time to retrain models.

5.     Fly by instruments.

Pilots refer to flying by instrument as setting aside subjective perceptions and trusting the numbers on their dashboards. Similarly, data science project teams need to listen to data over intuition.

Pursue big signals with an evident business explanation. Always involve people who understand real-world conditions behind a signal. If you sense the team is trying hard to find relationships in data, they are likely trying too hard.

Strong models result from carefully listening to a handful of strong variables, and generally not from creative new sources. For example, when modeling the performance of commercial engines, our data scientists considered a wide range of factors such as climate, detailed IoT telemetry, and customer usage patterns. However, facts showed that the best predictor of service events was prior service events.

On another project, a leading housewares retailer shared up to 1300 variables about each abandoned shopping cart on their web site. We were expected to find elaborate, subtle connections across consumer attributes such as disposable income or presence of children. In reality, abandoned shopping carts only indicated the general category of item a consumer wanted, e.g. outdoor furniture rather than a chaise lounge, and only held predictive value for a few weeks.

Finally, be prepared for null findings. For example, a team member worked with a technology company that operates a very large IT help desk. The client expected that mining patterns in support tickets would indicate underlying technical configuration issues. However, the data showed no relationships. Nearly all tickets reflected user error. A combination of humorous presentation and recommendations about redirecting budgets to training resulted in a satisfied client.

In conclusion, start data science projects by precisely defining the business question to answer. Fully understand the data including why and how it’s produced. Take a cautious approach to data quality. Plan to spend the most time on sourcing and building training data. Ensure the whole business understands the development process and contributes to training data sets. Trust the numbers as many intuitive data relationships will not be found in practice. Signals should be strong and clearly related to real world actions. Following these steps can reduce project risk and confidently deliver business results.