Machine Learning: A Brief Intro

Do you remember the first time you saw a commercial about “the Cloud?” That was one of the pivotal moments for technology buzzwords going mainstream. It’s been a nonstop thrill ride since then: Web 2.0. Internet of Things. Big Data. Machine Learning.

Like “the Cloud,” the term “machine learning” is thrown around a lot, but it’s not entirely clear who it is useful to. People who follow it are aware that machine learning techniques were used by Google to create an unstoppable Go playing machine, and that it allows computers to drive with abilities getting closer to human drivers by the day.

What is machine learning for? At its essence, machine learning is a useful addition to the toolkit of a data scientist or data analyst. Anything that you would use statistical analysis for is a good candidate to look at from a machine learning perspective.  

“How can I group my users into market segments?”

“What is the expected worth of this property I’m purchasing?”

“Which of my 500 stores is an outlier for productivity?”

Machine learning can provide insight to help answer these types of questions. It is powerful, but it is not magic. It will help you solve incredibly complex yet specific problems, but it’s not going to solve poorly defined general problems.

Why is machine learning useful?

So then, machine learning techniques solve problems similar to the problems that are solved by statistical analysis. What makes machine learning algorithms worth looking into?  

The first thing to consider is the scale of data. Other forms of analysis look at a small number of variables, oftentimes only two. Machine learning can take in a massive number of variables and automatically analyze and build a model using all of these variables. These large sets of variables are generally called “feature vectors.” Machine learning algorithms also excel when the amount of data coming in is very large.

Rich data sets can yield surprising relationships between feature vectors, and models built through machine learning are capable of identifying these relationships easily. A simple case is estimating the value of a home based on the square footage of the home. Using simple statistical techniques, you can identify a relationship and use square footage to estimate value.

Where machine learning really shines is when you want to estimate the value using not just square footage, but also variables like city population, local crime rates, average temperature, driving distance to the closest Whole Foods, how difficult it is to pronounce the name of the road for an English speaker, and so on. Unseen relationships lie in all of these data points, and machine learning algorithms can identify these without a human having to perform additional analysis.

It is also possible that the patterns in the data and relationships will change. Because machine learning algorithms can be retrained relatively easily or even continuously, they can work well when patterns change over time. A model tuned to summer sales data might be great until October, but going forward you would want to include new incoming data. This applies not just to data, but also to the features you have access to. If you’re breaking into the market of a new country with your product, a model can be retrained using additional features that are accessible there but not in your current markets.

Lastly, Machine Learning is as useful as it has ever been right now because we can now deploy at scale. Cloud based infrastructure and the significant development in supporting frameworks and libraries available is making application of machine learning feasible for more companies and organisations than even a few years ago.

What are the downsides?

Machine learning is the new tech on the block, and it’s hard to get right. Even doing a straightforward classification analysis requires reviewing many different algorithms to get the right one. Once that’s done, there’s still the possibility that the generated model is tied up with the data used to train it, and doesn’t reflect the real world as closely as you think. This is called overfitting the data, which has to be deliberately engineered out.

Machine learning is not routine software development. Most of the work done for machine learning will more closely resemble research. Processes and skill sets used for software development will be misleading for machine learning projects. Estimating the time to get a valuable answer from the data is even more difficult than normal software development estimation.

Machine learning tools are also very new. This means that there will be more bugs and fewer resources on how to properly use the tools. Not only that, but finding the right tool for the job can be very difficult. Learning a new tool for a specific problem will be difficult. To see an example of the complexity, take a look at this TensorFlow reference document discussing doing classification using neural networks: https://www.tensorflow.org/api_docs/python/nn/classification.

To sum up, machine learning is incredibly powerful but incredibly difficult to get right. If you have a difficult analytical question with a potentially valuable answer, machine learning is something you’ll want to look into.