Data Alchemy

Turning Numbers into Gold

Posted by Marco Pineda on August 8, 2016

Never in all their history has humanity been blessed with so much information. By the end of 2016, global IP traffic will reach 1.1 ZB per year or 88.7 EB per month. This means that in the course of a day, the average person in a Western city is said to be exposed to as much data as someone in the 15th century would encounter in their entire life.

Data Science industry paragraph. Major innovations (e.g., Hadoop, Cassandra, HBase, Pig, Hive etc) have made data products easier to build. While the tools surrounding the industry have improved, data products are unique in that they are often difficult, complex and expensive for small teams with limited funds.

To begin, we define a Data Product as a product that facilitates an end goal through the use of data. The fundamental idea in Data Mining is that you shouldn't solve the whole problem at once. Solve a simple piece that shows you whether there is interest, then build the MVP. Before you begin you must answer some questions:

  1. Does anyone want or need your product?
  2. Does the customer care?
  3. Is there a market fit?
  4. How long do we have to answer questions

Below are some methods that should help anyone create a product out of data.

Use Product Design

The point is to have a conversation with your user...

One of the biggest challenges in Data Mining is getting the data in a useful form. One way to ensure clean data is to build a user interface that helps the user, and you, in the long run. To do this you can:

  • Support type-ahead
  • Prompt the user with "Did you mean...?"

The point is to have a conversation with your user rather than just a form. Engage the user to help you. This makes your analysis that much easier. Doing this solves two problems. 1) You're getting the user more involved and 2) you're getting clean data.

Make Winning Easy

Back when Amazon first began, pages contained product details, reviews, price and a button to buy the item. A user had no way to do comparison shopping. The user either went to the search box or left the site and went back to Google. Amazon needed to build pages that channeled users to other relevant products. They could have built a sophisticated recommendations engine but opted for a simpler system. They built 'collaborative filters' that added "People who viewed this product also viewed" to their pages. This was huge! Now, users can do product research without leaving the site and if they don't see what they want right away, Amazon channels them to another page.

Collaborative filters is a great example of starting with a simple product that becomes more complex, once you know it works. As you begin to scale, you have to track the data for all purchases, then build the data stores to hold the data, then a processing layer, then developing processes to update the data

Create a Product for the Real World

When you go into a store to buy something, you might look at prices and or reviews. Most likely you'll look at similar products near by. By adding a collaborative filter, Amazon built this experience into their web page. They crafted their digital product based off every day experiences centered around buying a product in the real world.

Post Sample Image To go places and do things that have never been done before – that’s what living is all about.

Lightweight testing by creating for a small section of the larger problem will give you the flexibility to add limitations and constraints based off real world problems.

Give Data Back to the User to Create Additional Value

Give data back to the user and you can create both revenue and engagement. Users are no longer the customer, they are the product. They are 'data generators' that either assist in ad targeting or are sold to the highest bidder, or both. Giving data back to the user shows them that you're on their side and engagement should increase. With added engagement, you can test and analyze data and gain new insights and continue this cycle until your engagement tapers off.

How do you give data back to the user? Some examples are LinkedIn's product called "Who's Viewed Your Profile." This product lists the people who've viewed your profile. The data is timely and actionable and most importantly addictive. Mint studies your credit card transactions to help you understand expenses and income and compare them to others in your demographic.

Placeholder text by Space Ipsum. Photographs by NASA on The Commons.