Email's format is rigorously defined in IETF RFC-5322 (Request For Comments by the Internet Engineering Taskforce). To view a raw email in Gmail, select a message and then select the 'Show original' option in the top-right drop-down menu.

Figure 2.1. Gmail Show original

A raw email looks like this:

From: Russell Jurney <russell.jurney@gmail.com>
Mime-Version: 1.0 (1.0)
Date: Mon, 28 Nov 2011 14:57:38 -0800
Delivered-To: russell.jurney@gmail.com
Message-ID: <4484555894252760987@unknownmsgid>
Subject: Re: Lawn
To: ****** Jurney <******@hotmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Dad, get a sack of Rye grass seed and plant it over there now.  It
will build up a nice turf over the winter, then die off when it warms
up.  Making for good topsoil you can plant regular grass in.

Will keep the weeds from taking over.

Russell Jurney datasyndrome.com

This is called semi-structured data.

Wikipedia defines semi-structured data as:

This is in contrast to structured data, which breaks data up into rigorously defined schemas before analytics begin, for more efficient querying therafter. A structured view of email is demonstrated in the Berkeley enron dataset by Andrew Fiore and Jeff Heer:

Figure 2.2. Enron email schema

Although we can work with semi-structured data as pure text, it is still helpful to impose some kind of structure to the raw records using a schema. Serialization systems give us this functionality. Available serialization systems include Thrift, Protobuf and Avro.

Although it is the least mature of these options, we choose Avro. Avro allows complex data structures, it includes a schema with each file, and it has support in Apache Pig. Installing Avro is easy, and it requires no external service to run.

We'll define a single, simple Avro schema for an email document as defined in RFC-5322. It is well and good to define a schema up front, but in practice... much processing will be required to extract all the entities in that schema. And so our initial schema might look very simple, like this:

  {
      "type":"record",
      "name":"RawEmail",
      "fields":
      [
          {
              "name":"thread_id",
              "type":["string", "null"],
              "doc":""
          },
          {
              "name":"raw_email",
              "type": ["string", "null"]
          }
      ]
  }
      

We might extract only a thread_id as a unique identifier, and then store the entire raw email string in a field on its own. If a unique identifier is not easy to extract from raw records, we can generate a UUID (Universally Unique IDentifier) and add it as a field.

Our job as we process data then is to add fields to our schema as we extract them, all the while retaining the raw data in its own field if we can. We can always go back to the mother sauce.

As Pete Warden notes in his talk, Embracing the Chaos of Data, most freely available data is crude and unstructured. It is the availability of huge volumes of such ugly data, and not carefully cleaned and normalized tables, that make it 'big data.' Therein lies the opportunity in mining crude data into refined information, and using that information to drive new kinds of actions.

Extracted features from unstructured data only get cleaned in the harsh light of day, as users consume them and complain, if you can't ship your features as you extract them, you're in a state of free-fall. The hardest part of building data products is pegging entity and feature extraction to products smaller than your ultimate vision. This is why schemas must start as blobs of unstructured text and evolve into structured data only as features are extracted.

Features must be exposed in some product form as they are created... or they will never achieve a product-ready state. Derived data that lives in the basement of your product is unlikely to shape up. It is better to create entity pages to bring entities up to a 'consumer-grade' form, to incrementally improve these entities and to progressively combine them than to try and expose myriad derived data in a grand vision from the get-go.

While mining data into well structured information, using that information to expose new facts and make predictions that enable actions offers enormous potential for value creation... data is brutal and unforgiving, and failing to mind its true nature will dash the dreams of the most ambitious product manager.

As we'll see throughout the book - schemas evolve and improve, and so do features that expose them. When they evolve concurrently, we are truly agile.

We'll be working with semi-structured data in data-pipelines to extract and display different features of the data. The advantage of working with data in this way is that we don't invest time in extracting structure unless it is of interest and use to us. Thus, in the principals of KISS (Keep It Simple, Stupid!) and YAGNI (You Ain't Gonna Need It), we defer this overhead until the time of need. Our toolset helps make this more efficient, as we'll see in chapter 3.

A data pipeline to calculate the number of emails sent between two email addresses looks like this:

Figure 2.3. Example dataflow to count the number of emails between two email addresses

While this data-flow may look complex now if you're used to SQL, we'll quickly get used to working this way and such a simple flow will become second-nature.

A social network is a group of persons (egos) and the connections or links between them. These connections may be directed, as in 'Bob knows Sara.' Or they may be undirected, 'Bob and Sara are friends.' Connections may also have a connection strength, or weight. 'Bob knows Sara well,' (0.5 out of 0-1) or 'Bob and Sara are married' (1.0 out of 0-1).

The sender and recipients of an email via the from, to, cc and bcc fields can be used to create a social network. For instance, the email above defines two entities, russell.jurney@gmail.com and ******@hotmail.com.

        From: Russell Jurney <russell.jurney@gmail.com>
        To: ******* Jurney <******@hotmail.com>
      

The message itself implies a link between them. We can represent this as a simple social network.

Figure 2.4. Social network dyad

A more complex social network might look like this:

Figure 2.5. Social network

A social network of some 200 megabytes of emails from Enron looks like this:

Figure 2.6. Enron corpus viewer, by Jeffrey Heer

Social network analysis, or SNA, is the scientific study and analysis of social networks. By modeling our inbox as a social network, we can draw on the methods of social network analysis to reach a deeper understanding of the data, and of our interpersonal network.

A time series is a sequence of data points ordered by a timestamp recorded with each value. Time series allow us to see changes and trends in data over time. All emails have timestamps, and so we can represent a series of emails as a time series.

        Date: Mon, 28 Nov 2011 14:57:38 -0800
      

Looking at several other emails, we can plot the raw data in a time series:

Figure 2.7. Raw time series

Since we aren't looking at another value associated with the time series, we can see the data more clearly by bucketing the data by day. This will tell us how many emails were sent between these two addresses per day.

Figure 2.8. Grouped time series

Time series analysis might tell us when we most often receive email from a particular person or even what their work schedule is.

The meat of an email is the text message it contains. Despite the addition of MIME for multimedia attachments, email is still primarily text.

        Subject: Re: Lawn
        Content-Type: text/plain; charset=ISO-8859-1

        Dad, get a sack of Rye grass seed and plant it over there now.  It
        will build up a nice turf over the winter, then die off when it warms
        up.  Making for good topsoil you can plant regular grass in.

        Will keep the weeds from taking over.

        Russell Jurney
        twitter.com/rjurney
        russell.jurney@gmail.com
        datasyndrome.com
      

We might analyze the body of the email by counting the word frequency within it. Once we remove non-coding common 'stop words,' this looks like:

Figure 2.9. Email body word frequency

We might use this word frequency to infer the topics of the email are 'plant' and 'grass', as these are the most frequent words in the email. Processing natural language in this way helps us to extract properties from semi-structured data to make it more structured. This enables us to incorporate these structured properties into our analysis.

A fun way to show word frequency is via a wordle:

Figure 2.10. Email body wordle

In probability theory we model seemingly random processes by using inputs to create probability distributions. We can then employ these probability distributions to make suggestions and to classify inputs.

We can use probability distributions to make predictions. For instance, we might create a probability distribution for our sent emails. Given that our email address appears in the from field, and another email address appears in the to field, what is the chance that another email will appear?

In this case, our raw data is the to, from, cc and bcc fields from each email:

        From: Russell Jurney <russell.jurney@gmail.com>
        To: ****** Jurney <******@hotmail.com>
        Cc: **** Jurney <****@hotmail.com>
        Bcc: Russell Jurney <russell.jurney@gmail.com>
      

First, we count the pairs of from email addresses with each other field:

Dividing these values by the total number of emails gives us a probability distribution characterizing the odds that any given email from our email address will be to any given email.

Finally, we list the probabilities for a pair recipients co-occurring, given that the first address appears in an email.

We can then use this data to show who else is likely to appear in an email, given a single address. This data can be used to drive features like Gmail's suggested recipients feature.

Figure 2.11. Gmail suggested recipients

We'll see later how we can use Bayesian inference to make reasonable suggestions for recipients, even when Table 2.3 is incomplete.

As we've seen, viewing semi-structured data according to different algorithms, structures and perspectives informs feature development more than normalizing and viewing it in structured tables does. We'll be using the perspectives defined above to create features throughout the book, as we climb the data-value stack.