A personal example of Big Data crunching

I saw Stephen Wofram's blog entry where he published analytics of his life for the last 33 years of emails, telephone calls, calendar entries etc.

http://blog.stephenwolfram.com/2012/03/the-personal-analytics-of-my-life/

At first, some people may wonder whether this has any benefit and what the data analytics are for, but I think it shows firstly the sort of data that can be graphed and (having always loved graphics more than text myself) the greater benefits are from seeing visually any long-term trends and allow the individual to decide whether to change some things that they do (emails on a Friday night, perhaps).

For the rest of us, it has information too.  For example, there have been discussions on when to email or tweet for maximum impact, with a large dataset we could see when users tend to be at email already and also when other emails aren't being sent, perhaps both can help show the most productive times.

We can see when meetings are set - knowing a target's norms allows you to fit in with them.

Changes in behaviour over time can also be shown - personally I'd love to know what percentage of emails I bother to open, what number I read on a mobile device and whether I do or don't download the images - I am sure the percentage of fully-read emails has reduced over time.

I think in the workplace one very useful piece of data could be the ratio between meetings and "non-meetings", I wonder sometimes how some people manage to achieve anything at all if they have 6 back-to-back meetings each day, as there's so little time to actually perform the actions agreed.

So, though reading his blog may at first make you wonder whether it is useful, I think its a great indication of what can be gleaned and if we multiply that data by every individual in an organisation, it can show the best time for internal meetings, the best way of communicating, the types and methods of communication being used - first get the data, then analyse it, then look for patterns and make the difficult jump between facts, data and information.

Of course, he has the benefit of using the same systems for many years - for most of us bouncing between jobs and various email addresses, phones etc. we probably don't have the data itself.  So, step one is to make sure the data is being tracked and archived, even if we can't work out how to extract the value today, that may come in months of year's in the future.

Someone once said, the best way to find a needle in a haystack is to remove all the hay and what you are left with is the needle.  Step two is then crunching the numbers and looking for the patterns that are useful.

Call me a geek, but I think its rather fascinating.