Big Data is changing the world for us all

Big Data is coming soon to have an impact on your life.  In fact, the use of big data is already changing the way you shop and the way things are sold to you.  Every time you go to buy a book on Amazon it will make recommendations about other books you might like.  It achieves this by crunching untold numbers of tiny bits of information, not only about you but more importantly about all the other people around the world who read the books you like.  There is a lot of big data out there.  Google Chief Executive, Eric Schmidt, says five Exabytes of information were generated from the dawn of civilisation up until 2003.  That same amount of information is now created every two days.  If that weren’t enough, one Exabyte is one billion gigabytes or 1,000,000,000,000,000,000 different pieces of information.

With that in mind, MK:Smart, the multi-agency project to analyse big data relating to transport, water and energy in Milton Keynes, is going to have to look at a lot of information.  We’re going to be ready to go in March, but we’re still working out which pieces of data are the right ones to use to provide answers to the questions at the heart of the project.  The clever bit is not just working out how to process it and relate different collections of data to each other but deciding what to keep and analyse and what to ignore.  Not that it’s that simple of course.  Data which is useful for improving traffic flow may be of no value to the study of water usage and vice versa.  Information can only be defined as junk in relation to a specific project and it might be absolutely pivotal to another.  Therefore, everything we collect we will need to store in such a way that it can be accessed when it does become relevant.

If this all sounds a bit esoteric it’s important to understand just how important big data can be.  The American McKinsey Global Institute produced a report about big data eighteen months ago.  It points to the experience of the German Federal Employment Agency.  In British terms this would be a combination of Job Centre Plus, the National Careers Service and a big chunk of the Department for Work and Pensions.  The Agency employs 120,000 people and it has embraced big data techniques to help it manage such a sprawling bureaucracy more efficiently.  The results have helped it save more than £8bn to date, a little less than the entire annual budget of the Home Office.  If MK:Smart can be anything like as successful the results for everyone who lives here will be transformational.

In such a new world there are opportunities and gaps to be filled.  From September 2014 UCMK plans to offer a new Bachelor of Science Honours Degree in Data Science.  The same McKinsey report I referred to earlier reckons on an immediate shortfall of up to 190,000 graduates in this field in the United States alone.  It’s an area where government, the corporate world and academia are going to be competing for the most talented people and there will be some very exciting and well-rewarded careers to be had.

Big Data will change the world, and it’s incredibly exciting and exhilarating to think that here at UCMK we’re going to be right at the heart of that change.