The British government is a keen advocate of big data and better use of analysis – both for delivering its own services more efficiently and effectively and as a way to improve business productivity.
A recent report from Parliament’s Science and Technology Committee estimated that British firms are only analysing about 12 per cent of all the data they’re creating. They predicted that better use of data by the remaining 88 per cent of firms could add as much as three per cent to UK GDP.
Individual companies can see an even bigger boost to the bottom line with improved productivity of more than 10 per cent.
The Committee looked at barriers to business adopting big data projects. The primary concern from business leaders, as with many aspects of enterprise technology, was the skills shortage.
The total number of graduates and postgraduate students in the UK has fallen from its peak in 2011.
Although there are a wide variety of courses available in the UK the Committee called for urgent government action to increase the number of people entering the workforce with relevant data skills.
The difficulty is that the successful data analyst requires a complex mix of technical and business skills. As software tools improve so the need for business sense and softer skills increases.
Graduates with good maths and statistics skills, which might be gained in various disciplines, have all gone into data careers – whether in market research, logistics planning or business intelligence.
The other area where government can play a role is in opening up datasets to the public and to business.
The UK has undergone a quiet revolution in this area and is now one of the world’s best governments at providing open data sources.
There are now well over 20,000 separate datasets published in machine-readable formats by the UK government.
A spin-off from the University of Southampton is helping farmers improve crop yields by bringing together open source weather data with information collected from Internet of Things sensors placed in fields where crops are growing. Bringing these datasets together allows better prediction of exactly when crops will be ready to harvest on a field by field basis.
Transport is another big provider of data to UK businesses. Transport for London provides data to over 5,000 developers working on a huge variety of big data projects.
There are still challenges for government which must balance openness with protecting the privacy of citizens.
The Science Committee’s investigation into the Big Data Dilemma found several examples of UK success stories which were showing the benefits of big data while still protecting individual privacy.
The Ministry of Justice is a pioneer in this regard not by offering data but rather collecting data from charities which work with offenders and then analysing it alongside re-offending rates in order to help those charities improve their service delivery.
The government is also working to break down the silos which separate the data held by corporations and private companies.
“Digital Catapult” programme aims to provide a safe platform for sharing this type of information while still protecting privacy.
There are several ways government can help companies embarking on their own big data projects. Firstly consider what data civil servants might already be collecting in forms which could be useful to add to your systems.
Secondly see what practical help programmes like Digital Catapult can offer in terms of advice and even infrastructure to help get your project off the ground.
The Science Committee continues to push the government to offer more data, especially more real-time, or near real-time, data sets and to work harder to break down data silos between and within government departments.