login    register    help     

About Us
Ash Center
Innovation Awards
Innovators Insights Newsletter
All Topics
Criminal Justice and Public Safety
Economic and Community Development
Education and Training
Environment & Natural Resources
Governance and Politics
Health and Social Services
  Access & Insurance
  Gender, Race & Ethnicity Issues
  Health Care Administration
  Long-Term Care
  Medical Education & Research
  Mental Health Services
  Primary Care
  Public Assistance
  Public Health
  Services to Specific Populations
  Substance Abuse Treatment & Prevention
  Violence & Abuse
Organizational Management
Public Infrastructure
All Themes
*Access to Services & Social Justice
*Government Performance & Management
*Government, Civil Society, & Private Sector Partnerships
*Justice & Security
*Pluralism, Diversity, Gender, & Inequality
*Social Services
*Sustainable Development

Big data: what is it and how can it help?

John Burn-Murdoch
The Guardian
October 26, 2012
I hate it   I love it





Big data could change the way we see the world. This week experts have gathered in Washington DC to discuss it, these are some of the examples that came up
More data

Unless you work with it on a daily basis, big data is relatively unknown, both as a term and a concept. Even among industry experts its definition is the subject of intense debate. To some the key characteristic is a database's volume, to others it is its complexity, or even the speed at which data is gathered and analysed.

To you and I, big data is probably best understood as the enormous datasets held by corporations, governments and other large organisations whose activities affect millions of people. Big data is used to determine your recommended friends on Facebook, suggested purchases on Amazon and the point at which your mobile phone network offers you a freebie to keep you on side.

Some of these datasets, such as details of every one of a retailer's sales, will have a consistent format, with others stored as unstructured text. While it can be easy to dismiss big data as overly corporate, irrelevant outside the business world, it is becoming increasingly clear that it can be - and is being - a force for good in the wider world.

Below are some examples.

Fraud detection

Tax fraud is a growing concern for governments looking to cut their deficits, and big data is being used to increase the efficiency of fraud detection processes. Where privacy law allows, government departments can now cross-reference tax databases with other information such as vehicle registrations and overseas travel data to find individuals whose spending patterns and tax contributions do not add up. While a suspicious result is not direct evidence of fraud and cannot be used to incriminate an individual, it can help officials better target their auditing and other checks and procedures.

Healthcare and medicine

The mapping of the human genome is widely regarded as one of the most significant advances in the history of medical research. The ability to map a person's genetic profile now allows doctors and scientists to predict a patient's susceptibility to certain diseases and other adverse conditions, and the major reductions in the time and cost of carrying out the procedure would not have been possible without big data.

Improvements in the speed and functionality of data collection, storage and analysis tools have lowered the cost of sequencing from almost £2bn to around £2,000 today, and cut the time it takes from over a decade to a week. While more incremental gains would have taken place at any rate, such major strides have only been made achievable by the cloud computing services offered by - among others - Microsoft, Amazon and Teradata.

Humanitarian efforts

The United Nations launched UN Global Pulse in 2011 with the primary aim of using best practices in the big data industry to enable the organisation to make faster and better informed responses to humanitarian crises. A project using social media analysis to track public concerns in Indonesia and the US has already showed the value such methods hold for crisis response and social science research as a whole.

There are a number of other organisations working towards similar goals, including DataKind, whose founder Jake Porway led a 'DataDive' earlier this year in London, where data scientists collaborated with charities to offer analysis-driven solutions to recurring problems.

US organisation aWhere is carrying out similar work, and in one of flagship projects uses data from satellite imagery to find and map pools of stagnant water in developing nations that could become breeding grounds for malaria-carrying mosquitos.


These datasets often contain detailed and potentially revealing information about us, giving rise to concerns over the implication for privacy, and this an issue taken seriously by firms working with big data.

US firm Teradata, for example, impresses upon its data scientists that simply obeying privacy laws is not enough - they should take Google's 'don't be evil' motto one step further, using their work to make a positive change.

This ideology was yesterday given substance when leading British data scientist Duncan Ross launched the 'Doing good with analytics' pledge. Signatories commit to taking account of the potential wider impacts of their analysis and to using their skills to help those who could benefit the most.

Full disclosure: I spent the last four days in Washington DC attending the Teradata Partners' Conference at the expense of Teradata Europe, Middle East and Asia.

NEW! Buy our book

• Facts are Sacred: the power of data (on Kindle)

More open data

Data journalism and data visualisations from the Guardian

World government data

Search the world's government data with our gateway

Development and aid data

Search the world's global development data with our gateway

Can you do something with this data?

Flickr Please post your visualisations and mash-ups on our Flickr group
• Contact us at data@guardian.co.uk

Get the A-Z of data
More at the Datastore directory

Follow us on Twitter
Like us on Facebook

guardian.co.uk © 2012 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

Visit source to read full article


© 2014 by Harvard University Kennedy School of Government. All Rights Reserved.
About Us | Privacy Policy | Partners | Contact Us | Provide Feedback | Advanced Search

ksg Version:   Host: ip-10-83-51-59  C3_DB=c3@localhost:3306; GEO_DB=plex-sandbox@localhost; KPLEX_DB=ksg@localhost:3306; SESSION_DB=session@localhost:3306;