Big data could change the way we see the world. This week experts have gathered in Washington DC to discuss it, these are some of the examples that came up
• More data
Unless you work with it on a daily basis, big data is relatively unknown, both as a term and a concept. Even among industry experts its definition is the subject of intense debate. To some the key characteristic is a database's volume, to others it is its complexity, or even the speed at which data is gathered and analysed.
To you and I, big data is probably best understood as the enormous datasets held by corporations, governments and other large organisations whose activities affect millions of people. Big data is used to determine your recommended friends on Facebook, suggested purchases on Amazon and the point at which your mobile phone network offers you a freebie to keep you on side.
Some of these datasets, such as details of every one of a retailer's sales, will have a consistent format, with others stored as unstructured text. While it can be easy to dismiss big data as overly corporate, irrelevant outside the business world, it is becoming increasingly clear that it can be - and is being - a force for good in the wider world.
Below are some examples.
Tax fraud is a growing concern for governments looking to cut their deficits, and big data is being used to increase the efficiency of fraud detection processes. Where privacy law allows, government departments can now cross-reference tax databases with other information such as vehicle registrations and overseas travel data to find individuals whose spending patterns and tax contributions do not add up. While a suspicious result is not direct evidence of fraud and cannot be used to incriminate an individual, it can help officials better target their auditing and other checks and procedures.
Healthcare and medicine
The mapping of the human genome is widely regarded as one of the most significant advances in the history of medical research. The ability to map a person's genetic profile now allows doctors and scientists to predict a patient's susceptibility to certain diseases and other adverse conditions, and the major reductions in the time and cost of carrying out the procedure would not have been possible without big data.
Improvements in the speed and functionality of data collection, storage and analysis tools have lowered the cost of sequencing from almost £2bn to around £2,000 today, and cut the time it takes from over a decade to a week. While more incremental gains would have taken place at any rate, such major strides have only been made achievable by the cloud computing services offered by - among others - Microsoft, Amazon and Teradata.
The United Nations launched UN Global Pulse in 2011 with the primary aim of using best practices in the big data industry to enable the organisation to make faster and better informed responses to humanitarian crises. A project using social media analysis to track public concerns in Indonesia and the US has already showed the value such methods hold for crisis response and social science research as a whole.
There are a number of other organisations working towards similar goals, including DataKind, whose founder Jake Porway led a 'DataDive' earlier this year in London, where data scientists collaborated with charities to offer analysis-driven solutions to recurring problems.
US organisation aWhere is carrying out similar work, and in one of flagship projects uses data from satellite imagery to find and map pools of stagnant water in developing nations that could become breeding grounds for malaria-carrying mosquitos.
These datasets often contain detailed and potentially revealing information about us, giving rise to concerns over the implication for privacy, and this an issue taken seriously by firms working with big data.
US firm Teradata, for example, impresses upon its data scientists that simply obeying privacy laws is not enough - they should take Google's 'don't be evil' motto one step further, using their work to make a positive change.
This ideology was yesterday given substance when leading British data scientist Duncan Ross launched the 'Doing good with analytics' pledge. Signatories commit to taking account of the potential wider impacts of their analysis and to using their skills to help those who could benefit the most.
Full disclosure: I spent the last four days in Washington DC attending the Teradata Partners' Conference at the expense of Teradata Europe, Middle East and Asia.
NEW! Buy our book
• Facts are Sacred: the power of data (on Kindle)
More open data
Data journalism and data visualisations from the Guardian
World government data
• Search the world's government data with our gateway
Development and aid data
• Search the world's global development data with our gateway
Can you do something with this data?
• Flickr Please post your visualisations and mash-ups on our Flickr group
• Contact us at firstname.lastname@example.org
• Get the A-Z of data
• More at the Datastore directory
• Follow us on Twitter
• Like us on Facebook