Ann Vanderbijl, MA - Freelance Journalist : January 2014

In a big-data world, it takes an exponentially rising curve of statistics to bring home just how subjugated we now are to the data cruncher's powers. Each day, according to IBM, we collectively generate 2.5 quintillion bytes—a tsunami of structured and unstructured data that's growing, in IDC's reckoning, at 60 per cent a year. Walmart drags a million hourly retail transactions into a database that long ago passed 2.5 petabytes; Facebook processes 2.5 billion pieces of content and 500 terabytes of data each day; and Google, whose YouTube division alone gains 72 hours of new video every minute, accumulates 24 petabytes of data in a single day. No wonder the rock star of Silicon Valley is no longer the genius software engineer, but rather the analytically inclined, ever more venerated data scientist.
Certainly there are vast public benefits in the smart processing of these zetta- and yottabytes of previously unconstrained zeroes and ones. Low-cost genomics allows oncologists to target tumours ever more accurately using the algorithmic magic of personalised medicine; real-time Bayesian analysis lets counterintelligence forces identify the bad guys, or at least attempt to, in new data-mining approaches to fighting terrorism. And let's not forget the commercial advantages accruing to businesses that turn raw numbers into actionable information: according to the Economist Intelligence Unit, companies that use effective data analytics typically outperform their peers on stock markets by a factor of 250 per cent.
Yet as our lives are swept unstoppably into the data-driven world, such benefits are being denied to a fast-emerging data underclass. Any citizen lacking a basic understanding of, and at least minimal access to, the new algorithmic tools will increasingly be disadvantaged in vast areas of economic, political and social participation. The data disenfranchised will find it harder to establish personal creditworthiness or political influence; they will be discriminated against by stockmarkets and by social networks. We need to start seeing data literacy as a requisite, fundamental skill in a 21st-century democracy, and to campaign—and perhaps even to legislate—to protect the interests of those being left behind.
The data disenfranchised suffer in two main ways. First, they face systemic disadvantages in markets which are nominally open to all. Take stockmarkets. Any human traders today bold enough to compete against the algorithms of high-frequency and low-latency traders should be made aware how far the odds are stacked against them. As Andrei Kirilenko, the chief economist at the US Commodity Futures Trading Commission, along with researchers from Princeton and the University of Washington, found recently, the most aggressive high-frequency traders tend to make the greatest profits—suggesting that it would be wise for the small investor simply to leave the machines to it. It's no coincidence that power in a swathe of other sectors is accruing to those who control the algorithms—whether the Obama campaign's electoral "microtargeters" or the yield-raising strategists of data-fuelled precision agriculture.
Second, absolute power is accruing to a small number of data-superminers whose influence is matched only by their lack of accountability. Your identity is increasingly what the data oligopolists say it is: credit agencies, employers, prospective dates, even the US National Security Agency have a fixed view of you based on your online datastream as channeled via search engines, social networks and "influence" scoring sites, however inaccurate or outdated the results. And good luck trying to correct the errors or false impressions that are damaging your prospects: as disenfranchised users of services such as Instagram and Facebook have increasingly come to realise, it's up to them, not you, how your personal data shall be used. The customer may indeed be the product, but there should at least be a duty for such services clearly to inform and educate the customer about their lack of ownership in their digital output.
Data, as we know, is power—and as our personal metrics become ever easier to amass and store, that power needs rebalancing strongly towards us as individuals and citizens. We impeded medical progress by letting pharmaceutical companies selectively and on occasions misleadingly control the release of clinical-trials data. In the emerging yottabyte age, let's ensure the sovereignty of the people over the databases by holding to account those with the keys to the machine.

Google as our editor...

W. Daniel Hillis

Physicist, Computer Scientist, Chairman of Applied Minds, Inc.; author, The Pattern on the Stone

The Opinions Of Search Engines

Last year, Google made a fundamental change in the way that it searches. Previously a search for, say, "Museums of New York", would return web pages with sequences of letters that matched your search terms, like M-U-S-E-U-M. Now, besides the traditional keyword search, Google also performs a "semantic search" using a database of knowledge about the world. In this case, it will look for entities that it knows to be museums that are located within the geographic region that is named New York. To do this, the computers that perform the search must have some notion of what a museum is, what New York is, and how they are related. The computers must represent this knowledge and use it to make a judgment.
The search engine's judgments are based on knowledge of specific entities: places, organizations, songs, products, historical events, and even individual people. Sometimes these entities are displayed to the right of the results, which combine the findings from both methods of search. Google currently knows about hundreds of millions of specific entities. For comparison, the largest human-readable reference source, Wikipedia, has less than ten million entries. This is an early example of semantic search. Eventually, every major search engine will use similar methods. Semantics will displace the traditional keywords as the primary method of search.
A problem becomes apparent if we change the example from "Museums of New York" to "Provinces of China." Is Taiwan such a province? This is a controversial question. With semantic search either the computer or the curator of the knowledge will have to make a decision. Editors of published content have long made such judgments; now, the search engine makes these judgments in selecting its results. With sematic search these decisions are not based on statistics, but on a model of the world.
What about a search for "Dictators of the World"? Here the results, which include a list of famous dictators, are not just the judgment of whether a particular person is a dictator, but also an implied judgment, in the collection of individual examples, of the very concept of a dictator. By building knowledge of concepts like "dictator" into our shared means of discovering information, we are implicitly accepting a set of assumptions.
Search engines have long been judges of what is important; now they are also arbiters of the truth. Different search engines, or different collections of knowledge, may evolve to serve different constituencies - one for mainland China, another for Taiwan; one for the liberals, another for the conservatives. Or, more optimistically, search engines may evolve new ways to introduce us to unfamiliar points of view, challenging us to new perspectives. Either way, their invisible judgments will frame our awareness.
In the past, meaning was only in the minds of humans. Now, it is also in the minds of tools that bring us information. From now on, search engines will have an editorial point of view, and search results will reflect that viewpoint. We can no longer ignore the assumptions behind the results.