Some data is inherently urgent and some is not. Consider the situation where: if you don’t make a payment on your mortgage in the next two weeks then the bank will foreclose and you’ll be out on the street. So you write a check and put it in the mail. The check is data; data on paper. It will arrive in a couple of days, get processed and everything will be fine. It may feel as though that data is urgent, but in truth, it’s got plenty of time.
But if the last possible day to make the payment dawns and the payment has not yet been made, then the data really is urgent. You’ll probably drive the check to the offices of the mortgage company just to make sure it gets there on time.
We can classify some data, like that example, as “data with a deadline.” Most emails are not urgent data and don’t have a deadline. Most people expect them to arrive within a day and get responded to in a day or two. But in reality, they usually arrive within about a minute and then sit in an inbox for a while and then get responded to. There’s lots of acceptable latency in email communications. The data is not urgent.
Data You Can Wait For
We can define data urgency in the following way:
Data is urgent when it has to get to its destination in as usable form as fast as is physically possible.
So if data is up against a specific deadline, the data doesn’t become “urgent” until “we have to send it right now.” Just like the mortgage check, if there’s more than enough time to deliver it, it is not urgent.
There are two types of non-urgent data:
- Data that is up against a deadline.
- Data with no specific deadline
Aside from email, another example of the latter type is archived data. When we archive data we are not throwing it away, otherwise we would just delete it. We are recognizing that it no longer needs to be available “on-line.” Its urgency has decreased. There is a lot of data like this. It has some value, so we keep it, but it has very little urgency.
Most of the 5 or 6 exabytes of data that currently exist in digital form are non-urgent. In fact, much of that data is out there somewhere on the Internet. You can get to it reasonable quickly if you know its URL (or know how to find its URL).
Data You Have To Wait For
Let me revise our definition of data in the following way:
Data is urgent if it loses value while the receiver is waiting for it.
This says the same thing as before, but the definition differs slightly because it depicts latency as a cost. Some data is less valuable the later it arrives. An obvious example of this is news. Organizations like AP and Reuters make a business by being first with the news. You pay a premium for their up-to-date information. Old news is, by definition, not news at all. Who wants yesterdays papers? Nobody.
While there is no specific deadline for delivering an item of news, it is immensely urgent. If there’s a race, the first person to broadcast the news wins the race and everyone else with the same information comes in nowhere.
This might sound like an unusual situation, but actually it isn’t. We live in capitalist economy. Competition is fundamental to capitalism and it makes a huge contribution to efficiency. In most markets, the company that gets to market first with its products has an advantage. So the computer systems involved in bringing the product to market need to be as fast as they can. Urgency permeates the whole operation.
The cost of money is also a fundamental driver in a capitalist economy. But there’s a permanent tug-of-war there. All companies would like their receivables to arrive early and their payments to be made late. Payment data is urgent but only to the receiver.
If you want real-time urgency, think about the financial markets where stock and bonds and currencies and commodities are traded. Imagine that you have a bullet-proof algorithm which can recognize a move in the market – a price about to go up or down – and that the algorithm can also predict how far the price will move. Imagine too that your competitors have the same algorithm. Then the company that wins is the one with the fastest computer that can place the order fastest. Not only do they win, but they can buy exactly the right number of stocks or bonds or whatever to maximize the take. Everybody else comes last.
So that data is truly urgent. It’s either very valuable or it’s worth nothing, and the difference in value is determined entirely by speed.
The Ubiquity of Urgency
There’s nothing special about financial markets, except that what they trade is always a commodity of some kind. Most commercial activity is similar but the purchase transactions are more complex involving many more things, like delivery, payment terms, volume discounts and so on. There will always be a best time to buy and there will usually be competing buyers looking for the best deal. Wherever there is competition, wherever there is a queue, the data surrounding the activity will be urgent.
If you want your web site to respond directly to each visitor, you have to analyze user behavior and respond to it in real time. That’s a factor for every web business. The response of eretail web sites like Amazon, Dell and Best Buy are urgent, but so is the response of Google, Yahoo, Facebook and Twitter.
There can also be urgency in activities that are not “real-time” or even transactional. Launch a query against a huge petabyte database and it may take hours or even days to arrive at an answer. Now think about it. Petabytes of data cost a great deal of money to accommodate. You need a good deal of computer power to query such data and if it takes hours or days to get to an answer, you are tying up expensive resource for a long time. A single query will be very expensive and, almost every analysis activity you can envisage will require many queries.
So, although it might not seem to be the case when queries take hours to run, the answers to such queries have to be classified as urgent, if only because of the cost. But unless someone is being dumb, no-one would be assembling such a database if the answers to the queries didn’t have commercial value. And commercial value is always urgent. The time to cash it in is always now. So if you can cut down the query time on such a database by a clever use of parallelism, or a deploying some purpose designed database product, you will.
Ultimately, the urgency of data depends first upon how valuable it is when it is delivered. And if data is valuable when delivered, then delivering it even faster will likely increase its value.