So it’s done. The death knell for unlimited mobile data seems to have been rung following the announcement that AT&T in the US and O2 in the UK will scrap unlimited data plans for new subscribers.

The industry has created something of a perfect storm for mobile data. After years of being left in the wilderness, a combination of cheap price plans, heavily subsidized smartphones and compelling apps and services, has seen mobile data usage explode to a level that is now causing concern among industry executives keen to balance quality of service and profitability.

In the early days of data, pricing products and services to lower the barriers to entry made sense. Mobile operators needed to capitalize on their 3G investments and voice revenues were stagnating. The launch of the Apple iPhone in 2008 delivered mobile data to the masses in the form of compelling applications and the industry responded.

In the following years, the availability of cheap smartphones and products such as mobile broadband dongles, air cards and mifi units came thick and fast. In Western Europe and North America, half of all new handsets sold by the end of 2011 will be smartphones and it’s expected that that over the next five years there will be a 20 fold increase in data traffic. Central to this success was the availability of unlimited data plans. The ‘all-you-can-eat’ pricing model finally removed the single biggest barrier to mobile data, the threat of bill shock.

It seems, however, that the level of consumer demand for mobile data has surprised even the largest operators. AT&T and O2, both beneficiaries of  the iPhone’s success thanks to early exclusivity deals, have made very public statements over the last year regarding the ‘data crunch’.

Both operators have experienced quality of service issues from the growing population of smartphones attaching to their networks. The CEOs of both companies have made public statements about the data ‘crunch’; the CEO of O2 going as far as making a public apology to his London customers where the huge popularity of uber-cool iPhones, and a highly dense urban environment was only ever going to end one way.

Much of the problem comes from the phones themselves, and not necessarily from the amount of data being downloaded. Instead it’s how today’s smartphones manage ‘signaling’.   Many other connected devices, such as laptops and netbooks, maintain an open channel to the data network. Smartphones just don’t have the juice to maintain this connection and instead many are built to drop the data connection as soon as requested data is received. However, many smartphones, more precisely their applications, routinely signal the network for status updates. Nokia Siemens Networks estimates that each update can generate up to 20 signaling messages. With some smartphones repeatedly awakening themselves to check for updates every two minutes, the signaling traffic from a single device in a day is comparable to 1,000 voice calls.

Strictly speaking, changes to price plans won’t have an immediate and direct impact on the signaling burden; that’s a handset and application architecture issue. What it will do is allow operators to better control usage and meter it for the benefit of all subscribers, rather than the minority that represent the majority of traffic.

Suggestions that change was needed came from AT&T CEO, Ralph de la Vega last year. He told the Wall Street Journal. “Many customers don’t know how much bandwidth they’re consuming. When AT&T conducted a broadband test, customers often reduced their data use. Longer-term a pricing scheme based on usage is likely, though it will be determined by industry competition and regulatory guidelines.”

This week (following the iPhone 4 launch) that change happened and a few days later it was followed by O2. Both have introduced tiered pricing, with a capped limit depending on the price plan. The argument from both is that in reality, even with unlimited data plans, consumers get nowhere near the fair-use policy applied to unlimited plans. O2’s cheapest contract plan will be 500mb p/month plan (£25). The operator says that 500mb is actually two and half times the average O2 consumer’s current use. In practice this means that consumers should be better off financially (the cheapest monthly data tariff is currently £30). The operator promises it will be able to deliver a more consistent experience and increase the quality of service for the subscriber base as a whole. Forecasting data consumption will also be more accurate for the operator, in turn helping to grow the thin margins that face today’s competitive mobile data markets.

AT&T consumers will be faced with a choice between DataPlus, a $15 data plan capped at 200mb, or DataPro, a $25 plan capped at 2GB. Overage charges will be $15 for a further 200mb or $10 for a further 1GB on the respective plans. Both plans provide a saving of $5-$15 when compared to the previous unlimited plan. AT&T argues that this effectively makes data even cheaper for the majority of users.

By introducing bands of usage the operator can impact consumer behavior in data consumption. When consumers are confronted with the choice, many will instinctively look to manage their data consumption to fall in line with the lowest tier, actively looking to restrict their data usage to reap greater savings. AT&T plan to encourage better management of data consumption with regular (and free) text and email notifications to alert subscribers surpassing 65%, 90% and 100% of their data allowance. The intention is that this change in behavior will decrease overall data traffic suffered by the network and provide a higher QoS for consumers.

However, the question remains of whether consumers actually know what a megabyte of data gets them. If they don’t, and they start to ration their data usage for fear of surpassing their data cap, are we in danger of stifling the growing market for data applications and services?  Tiered pricing also doesn’t remove the signaling burden.

Ultimately, with the spectrum crisis looming, data has become a commodity in progressively short supply. Allowing subscribers to consume excessive levels of data at no extra cost is unrealistic and unfair to the majority of consumers who subsequently suffer poor QoS as a result. Others would argue that the operators have brought this upon themselves and if they offer unlimited data then a consumer should not be penalized for taking full advantage of that. And that if their network infrastructure isn’t up to scratch then fix it.

Conspiracy theorists among you will have probably noted that both AT&T and O2 have strong links with Apple and that the removal of unlimited data plans coincides neatly with the launch of the iPhone 4. The iPhone is notorious for fueling data consumption, and both operators will want to ensure that their networks are not placed under undue strain as existing iPhone owners upgrade and leverage increasingly data hungry applications such as FaceTime.

Like it or not, the days of unlimited data are numbered. The data goldrush is over. Operators have spent millions acquiring customers, it’s now time that they start focusing on maintaining profitability. At current rates, despite a predicted 20 fold increase in data traffic over the next five years, operators only stand to see a two fold increase in associated revenue. The goal, therefore, has to be lowering the cost of delivering a megabyte of data. Tiered pricing won’t do this directly, but in time, if operators can better educate consumers on data consumption then it’ll certainly help in the creation of a more ordered and metered ecosystem.

This post has been cross-posted on the WIF blog.