« Manage virtualized sprawl with VRMs | Main | Big Data on Grids or on Clouds? »

October 28, 2009

Need for change in your IT infrastructure

Companies earnings outstrip forecasts, consumer confidence is retuning and city bonuses are back. What does this mean for business? Growth! After the recent years of cost cutting in IT budgets, there is the sudden fear induced from increased demand. Pre-existing trouble points in IT infrastructures that have lain dormant will suddenly be exposed. Monthly reporting and real time analytics will suffer as data grows. IT departments across the land will be crying out “The engine canna take no more captain”. What can be done?

What we need is a scalable system that grows with the business. A system that can handle sudden increases in data growth without falling over. There are two core principles to a scalable system (1) Users experience constant QoS as demand grows (2) System Architects can grow system capacity proportionally with the available resources. In other words, if demand increases twofold, it is “enough” to purchase twice the hardware.

This is linear growth. Is it enough to satisfy CEOs IT demands? Generally, the achievement of linear growth is favorable. In computational science, software problems are characterized by their complexity denoted as O(n). Trivial problems to solve are O(n). An example, is the simple summing of employee salaries. With increasing problem complexity we can observe O(n2), O(n3), or O(2(n)). How does this relate to ensuring I get the same performance as my data grows? Well, an O(n3) problem means that if the number of database records grows 3x, the amount of hardware needs to grow 27x to maintain QoS.

Computational complexity can be minimized using code level techniques, especially if the code is badly written.  However, the fact is that with certain problems, their complexity will never be lowered. For instance, NP-complete problems require non polynomial computation time. In other words, only small datasets can be processed due to their massive computation times - optimization of cargo storage is one such example.

However, if badly implemented, the  simple summing of employee salaries can become NP-complete! What can be learnt? Consider your-self lucky if the problem grows linearly.

So, are we cursed to buying X times hardware each time our domain grows X items? No. Polynomials have many parameters (y=a+ b*x + c * x*x…). “a” indicates the starting barrier. “b, c” indicate the steepness of the curve. For small sets, it is possible that the problem of a cubic complexity grows slower than linear. For large sets, it is impossible.

Need for change

The spiraling growth in data can be anticipated and planned for. The need can be predicted based on (1) past experience, (2) known amount of data, eg by market impact, (3) complexity calculations. Take Facebook, which has 850 million photos and 7 million videos that need storing per month. To cope with exponential data increase, expenditure on its data centers has risen from $30 million, 2007 to $100 billion, 2009.

Facebook's rapid expansion was only achieved with a clear and forward thinking IT strategy. Successful IT planning needs (1) responsible people (2) brave decision makers. A culture of responsibility needs to be instilled within businesses. Underlying problems of the IT infrastructure must not be hidden from upper management. Decision makers need to know about problems. Only then, can pro-active decisions be made that guarantee an agile response to growth. IT cannot be allowed to block business.

Often, warning signs are ignored and the need for change comes too late: “We need our new system tomorrow”. Tomorrows systems only add up to more steel. This is better known as “save your ass tactics” - a quick fix with poor ROI.

In most business, the need for change can be predicted. We have instruments for it and the data is available. Yet, many business only take action when it is too late. There is no blanket solution, but there are methods. My colleagues will be talking about the issues this Thursday, 29th October, 2009, at the Price/performance of your ORACLE infrastructure, Krakow, Poland.

Bookmark and Share


TrackBack URL for this entry:

Listed below are links to weblogs that reference Need for change in your IT infrastructure:


Feed You can follow this conversation by subscribing to the comment feed for this post.

Just to be pedantic: summing salaries can never be NP-complete. NP-complete is a statement about how hard a problem is to solve *in the best case*. It says that their is no quicker algorithm waiting to be discovered, ever.

You might have a very bad method for summing salaries that takes polynomial time or worse, but the problem itself is not NP-complete as there is an linear algorithm to solve the problem.

Arthur, Thanks for taking the time to highlight my lack of clarity. When I talk about summing salaries, I do not mean a simple:
salary = number of hours worked * hourly rate

Of course, this is not a NP-complete problem.

A NP-complete problem can be found in the mining industry where, in some European countries, the computation of employees salary depends on 170 non-linear and/or indeterminate elements.

But Artur's right - that's still linear or, in the worst case, polynomial.

I guess Kevin's message is more general though. There are many other NP-complete problems in the industry, let alone things like optimizing cargo delivery routing in logistics, or matching multiple customers to multiple service brigades that differ in skills, cost and geography. If you happen to have one, your resources will grow faster than you think.

The argument here is that the business challenges needs to be properly classified and complexity-ranked before proceeding to planning the infrastructure growth. In contrast, I've seen people (most people!) committing for significant hardware purchase expenses without performing this exercise, incorrectly assuming linear growth. This led to severe mismanagement of funds. This was also the message in the class that we gave for Oracle customers, mentioned in the article.

This is immediately differ progression. Is it enough to satisfy CEOs IT demands? Usually, the achievements of immediately differ progression is suitable. In computational analysis, strategy problems are acknowledged by their side-effect denoted as O(n).

The comments to this entry are closed.