Your AI strategy is only as good as the architecture beneath it

There is an optimism at the start of every major technology buying cycle. Vendors arrive with enticing promises, budgets unlock, and procurement teams sign contracts with confidence in their decisions. The full-stack purchase, which usually ends up being made, looks especially appealing during this stage, appearing as the safest and most manageable option.

Five years later, it is something else entirely. Vendor dependency leads to technical debt and migration costs which exceed whatever had been saved through integration. The industry has seen this before, with cloud and with digital transformation, but AI is making the underlying problem increasingly impossible to ignore. 

The conventional approach to architecting is stuck in the 90s, with its monolithic estates and hardware-centric thinking. This made sense when workloads were predictable and compute was scarce. It no longer makes sense in the age of AI where models evolve faster than any hardware contract cycle.

Infrastructure is no longer a hardware question

The first thing CIOs need to reckon with is that the question of infrastructure has fundamentally changed. For most of the last three decades, infrastructure decisions were really hardware decisions, looking at servers, storage arrays and networking kits. The software would run on top of this, and the vendor would be responsible for the full stack.

That model worked when workloads were relatively stable and the pace of change was measured in years. AI operates on a different timescale with models being superseded in months instead. The compute requirements for inferencing at scale are not the same as those for training and the data gravity problem has become orders of magnitude more complex than anything a conventional on-premises architecture was designed to handle.

Infrastructure has now evolved into software-defined question. The underlying hardware matters, but what governs competitive advantage is the ability to abstract, orchestrate, and move workloads dynamically across infrastructure. 

The case for virtualisation

Virtualisation is not a new concept. Compute virtualisation has been popular for well over a decade, and most organisations are reasonably comfortable with the idea that a physical server can host multiple logical workloads. Network virtualisation has followed a similar path, with software-defined networking now standard in most enterprise environments.

Storage virtualisation however has lagged behind and, considering data is the raw material of every AI workload, how it is stored is becoming increasingly important. Where it lives, how quickly it can be accessed and how efficiently it can be moved between environments determines whether an AI initiative delivers value or simply consumes infrastructure budget.

What AI actually requires  

Optimising infrastructure in the AI context therefore means abstracting storage from the underlying hardware so that data can be tiered, moved and managed according to need rather than physical location.

This allows the workload layer to remain stable despite external changes such as GPU generations progressing, new cloud capabilities or on-premises hardware reaching end of life. Containerisation is the other side of the virtualisation coin, allowing AI workloads to be portable so the same model can run on different infrastructure without requiring a rebuild every time the underlying environment changes.

These both enable storage and compute to scale independently, a necessity considering AI workloads are inherently uneven. For example, whilst training demands massive compute bursts, inferencing needs sustained, lower-intensity throughput. This means compute for AI fluctuates in a way which will not be aligned with storage requirements.

Coupling storage and compute scaling therefore forces organisations into a lowest-common-denominator architecture: either over-provisioning one to keep pace with the other, or accepting bottlenecks which constrain the entire pipeline. 

Independence on the other hand allows each layer to be right-sized and upgraded on its own cycle, dramatically reducing both cost and the risk of a single infrastructure decision becoming a ceiling on AI ambition.

Staying flexible for the future

Modernised virtualisation is thus the architectural foundation on which any serious AI capability should be built. The organisations that skip this step are making a bet that their current vendor's roadmap will remain relevant for the lifetime of their infrastructure contract. Given the current pace of change in AI hardware and software, that is an unusually optimistic assumption.

Ensuring successful architecture means resisting the gravitational pull of the full-stack deal and treating infrastructure as a strategic rather than a procurement question.

We are still early in the AI infrastructure buying cycle so it important now to look out for buying habits which will lead to technical debt down the line. The decisions being made right now about architectural models and how deeply to couple AI workloads to specific hardware will come to define the migration bills of 2030 so the time to build for flexibility is before the contract is signed, not after.