THE CHALLENGES OF LONG-TERM DATA STORAGE AND THE TECHNOLOGIES THAT CAN ENABLE IT

We live in an age of data as the lifeblood of human activity. This era was founded on data as a valued commodity, with the big beasts of the internet betting on it more than two decades ago. Since then, and as big data gave way to AI, data as the source of intelligence in all our activities came to be firmly recognised.

So, the default stance for all organisations is to store data rather than delete it. That’s partly because the ability to interrogate and transform data into information is fast becoming table stakes, and ever-improving. But it is also because data must be kept for reasons of legal and regulatory compliance, often for decades. 

That means the subject of long-term data storage, and the technologies that will support it in the coming years, are becoming increasingly top of mind. 

So, we thought it was time to ask some questions about how we will store data long into the future. We assembled a panel of experts and asked them about the biggest challenge for organisations storing data for decades or longer, the main myths or misconceptions that exist about long-term storage, and which emerging technologies have the most potential for long-term data storage.

Long term storage challenges: Contradictions of conflicting timescales

The essence of the challenge faced in long-term storage is that we may require data to be available for decades to come, or even longer. Meanwhile, however, many things about how we keep and manage data tend to change on much shorter timescales. 

Tim Pfaelzer, GM & SVP EMEA with Veeam, summed it up well. He pointed to the need for data to remain secure, accessible, and compliant over time, even against a backdrop of data growth, changing technologies, and shifting laws and regulations. 

“As data volumes grow and become more distributed across cloud and on-premises environments, maintaining data integrity, governance, and discoverability becomes increasingly complex,” said Pfaelzer. “Technology standards, storage media, and data formats can become obsolete, making long-term readability and usability a major concern.”  

The danger to organisations is that this changing landscape, and the need to keep pace with it, can bring exposure to risk. Scality CMO Paul Speciale noted that the primary challenge is maintaining data durability and access amid continuous technological change. 

“Hardware lifecycles are shortening, media formats are constantly shifting, and software stacks evolve at a rapid pace,” he said. “Over the span of decades, the risks extend well beyond media degradation to include format obsolescence, system incompatibility, and the erosion of operational expertise.”

But where there’s risk, there is potential advantage. Skip Levens, head of product marketing at Quantum, highlighted that organisations which master the challenges of long-term data storage will be those that turn it into a strategic capability. 

“The risk is no longer losing data – it’s falling behind because the data you do have isn’t stored, organised, or accessible in ways that future workflows will demand,” said Levens. “The companies that win will be the ones that bring all their historical content, research, operations data, and customer signals into a pipeline where it can be indexed, enriched, and activated by new analytics and AI models.”

He added, “No one knows exactly which questions they’ll ask of their data five or ten years from now, but we do know this: Organisations that store everything they can, in formats they can afford, with metadata they can trust and retrieve, will have an overwhelming advantage. They’ll innovate faster, train better models, deliver stickier services, and make decisions with decades of institutional knowledge at their fingertips.”

Martin Kunze, CMO & Co-Founder of Cerabyte, put a finer point on the challenge here. Namely that a key task is to structure and describe data today with metadata “so that future search and analysis methods, which we cannot fully anticipate today, can still operate effectively on it. In other words, we are being asked to design and prepare for access patterns, tools, and use cases that do not yet exist.”

Meanwhile, Robson Andrade, CRO at Geyser Data, highlighted the datacentre costs of maintaining data on current technologies such as hard disk drives.

“The adoption of large volumes of mechanical disks, for example, involves much more than hardware investment. It also demands significant physical space, energy consumption, and continuous cooling systems.”

Scott Shadley, from the board of directors of SNIA – the Storage Networking Industry Association – emphasised the work the organisation does to provide standards for long term storage technologies. 

Shadley said, “The challenges that exist for corporations are governed by different rules and the solutions they require to store data are driven by the standards and solutions that SNIA as a conglomerate of these companies can provide. These solutions range from how to use tape, HDD, SSD, and even DNA to understand archiving data. We have several work groups that focus on different methods, different products and even different interfaces to store the data.”

The myths and misconceptions of long-term storage 

The biggest myth in long-term data storage, as highlighted by David Norfolk, practice leader, development and governance, at Bloor Research, is the belief that data can simply be left alone until we need to access it.

“It is a myth that you can just store data and worry about retrieving it when the time comes,” said Norfolk. “An archive must be designed and planned to accommodate anticipated usage, and reviewed regularly to ensure it still meets the organisation's needs.”

That’s a concept at the core of how many panellists think about long-term storage and how it will be implemented.

For Aleksandr Ragel, CEO of Leil Storage, this means we need to rethink how we categorise tiers of storage. 

He said, “AI, analytics and compliance are turning ‘cold’ data into ‘slow warm’ data, with infrequent writes but repeated reads over many years. Architectures that assume data is written once and forgotten – the classic tape mindset – fail when organisations start mining ten-year-old logs, simulations or video for new AI models.”

Quantum’s Skip Levens expanded on this, explaining that storage tiers shouldn’t be static but fluid, enabling agility rather than an “archive and forget” approach.

“That mindset made sense when storage was only about compliance or cost containment,” said Levens. “Long-term archives must be treated as living assets: Continuously validated, indexed, enriched, and cleaned so future models can extract meaningful patterns.”

“Organisations with a durable advantage are the ones that architect archives that can be rehydrated quickly, updated as metadata evolves, and folded back into active pipelines on demand.”

Scality’s Paul Speciale amplified this point, and suggested that access to archived data is never really “cold”.

He said, “That may have been true in the tape-only era. Today, AI, analytics, and regulatory inquiries regularly reach into archival data. Archives are becoming active data lakes and data is queried, analysed, and repurposed far more often than many expect.”

Speciale added, “Archival storage is about ensuring data remains intact and usable over decades. That requires continuous integrity checks, immutability, and automated self-healing.”

Putting a finer point on how archives may potentially be accessed for decades to come, Cerabyte’s Martin Kunze said it’s vital to be able to interpret data, to allow it to become information.

“One major misconception is that long-term or permanent storage is solved simply by storing bits reliably. It is essentially useless if no ‘manual’ is preserved that explains how to identify the media, read the bitstream, interpret the bitstream, and transform those bits back into meaningful information.”

Some panellists wanted to address what they believed were myths surrounding the suitability of particular storage media as a long-term option for data storage.

Tape came under fire from some.

Tvrtko Fritz, CEO with euroNAS, took aim at tape as the best option for long term storage.

He said, “While tape can be cost-effective and reliable, retrieving data from it can be slow and sometimes complicated – especially when time is critical.”

Leil Storage’s Aleksandr Ragel riffed on a similar theme. 

“It’s a myth that tape is always the cheapest and greenest. Tape cartridges are cheap, but total cost is often underestimated: Handling, robotics, multiple copies, restores that take hours or days, and human processes add substantial hidden cost and risk.”

Ragel also suggested the idea that SSD will replace HDD for long-term archives is a myth.

“For high capacity and decades-scale retention, HDDs – and especially HM SMR – still deliver far better cost per TB and capacity density than SSD, and will for many years. HDD remains the backbone of all the data that does not need to be flash-hot.”

Key emerging technologies for long term archiving

So, we know the challenges of long-term data storage. They centre on being able to access data, potentially decades into the future, and at as low a cost as possible in terms of energy use, maintenance, etc.

All our panellists agreed on those principles. Where they differed most was on what tech solutions they favour for long-term data storage.

Some were keen to draw attention to novel technologies yet to see widespread or mainstream marketisation.

Cerabyte’s Martin Kunze stressed the ability of storage media to operate without ongoing energy costs as a key part of lowering TCO.

He said, “Any technology that can retain data without continuous energy consumption – for both media and environmental conditioning – and offers high-performance write and read at low cost is a strong candidate for a new tier in the data storage hierarchy.”

“In that context, permanence is less about satisfying a marketing label of ‘long-term storage’ and more about structurally lowering the TCO,” he added. 

Leil Storage’s Aleksandr Ragel emphasised the benefits of Host Managed SMR – shingled magnetic recording HDDs – allied with intelligent data placement on drives that can be spun down at times of lower usage.

He said, “HM SMR already gives roughly 20% more usable capacity at similar cost, and all large HDD vendors are pushing towards ever larger zoned drives, including upcoming HAMR based SMR models.”

Many panellists said there are currently no real competitors to existing storage media. 

Quantum’s Skip Levens was among these, and promoted a combination of object storage and tape.

He said, “Most ‘next-generation’ archival technologies are still years from practical adoption. No one can afford to bet their long-term retention strategy on laboratory promises.”

He added that, “This is why the combination of object storage and tape is so powerful. It delivers cloud-like durability and simple, automated accessibility, backed by an intelligent object layer that keeps even massive archives within easy reach - all on a medium that consumes essentially zero power at rest.”

Geyser Data’s Robson Andrade emphasised the potential cost savings of tape, and that its relative slowness of access is now compensated for by new approaches. 

He said there has been, “reignited interest in magnetic tape technologies, such as LTO (Linear Tape-Open), now with a completely renewed approach. Modern tapes offer access and management similar to that of a standard drive, thanks to intelligent tiering and their high compression capabilities.”

Meanwhile, QiStor’s CEO, Andy Tomlin, thought existing media were likely to persist as key choices for long term storage.

He said, “Long term is all about cost. I don't think any of the emerging tech seems likely to displace HDD and SSD.”

For some, SSD is a contender as long as there’s no need to write to the media too frequently. That’s the case for Greg Matson, SVP of products and marketing with Solidigm.

He said, “Long-term storage in an SSD is always overshadowed by endurance and retention issues. This can be the case for always-on drives, where data is changing rapidly and regularly and there are wear-out concerns that have to be mitigated. But, for true write-once-read-never or write-once-read-occasionally, SSDs provide a very viable solution.”

Finally, some panellists were agnostic about storage media and placed the emphasis on layers higher in the stack such as access protocols and advanced storage functionality.

Here, Paul Speciale of Scality emphasised the utility of object storage and S3 in particular.

“Scale-out object storage stands out for its virtually unlimited capacity, geographic distribution capabilities, hardware abstraction, and open APIs,” he said. “All of which support long-term accessibility, while its self-healing design significantly lowers migration overhead.”

“Meanwhile, software-defined immutability and WORM policies are reshaping archival strategy by delivering regulatory-grade retention without relying on proprietary hardware,” added Speciale. “In parallel, metadata-driven intelligence and policy automation are improving how organisations classify data, manage tiering, and enforce governance across long retention windows.”

Finally, euroNAS’s Tvrtko Fritz pointed to Ceph as a data storage platform suited to the long term.

“Ceph is one of the most promising technologies for long-term archiving,” said Fritz. “Its scalability is virtually limitless, making it ideal for organisations expecting continuous data growth. What really sets Ceph apart is its ability to distribute data across multiple sites or datacentres to provide built-in redundancy and resilience.”

Summary: Many contenders in long term storage, but customers will decide

In the AI era, the need to retain data is at the forefront of our minds. The value we can gain from it, now but potentially much further into the future, is critical to the operations of many organisations.

So, while we often need or want to keep data indefinitely, this creates challenges, such as the need to be able to access it many (technology) generations into the future, to ensure it remains compliant over time, and to be able to maintain the infrastructure in which it exists and connects to the wider landscape.

To fail in these challenges is a huge risk. But there is also great opportunity for those organisations that can get across the tasks involved.

We don’t know what the storage technologies for long term retention will look like decades from now. But we can see some interesting contenders in technologies that aim at long life and low maintenance such as ceramic storage.

Technologies outside the mainstream will have a job on their hands to supplant variants on HDD, SSD and tape as well as metadata and object systems to facilitate future reading.  

The likelihood is that new formats will emerge into general use, but what will succeed and what will fall by the wayside will be determined ultimately by market and customer acceptance.