Empowering AI with User-centric Data

Artificial Intelligence (AI) has turbocharged appetites for massive datasets, especially in the world of big tech, as core technology providers look to refine the performance and accuracy of their AI models.

Companies like OpenAI and Google, for example, have famously gathered extensive datasets, scraping virtually everything available on the web to train their large language models (LLMs). This Big Data approach has worked well for these tech giants — but it’s not a one-size-fits-all solution for companies trying to leverage AI.

In fact, most businesses do not have the need, nor the resources, to develop such expansive models. For many businesses, the goal is not necessarily to build the next big GPT model but rather to harness AI to improve their offerings within specific contexts, such as customer service, personalised marketing, or improved recommendations.

When the goal is to deploy generative AI in targeted, specific use cases data must be highly curated and accurate. There are particular challenges when companies put generative AI-enabled experiences in the hands of their end users. Users are hesitant to turn over personal data, yet demand personal experiences. This shift in focus requires a different approach to data management — one that values quality over quantity and prioritises data integrity, user consent, and regulatory compliance.

The problem with AI systems architected for Big Data

Data lakes and data warehouses are great for holding the massive amount of general data needed to train LLMs like those from Google and OpenAI. However, they are a poor fit for most companies who need to apply these AI models to more specialised or sensitive data.

For example, if a company is working with personal customer information — which contains relatively few data points per customer, but can change frequently based on customer inputs and demands — they can run into major challenges when it comes to data quality, data privacy, getting consent, and complying with regulations like the General Data Protection Regulation (GDPR).

Traditional data warehouses and similar Big Data management tools are designed for broad data collection and analysis, rather than for handling the detailed requirements of privacy and consent on an individual level. GDPR, for instance, requires businesses to handle personal data with strict rules, like getting explicit consent from users, limiting who can access the data, and respecting individuals' rights to control their information.

Managing these requirements can be tough with Big Data as they often lack the flexibility to enforce consent, control data access dynamically, and handle data with the precision needed to meet strict regulations.

Therefore, businesses dealing with personal data that needs careful handling and permission-based access need to explore other solutions that are better suited to meet these needs.

The Solution: A User-centric Approach to Data Management

User-centric data management offers a fresh approach that better fits the needs of businesses aiming to use AI in more focused, specialised ways. Rather than merging data from 20 different systems into one massive data lake or warehouse, which now represents the data of millions of

customers, an organisation can connect all the data relevant to a user with that user’s identity, store it in a single place, and give each user access to the relevant data. This method not only boosts data accuracy and relevance but also makes it easier to manage data privacy and consent requirements.

This form of data management is already making a big impact in how businesses operate across different industries. In healthcare, for example, user-centric systems allow a single source of patient data to be securely accessed by a variety of hospitals, clinics, and research institutions, without creating unnecessary ‘copies’ of the patient. This boosts collaboration while keeping patient privacy intact. By making it easier to share information safely, this not only helps improve patient care but also speeds up research and innovation in the medical field.

User-centric data stored in Solid Wallets

Solid Wallets, built on the Solid protocol developed by Sir Tim Berners-Lee, are one example of a user-centric data solution currently out in the market. Solid Wallets allow individuals and businesses to share data via a common format and interface, ensuring users own and control their information, decide who can access it, and maintain privacy and security.

These Wallets solve a lot of the issues we see in Big Data systems: they facilitate data interoperability, allowing data to be used across multiple platforms without being siloed, all while supporting user-centric data sharing with explicit consent. For businesses, Solid Wallets enhance data privacy and security, helping to build trust with customers and comply with data privacy regulations. They improve collaboration by enabling seamless data sharing across departments and with external partners, foster innovation, and reduce infrastructure costs associated with maintaining large centralised databases.

These data models are not just about staying current with technology trends; they’re about fundamentally changing how data is managed and utilised to drive innovation and growth. User-centric data solutions represent the future of data management, offering a path to more secure and efficient data practices.