Home Critical engine A Bridge Over Problematic Data: Giving Enterprises Access to Advanced Machine Learning

A Bridge Over Problematic Data: Giving Enterprises Access to Advanced Machine Learning


From banking to healthcare, nearly every organization wants to implement advanced AI and machine learning-based applications that transform efficiency and create new services and business opportunities.

They want smarter apps for important use cases like real-time fraud prediction, better customer experience, or faster, more accurate analysis of medical images.

The problem that most organizations face is that they store data in different forms and locations, each of which may belong to a business unit or department. Making this data usable by advanced applications is demanding.

Before the advent of the new paradigm – the smart data fabric – the approach would have been to create a data lake or warehouse, using the relatively low cost of storage and compute. The organization also likely uses time-consuming ETL processes to standardize data.

The traditional data lake is slow and increasingly swampy

This approach, which is still widely used, has had its victories but creates a centralized repository that makes data difficult to analyze and often fails to provide consistent or timely answers to business questions. It tends to bring the data to the query rather than the query to the data, creating latency and often causing significant and unnecessary duplication.

This makes it very difficult to adapt new data sources in response to changing business needs, which undermines organizational agility. It is also unable to meet the current demand for clean data suitable for new composite applications that are enabled by AI and use machine learning (ML) and integrate with massive, pre-existing datasets.

In truth, almost every organization still struggles to provide a consistent, accurate, and real-time view of their data. The vast majority still keep data in separate silos, with perhaps only 5% able to use data that is less than an hour old. It will not, for example, allow the transition from relatively simple fraud detection to prediction, capable of identifying and tracking money laundering activity in extremely complex financial flows.

Organizations make too many decisions using outdated information, overwhelmed by the variety of data sources and the complexity of unifying them. A global study conducted earlier this year by InterSystems found that nearly all participating finance organizations (98%) have data and application silos and significantly more than a third (37%) said their biggest challenge in terms of data was the time it took to access that data. Like so many organizations, these financial firms need to be able to see into their complex, heterogeneous data and receive fast, consistent answers to their business questions. They need an architecture built around business needs, rather than a large, convoluted data warehouse or lake that becomes just another rigid silo.

This will allow companies to use ML algorithms that they know will bring them great benefits. But advanced analytics and AI depend on clean, harmonized data, which is hard to achieve in a repository. This is why the level of innovation in ML models currently exceeds the rate and scale of deployment. The absence of reliable data makes it impossible to integrate these models into the operational applications that generate them. In the meantime, the volume and complexity of data continues to grow.

Bring the query to the data

Fortunately, the concept of Smart Data Fabric removes most of these data issues, bridging the gap between data and application. The framework focuses on creating a unified approach to access, data management and analysis. It builds a universal semantic layer using data management technologies that stitch together distributed data regardless of location, leaving it where it resides. A fintech organization can create an API-enabled orchestration layer, using the Smart Data Fabric approach, providing the business with a single source of reference without the need to replace systems or move data to a new one. central location.

Capable of performing in-flight analysis, more advanced data management technology within the structure provides real-time information. It connects all data, including all information stored in databases, warehouses, and lakes, and provides vital, seamless support for end users and applications.

Business teams can dig deeper into data using advanced features like business intelligence. The organization can deploy tools using machine learning algorithms that enable next-generation applications.

It’s a paradigm shift, bringing together two worlds of legacy and new data for advanced, ML-powered use cases. This is critical, enabling a single view of data across what can be a complex organization like a financial institution that has a lot of legacy silos. The technologies that make up the fabric transform and harmonize data from multiple sources on demand, making it usable for a variety of business applications.

Organizations need an intelligent data fabric to bridge all of their many types of data in different locations and sources, to gain seamless access in real time and to be able to deploy the next generation of applications powered by the AI. In fact, it’s not about technology, it’s about execution and how the fabric serves business agility, future-proofing businesses, and bringing revenue-generating transformation within their reach.

About the Author

Saurav Gupta is a sales engineer at Intersystems. InterSystems is the engine behind the world’s most important applications in healthcare, business and government. Everything we build is designed to drive better decisions, actions and outcomes for the people who stake their lives and livelihoods on our technology. We are guided by the IRIS principle, namely that software should be interoperable, reliable, intuitive and scalable.