Agencies have lots of data sources at their disposal and use many different technologies — but when those resources and IT don’t “talk” to each other, the result is fewer meaningful insights and more security gaps. It’s a failure in interoperability and data integration.
How It Happens:
It can begin easily enough: IT teams set up new hardware, software or systems in response to one need, then add more technology to address a subsequent challenge, and so on — something of a Frankenstein approach. Legacy tech often is incompatible with more modern options, but still may support critical functions. Agencies cling to siloed, duplicative databases with inconsistent formats. In addition, limited resources and expertise, employee resistance, and governance challenges can be hard to overcome.
Solution:
Begin by ensuring that your systems, applications and devices seamlessly connect and share data, regardless of technology and formatting differences. This interoperability depends on creating a common standard, so all the pieces work together — like when a country chooses an official language that various populations use to communicate. Interoperability has three core elements:
- Standardization — using compatible formats, protocols and structures
- Real-time accessibility — obtaining data when it becomes available
- Scalability — being able to add data sources and users without reengineering
All of this allows for data integration, the process of combining data from multiple sources into clean, unified datasets that can drive various analytic needs. From a technical standpoint, there are several ways to integrate data, including:
- Extract, transform and load (ETL) — Pull together and harmonize copies of multiple datasets, then load them into a database or data warehouse.
- Extract, load and transform (ELT) — Almost the reverse of ETL: Load raw data from multiple sources into a data lake or other big data system first, then wait to harmonize the data until you need it for a specific purpose.
- Data virtualization — Rather than uploading or moving all the data from various systems into a new data repository, combine the data virtually, so that end users have the same view.
Streaming data integration — For timely insights, continuously integrate real-time data into data stores and systems.
Public investments in digital development, transformation, and infrastructure can only meet citizen needs if data and systems are consolidated and interoperable. — Demystifying Interoperability
How Data Integration Works: Extract, Transform and Load (ETL)
A version of this article appeared in our guide Better Data Strategy for the AI Age. Download the guide for more insights into how agencies can adopt more coherent, effective ways of managing their data.