The Collection of Public Sector Data Now — And in the Future


All of the public sector that address the vulnerable and needy depend on data.  The Pay for Success and Social Impact Bond initiatives rely on data to determine the need for a particular intervention and for evaluating that intervention to determine the ROI.  Any evidence-building effort requires data.

What kind of data?  Currently, the data collected by government in the operation of these programs primarily facilitates 1) contractor payment and 2) federal compliance reports that states and counties must do.  Data is collected on services provided, to whom, by whom, where, and at what cost.  Sometimes, the outcome of the service encounter is also collected, like high school graduation, when it is known.  This includes, but is certainly not limited to, student data from kindergarten through graduate school, child care, welfare program participation, crime incidents, jails and prisons, job training, foster care, after school programs, home health aides, and a whole array of health programs.  Who pays for all of this data collection?  Who does it?

A great deal of social programming is funded by the federal government through block grants and federal financial participation of various sorts.  One of the strings attached is the reporting of how many individuals or households are served by each of the programs and the characteristics of those individuals and households (see  Of course, in many cases, the federal government also pays for the construction of the IT needed to collect these data (see  This type of reporting occurs in many domains, including education, human services, health care, and workforce training, although there are some notable exceptions that sit squarely on the shoulders of states—like the collection of crime data.

These data on program participation and outcomes is and probably has to be collected by the public sector workforce or contractors.  This is because the cost of the infrastructure and the staff time can only be borne by public agencies.  This is because government revenues dwarf that of what philanthropy or private investors might be willing to invest.  The data is a public good!  Also, the data collected is quite sensitive and one government function is to keep all of our data secure and private, including data about our education, health, mental health, substance abuse, earned income, and taxes

However, as state and local resources dwindle, we must find ways to collect data more efficiently.  Much of the data in the public sector is still collected on paper forms (usually entered into a database), but even those data that are collected directly into an electronic system are often done so in staff-intensive and antiquated ways.  (Some agencies still use the old green-screen CRTs!) When we think about who collects these data—whether it’s teachers collecting attendance and grades or police filling out crime incident or arrest reports or eligibility workers who take information for SNAP (Food Stamps)—there often is a way that would conserve their time do their primary jobs more effectively.

Many states are improving data collection methods by putting the applications for benefit programs online.  Putting tablets into the hands of teachers can speed the entry of attendance data and perhaps even send a text or email message to the parent of an adolescent who is skipping class (see  Speech recognition software can be used so that professionals, like police and teachers, can dictate their reports.  Those reports can then be mined for the information that needs to go into the database fields.  In theory, this process could speed up the data flow from event to data use (or to notification of parents!).

We can easily enhance we are currently collecting to take advantage of the data from fixed location or body cams to improve the data we have to act in the public interest.  While the technology is not quite there yet, computer vision could code video images to analyze crimes or interactions among public sector professionals and those they are intended to serve.  Of course, this has to be done in an environment where identities are protected and data is kept secure, but the potential benefit for the public of improved data collection is immense and the expertise that exists in the private sector and academia must be tapped to improve what we do for our most vulnerable individuals.

And, of course, we have to stop the duplication of data collection across system and sectors, which I will discuss in weeks to come.

Robert Goerge is part of the GovLoop Featured Blogger program, where we feature blog posts by government voices from all across the country (and world!). To see more Featured Blogger posts, click here.

Leave a Comment


Leave a Reply

Sonia Chakrabarty

Thank you for sharing this post! I especially enjoyed what you had to say about the relationship between federal and state/local levels and the data they use/collect.

Katarina Hong

I completely agree that there has to be a more efficient way to collect data. There’s always room for improvement. Thanks for sharing!