Data doesn’t make decisions; people do. Data can inform decisions.
I say this in response to the person who told me this week that performance measurement, like accountability, “only works on paper”. Well, if your performance measurement strategy only works on paper, than it isn’t
working at all. Here are a number of factors about performance data that you need to know to use it practically to inform your decision-making.
First of all, let’s clarify what I mean by performance data. Performance data requires both raw data and indicators, but these are not the same. Raw data is meaningless on it’s own. Indicators are usually two things measured in the context of each other to form ratios or percentages. I’m going to use the phrase performance data to mean raw data we collect to help us set context (eg. number of visits to a web site, demographics of visitors, number of phone calls) AND raw data that is used to form indicators, lest I get caught up in semantics.
If I tell you that I weigh 150 pounds, and then ask you “Am I fit?” What do you say?
You may guess yes or no. Or you may say that you can’t answer that because you don’t have enough information, so I tell you my height as well. Now you can figure out my body mass index so you’re in a better position to make a decision.
Weight & height on their own = raw data. Weight in the context of height=indicator.
Performance data should be used in the context of other factors: your organization’s priorities, budget & people available, larger context, willingness to change.
When I go to the Doctor, she checks my weight & height, but she also does a whole barrage of tests to collect data for indicators of healthiness – cholesterol, blood pressure, iron & vitamin levels in my blood, etc. She compares the data she collects against what is normal for my age and takes into account my family history and anecdotal evidence of how I feel to make an informed decision about whether or not I am healthy. Data is not the only factor she
considers, but she does not attempt to make a decision without any data
either.
Performance data is useful to evaluate whether or not something is effective. But first, decision-makers need to agree on what defines ‘effective’.
Logic and common sense must be built into the connection between what defines effectiveness and the indicators that are then used as to measure progress towards that goal. This should be a discussion amongst decision-makers. Sometimes using an independent source is helpful.
My Doctor & I agree that a fit body is free of disease and performs optimally. But after some discussion, we also agreed that a sane mind is an important component of health so we also collect information about that.
Once you know if an activity is effective on it’s own, you can compare it to other activities and factors, such as your priorities, to help decide if it’s worthwhile. “Worthwhile” varies for everyone. I volunteer somewhere and sometimes only one person shows up. Is that worthwhile? That’s up to me to decide.
Performance data can be used for course correction, so continue monitoring between evaluation cycles.
I only go to the Doctor once a year, but I want to make sure I’m healthy all year round, so I need indicators that I can track myself on an ongoing basis. For a fit body, I may track my weight (now that I know what weight represents a reasonable body mass index for me), the ability to run 10Km 3 X/week and the trend of how I feel from day to
day (am I getting sick? Am I tired all the time?).
Performance data has to be practical to collect. Ideally, the data collected serves multiple purposes when used in different ways.
Couldn’t other indicators be used? Of course! But I have to be practical – I don’t want to spend more time collecting information than I need or more than I can use, so I’ve chosen indicators that I can assemble from the data that are easy and convenient for me to collect. I also choose these because just by focusing attention on them for a
minute a day or for five minutes a week, I may reveal other problems (mmm, I feel eally tired this week, maybe I need more sleep) with my diet, fitness, mind and body.
Performance data can be used to track progress against a goal.
Performance data can be used to help set targets. Through monitoring, you know when you’ve arrived at the target and can shift focus – on maintaining status quo or setting an even more ambitious target.
Priorities change (I’m working hard to meet a deadline and don’t have as much time to exercise) and I find myself ballooned up to 200 pounds. I know my healthy weight is 150 so I adjust my activities and check in once a week on my progress. When I reach 150 I treat myself to a brownie each Friday, but I don’t resume the activities that got me
to 200 pounds in the first place. I’m then freed up to focus on some other priorities.
All of this is to say that measuring performance should make sense to you for your products and activities. It should be useful and beneficial to you. If you’re not making decisions with the help of data, you are making opinion-based or belief-based decisions, which may be fine too. It’s up to you to decide.
Modified & reposted from: http://usability4government.wordpress.com/2010/03/25/only-works-on-paper/
Thanks for posting, Laura – your ideas about data without context dovetail well with Danah Boyd’s remarks at the Gov 2.0 Expo, where she spoke under the title “Transparency is Not Enough.” She made three points that apply here:
1. Information is power, but interpretation is more powerful.
2. Data taken out of context can have unintended consequences.
3. Transparency alone is not the great equalizer.
She spoke about the critical need for information literacy – that we need to educate people how to read data. In light of your post, doctors are trained to interpret the data you describe, right. Likewise, citizens need to be trained to read the data/information that government shares with them…or, at minimum, it needs to be written in a way that can be understood by the average person.
One other quick thought (that is going to become a blog post of my own). While I was at the convention center, I noticed a large photograph on the all of a group of Native American children. There was no caption, no context. It was just ‘data’, right?
In sum, you’re spot on: data without context is meaningless at best and dangerous at worst, especially if not interpreted correctly.
Totally! I was following on Twitter. V. insightful.
Look forward to your post on…captionless photos?! 😉
Measuring Return On Investment (ROI) on knowledge management products?
Anyone know of a model or methodology for measuring ROI on knowledge management tools?
-social networking
-document management
-information management
-communications management
Preferably in the Public Sector but private sector will do.
Regards,
Oh wow. I don’t know where to start. In Canada we use a combination of Results-based Management and balanced-scorecard approaches. Not always well or completely; but those are the methodologies suggested. Basically, each scenario differs, so you have to plug your own goals, outputs and objectives into a logic model to determine what to measure to get your own ROI or whatever other ratios demonstrate your results. Here’s a how to: http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx