Mark Hammer

For a variety of reasons, some staffing actions will naturally proceed faster than others. For a variety of reasons, which of these there happens to be more and less of, at any given point in time, can also change. When looking at “improvements” in speed of hiring under various reforms, it is important to unconfound changes to the relative incidence of faster and slower types of staffing, and efficiencies in staffing per se.

Much like illusory/artifactual changes in “school performance” arising from one tail of the distribution being moved to another school, or being augmented by more students at that particular end of the distribution, one can commit several different types of errors in evaluating the apparent consequences of a policy change. For instance, the policy may well have rendered staffing more efficient, overall, but there are now more of the slower types of staffing actions in the mix, creating the impression that nothing has really changed and the policy is a failure. Alternatively, the policy was actually ineffective, but the proportion of staffing actions that are of the faster type has increased for some reason.

I am in absolutely no position to evaluate the impact of the hiring reforms, but I would hope that those whose judgment is important in the matter ARE in such a position, and take the various measurement caveats into account. It is important to have the data in place to be able to put such alternative hypotheses to the test, and be able to discount them on empirical grounds, with confidence.