At one level, what you say makes perfect sense, and I agree with you. The problem is that if a suitably tailored approached to merit-based pay on your floor, leaves people on my floor with the impression that people upstairs get merit pay for accomplishing less than folks on our floor, trouble isn't far behind.
Ultimately, the challenge is to come up with, and especially communicate and explain clearly and plausibly, what the basis for any seemingly differential merit-based pay scheme is. If you can achieve some degree of uniformity across the organization, more power to you, but a great many organizations face the challenge of having multiple branches or units that simply have to operate in a different way, making the perception of parity in merit-based pay particularly difficult to achieve. And since the purose of merit-based pay is to affect worker motivation, the perception of effort being valued can be undermined by the perception that it is valued more in one corner of the organization more than in one's own.