Scott, "most certainly a fallacy" is far too strong a statement, and you contradict it yourself by the end of your response.
Where I work the tendency is for the Feds to watch the clock pretty closely...but then again, to be fair, so do the contractors. It seems more a function of the climate in a particular office. Greater connectivity--and accompanying expectations of reachability--creates blurring of the work/home boundary. Employees are justified in wanting to try to re-establish some boundaries where they can, and leaving promptly may be part of that.
For me it's not about how long someone's at their desk (within reason). It's how they get through their tasks and the quality of their output. Unfortunately, here, I see Feds and contractors alike repeatedly fall down.