Certainly one of the concerns alluded to is the impact that such monitoring can have on employee trust and morale, which should not be treated lightly. That's as much a matter of how something is done/monitored as it is what is done/monitored. Being up front about the fact that one will be monitored is one means of establishing appropriate expectations between employer and employee, but that can also be perceived as a little heavy-handed if the grounds and terms for such monitoring seem unreasonable on the face of it.
All my surface mail at work (with the exception of internal stuff like pay stubs) gets opened up and resealed before I receive it, even things like newsletters and publications from other government agencies. On the other hand, I know this is happening to everybody and not targetting me specifically (a quick glance across everybody else's mailbox copnfirms this). Plus, the envelope closed back up with a piece of tape shows me that it has happened, and happened consistently, so there is little sense I am being specifically and clandestinely observed for any reason. With e-mail, that obvious presence doesn't occur so monitoring can happen without one's knowledge, which I think is a sticking point.
Some workplaces maintain several separate e-mail systems/servers, one for work internal to the organization and one for contact with the "outside world". That's one solution. Of course, that doesn't stop anybody monitoring that e-mail with the outside world.
Whatever approach is adopted by the employer, it should conform to all other existing guards and laws concerning invasion of privacy. People can't just tap your phone, they can hang around your bathroom window, they can't ask the bank how much money you have, and they can't open your mail. I suppose all of those things can be over-ridden if there is sufficient grounds to obtain the necessary warrants, but it can't just be something the employer does because they feel like it.