I agree with starting on time. I think it is rude to waste the time of those that did show up and are ready to tackle the agenda.
People will learn that you will start without them. I think it is usually a sign of disrespect to be late for a set meeting. (I realize that on occasion something will happen–I am referring to the habitual late arrivals)