Why do people give babies dirty looks?

When a mother gets on board of an aeroplane with baby, people give all kinds of looks like "Oh no, here we go". Luckily for me my baby never cried on his 8 hr flight from NY to London (he was only 3 months old) and you shoulkd have seen the surprising looks on people's faces LOL However, there were two twin girls who made more than enough noise. I'll be travelling again next monthto Africa, my baby will be 1 yr old then and hoping he will be just as peaceful as before. Our flight will be at night, so that might help! How do you handle such people's stares? My feelings would be hurt if someone looked at my baby the wrong way