There is, of course, the whole scientific-political debate about whether certain people should be regarded as "slower" and "behind" or simply differently abled. But there's a linguistic reason, too—lots of insulting nouns in the English language end with -ard. Again, from the Online Etymological Dictionary:
-ard also -art, from O.Fr. -ard, -art, from Ger. -hard, -hart "hardy," forming the second element in many personal names, often used as an intensifier, but in M.H.G. and Du. used as a pejorative element in common nouns, and thus passing into M.E. in bastard, coward, blaffard ("one who stammers"), etc. It thus became a living element in English, e.g. buzzard, drunkard.
Obviously, this isn't the case with inanimate nouns like "custard" and "mustard"—but it tends to work with words that describe people. "Dastard" came along in the 15th century to mean "one who is lazy or dull," probably from a marriage of daze plus -ard. "Drunkard" (originally "droncarde") came along in the 1500s from a similar marriage of drunk plus -ard, and so on.