Bastard, retard, dastard, dotard... where did all these insulting "-tard" words come from?
First, let's dispense with "leotard," which came from Jules Léotard, a trapeze artist from Toulouse, France—it doesn't fit in with the rest of the words.
The Online Etymological Dictionary says "retard" comes from the Latin retardare, by way of the French verb retarder, which first appears in the 13th century. So how did it come be considered an insulting word?
There is, of course, the whole scientific-political debate about whether certain people should be regarded as "slower" and "behind" or simply differently abled. But there's a linguistic reason, too—lots of insulting nouns in the English language end with -ard. Again, from the Online Etymological Dictionary:
also -art, from O.Fr. -ard, -art, from Ger. -hard, -hart "hardy," forming the second element in many personal names, often used as an intensifier, but in M.H.G. and Du. used as a pejorative element in common nouns, and thus passing into M.E. in bastard, coward, blaffard ("one who stammers"), etc. It thus became a living element in English, e.g. buzzard, drunkard.
Obviously, this isn't the case with inanimate nouns like "custard" and "mustard"—but it tends to work with words that describe people. "Dastard" came along in the 15th century to mean "one who is lazy or dull," probably from a marriage of daze plus -ard. "Drunkard" (originally "droncarde") came along in the 1500s from a similar marriage of drunk plus -ard, and so on.