The main role of villains and antagonists in stories is to generate suspense, anxiety, and fear in the reader. They make it possible for us to worry that the hero will not be able to avoid a significant threat or remove a lack. There are other effects–mystery, poignancy (about the human situation like in Les Mis). But those are secondary.
For the main things to happen the villain has to be a credible, significant, and immediate threat…all the way through the book until he or she is smushed or wins.
If he’s not credible, the reader realizes there is no real threat. If the threat is not significant, who cares? If it’s not immediate, again, who cares?
So how do we make a villain like that? We make him or her smart, powerful, a few steps ahead of the hero, and dedicated to doing something we root against (because it’s just plain wrong or because we love our hero and want the best for him and the villain is pitted against him). He has to be able to put the hero on his heels most of the way through the book, and our poor hero is scrambling to adjust.
You can have all sorts of villains–liked by many or few, kooky or calculating, eccentric or plain, noble or sadistic–just as long as they remain significant, immediate, and credible threats. The minute they lose threat status, the game’s over because at that point fear, anxiety, and suspense in the reader vanish.
