Compelling characters are the brain of any good story. Well written characters will keep the readers leaning forward, wondering if he can accomplish his goal. Bad characters, on the other hand, will make the readers go away.
Let's suppose you have a character who is not likable. Note that I say likable, not compelling. For example, your character is manipulative and selfish, or even worse, he's a savage serial killer. How do you get a reader to root for that character?
Step One: Make Sure He's Unlikable from the Start
If your character is unlikable, be sure he's like that through the entire story. The first impression of the character will tell the readers what the author allows that character to do in the story. If he's a serial killer, let me know that instantly. Otherwise, I won't root for him anymore once I learn of it.
Step Two: Weakness, Weakness, Weakness...
An internal weakness is a personality trait that is so awful that it ruins the character's life. Give your character one. It can be a tramatic past incident, a fear, or a negative emtion. Even better, give the character a conflict that requires overcoming the internal weakness.
For instance, let's say I'm making a serial killer. His weakness is that he had a crush on one girl when he was in high school and now can't bring himself to kill anyone who has black hair, as it reminds him of who he was interested in back then. This is bad for him, as the cop determined to get him into jail for what he does is black-haired, causing it to be nessesary to overcome this fear.
Step Three: Reasoning
If your character is unlikable for no reason, he will be viewed as random and incomplete. Whatever they do, be sure they have some motive behind any and all actions.
As for my serial killer, building off his past crush, his motive for a career in murder is as revenge against a person (Who we will call Bob for the sake of clarity) his crush was attracted to instead of him, and every kill he does is toward a person who is close to Bob. He's not killing for sport or because the plot demands it. He's killing to express his strong emtions of anger toward Bob.
Step Four: Personal Goals
You may be thinking, but he already has a goal. Don't all plots revolve around a goal? Yes, but the character should have a goal aside from the main storyline in order to make him unique. What does he want?
My serial killer, obviously, wants to overcome his fear of killing black haired people to avoid getting caught, but also to fulfill his revenge quest of killing all those who are close to Bob.
Step Five: Quirkiness
Giving characters traits that set them apart from everyone else make them interesting. These quirks may or may not contribute to the plot.
For exmaple, my serial killer, when he's not getting revenge on Bob, likes to read comics. This quirk is used to show that he is still human, despite what he does.
If you think making a compelling, yet unlikable character is impossible, I'll tell you: It ain't. Accroding to screencraft, many TV shows, from Breaking Bad, to Girls, to House of Cards, have made unlikable characters that we can't help but root for. If they can do it, you can do it! Bye!