Have you become the go-to babysitter for half the family or many of your friends? The kind of babysitter nobody pays for and everybody expects to always be available? Just because you have your own children doesn't mean you have to care for others "as a favor" all the time.
Tell your friend or family how great their kid is, and then immediately say you -- unfortunately -- cannot babysit him right now. Starting with a compliment will make the following bad news a bit less of a blow. Emphasize the "great kid" part and you might not even upset the other person too much.
Make yourself unavailable on the times where you're usually asked to babysit. Suddenly become interested in taking a photography class or taking the kids to dancing lessons. This makes it a lot easier to say that you're not babysitting for them anymore and it gives you a way out if you don't want to explain why.
Be honest about the child and your kid not getting along, if that's the true. You don't have to tell the parent his child is the devil's spawn. In fact, don't put the blame on anybody. Instead, just say the kids are not getting along and you can't babysit anymore.
Put the blame on yourself if all else fails. Even if it's not true, this is the kind of white lie that will spare feelings from getting hurt. Don't say "Your kid is always crying." Instead, say something "I'm sorry, but I don't think I'm the best person to care for Tommy anymore; I just don't seem to be able to calm him down or get through to him when he's upset." Since Tommy is not at fault, how can your friend be upset?