You're doing your job simply by keeping the man in front of you. You should be doing your best to win the ball and discourage him from playing the ball in, but if he does, he does. You need to count on the other defenders to pick up their marks, and make it very hard for them to score. Defending isn't the most glamorous job, and its far from easy, but you need to take pride in your work.
You shouldn't be 'walking' side to side, but keeping your body straight to them, and keeping one eye on their hips, and the other on the ball. The more you play, the better you will be at judging time and space, and being able to bounce on the ball when they make a rare harsh touch, or something. Most of all, never lunge. It's hard for some players to resist, but all those great dribblers really want you to do is show some kind of hesitation, or take your eyes off the ball... Anything, really. Stay on your toes, and be alert. Like the others said, patience is key.
I've played defense on some of the best teams in my province, but I'm also a naturalized midfielder. So I get to know more of what to look for in an attacker and how to combat it, mostly because I'm trying to do that myself a lot.
Hope to have helped you.