Would an exfoliating scrub be okay?

I scrub my whole body with a yellow exfoliating towel once a week. Scrubbing vigorously relieves stress, and I absolutely cannot stand it when I don't because I can see the white buildup of dead skin cells. My body also feels itchy if I don't scrub. I've heard that scrubbing is really bad for the skin, but is it really that bad to just lightly remove the pitch-black dead skin? For me, my skin actually feels softer and seems to have improved after I scrub.