In my gravel tank there is a bit of build up around the bases of plants and every week or 2 I disturb it a little in the areas towards the front of the tank and get rid of the main bits but generally leave it alone. I believe that tiny bugs and nutrients live in there and who's to say it's all poo, there will be all sorts of algae/plant matter that drops and accumulates, it's like the decaying matter you get around trees, it just goes back into the soil. As I have crypts I can't suck it out of the gravel or I disturb them anyway. In the sand tank I don't know where the crap goes, a little settles and some hides in the corners, but it takes a couple of months or 3 for the water flow in the filter to slow down.
Like Ally said, I reckon just worry less.