I would say, that all depends on how you fertilize and how many fish you have in your tank - a dirty gravel is not a good thing to have.
I normally vacum the mosses and riccia mostly, but also the gravel. I add liquid fertiliser, which helps me better to keep a good healthy nutrition for my plants.
If you are into nature aquariums (the natural), then never vacum gravel