Something must be irritating them and they are trying to rub it off. I would have thought you would see, if it was velvet, but hard to tell due to their colouring. I imagine it will be hard to anything on them!
Google will lead you to many sites with good pic and symptoms charts as well as suggested treatments.