Good question. I haven't thought about doing it but is does make me you think if there is a chemical risk.
Years ago I used to be right into collecting and studying rock and minerals. I was a member in a club blah blah blah, anyway. Something that came up more then once in club meetings and rock and mineral sales was some rocks and minerals can release chemicles into the water. And some of them can be very nasty.
Be careful what stones you put into the tank if you are unsure.