I'd give the new gravel a good wash and perhaps boil it if I was concerned about the source.
Then I'd work quickly - move all the fish and most of their water into another container or tank, remove all the gravel and the remaining water, then rebuild tank and add the fish and their water back. I'd also try to find a way to keep the internal filter running during this process so the good bacteria don't die.
It would be a lot of work but would ensure you didn't release any bad stuff into the water. You probably would have to do frequent small water changes after that just to be on the safe side.
If I couldn't be bothered with all that fuss I'd clean or boil the gravel then do the change in several small stages - scoop out an area of the old gravel and replace it - wait a few days then repeat the process until its all done.
Cheers
Jude