ours comes from Taumaranui, it has a bit of pumice (annoying) and a few sharp edges (most are round) and is rather dirty. but is cheap and does the job well, use it myself and never had a complaint about it
did you wash the silt out thoroughly??? depending on the source the silt may be the issue.
personally I have never run into gravel that leaches phosphates, it would not be good for landscaping as it would promote weed growth, the opposite of what landscapers want gravel for.
maybe try landscaping gravel from a different shop