It is probably the latter reason, but I have rarely ever seen dead goldfish in my pond. There has seemingly been a pile-up of mud at the bottom of my pond (which has prevented the fish from reaching its depths) that would be near-impossible to clean.
I do not wish to put them back in the pond out of fear for their safety, but I cannot afford a tank large enough for them either, so I am between a rock and a hard place as it stands.