I've heard people use "Goldfish" to refer to any kind of pet fish. I wouldn't be surprised if these people aren't fishkeepers, noticed there's something swimming in their pond and went, "Baby goldfish!"
Yeah, they're hard to tell from colorless guppies. Similar size, similar shape, given the difficulty getting a good look at fish in a bucket, bag, etc I'd say they're extremely easy to get confused.