Swim baldder problems can often affect a fish aftr it has eaten. Fancy goldfish are prone to balance problems due to their convoluted digestive tracts. They eat, the dried food swells up inside, and it takes a while for it to digest. While doing so, the fish develops balance problems due to intrenal pressure. Once the food as passed through the system the balance corrects itself until next time.
Some pellets swell up more when wet than others. It may be you are feeding pellets that are too large.