Don't worry if it's not a huge amount, it won't harm the fish.
And if it's pretty much made the water go yellow-ish, still no harm done.
Most fish love the it and can sometimes make a fish more colourful.
Although the water may fade slightly as the filter might clean it up a bit.
Or depending on how messy your fish are, I'd say it might get a bit darker.
I did the same thing, except I got my drift wood from the beach.
So many interesting pieces there but a much longer process before putting them in the tank.