I've used both drift wood from the beach, and swamp wood off several farms and never had any probs. The drift wood seems to sink reasonably quickly and may slightly tinge the water but after a few water changes clears up.
The swamp wood seems to take a lot longer to sink and can leach and make the water go tea colored, which isn't a problem if you have a Amazon biotope I guess. You really need to soak it for several months, but I never have. Our latest driftwood is Macrocarpa and hasn't leached at all but after about 3 months it still isn't sinking......hmmmph. I guess if you get swamp wood from a peat paddock it may cause your PH to drop, though I'm not positive about that, I'm sure someone here would be able to clear that up.