moldy smoke wood

Anyone know of a problem, taste or safety, if using smoke wood that has very minimal mold on the surface of the wood-I'm talking about a few small white or brown patches that are flat without any thickness or substance? I've read that it can release toxins, someone else told me the smoke would ruin the taste of the meat. I have a hard time believing any of that, but I'm not an expert on the subject. Thanks.