Question of those with Apple AirPods Pro

I have some AirPod Pros and I have a question regarding talking on the phone with them. I usually use them so that I can multitask and be using my hands for something. The caller on the other end, however, cannot hold a conversation with me because of the noise that the microphones pick up. Is that normal? It's pretty annoying. These are normal tasks that would not be too noisy for a normal phone microphone. I pretty much can't do anything while on the phone with my AirPods pros. If this isn't a problem for anyone else, I guess I'll contact Apple.