You do not get a randomized study (as people think randomized connotes control and rigor), you get randomized group assignment. You only have control of assignment, and assignment is not the independent variable. The relationship between treatment (the IV) and outcome is not controlled. Modern observational studies often use propensity scores to achieve balance in the groups that serves as proxy for randomization, yet the study is still observational. Whether you randomize assignment or use propensity scores, you still end up with an observational study with regard to the relationship between IV and DV.

When you randomize groups to A and B and set them loose, then later look at an outcome, you are now in an observational study. You have no control of the groups, they will do what they want. You have no guarantees that group A follows the A protocol, etc. You can analyze it as an intention to treat model but that provides limited value.

This is not a knock on observational studies. My main point here is that when you think you have a randomized study, with randomization producing a degree of rigor and control, you're most likely not achieving that. People constantly say "randomized" and think it carries some magical wand of rigor and it doesn't.

quote:
Compliance rate is a hugely important factor in the success of any diet, probably the most important. You shouldn’t ignore that data. A study of just the compliant patients would also provide interesting data but introduces major bias.

Yes, compliance is important. I'm not suggesting you only study the compliant subjects. I'm saying that assuming you have rigor just because you randomize group assignment is going to lead to bias, which is going to be dangerous to the validity of your conclusions because you're not paying attention to it, because you've assumed that randomized group assignment took care of everything and it certainly does not.

You have to look at the data generating mechanism. If you don't, you are playing a dangerous game and won't even know it.

Tukey and others warned us about these things back in the 60s as computing power started to ramp up and people could ask a computer to crunch numbers. They knew that people would start slapping analysis schemes onto research problems without properly understanding the methods, how the data got to be what it is, and so forth. And that's exactly what has happened.