The single most shocking thing about this study

David Alwin

I have been mulling over the NEJM cervical cancer/circumcision study with growing anger and disbelief. My statistical analysis skills are quite rusty, but I get the impression that a knowledgeable statistician could blow it out of the water. I think we need professional (statistical) help here, and that there is a reasonable potential for embarrassing the NEJM, possibly even forcing a retraction of the whole article. We also need tactical skill to decide the best way to publicize our criticisms. Sorry about this incredibly long post. Justhit delete if you aren't up for it.

To refresh my statistics, I found some useful information at:

 http://faculty.vassar.edu/lowry/VassarStats.html

In particular, the following web page allows the calculation of odds ratios in a 2x2 contingency table (odds ratios are used extensively in the NEJM study):

 http://faculty.vassar.edu/lowry/odds2x2.html

You can plug in your own numbers to check their math (so far I haven't found a single number in agreement…) One drawback with these web-based stats calculators is the inability to calculate Fisher's exact test for samples of greater than 100. For that, I downloaded a trial version of Analyse-it which is a plug in for Excel tables.

The single most shocking thing about this study is that 5.3% of the men were knowingly included in the wrong category with regard to their circumcision status. That's right!!! For the most part, the study used the self-reported circumcision status of the male partner to classify the partners into groups. In a subsample of the couples (97.4 percent of the men from the Brazil, Thailand, and Philippines studies) the circumcision status of the male partner was verified by a physician. They found a discrepancy between the self-reported and actual circumcision status in 42 of these 794 men (the physicians were uncertain in 4 more men). Five men who reported themselves to be circumcised were not, and 37 men who reported themselves as uncircumcised were in fact found to be circumcised.

You would think that for the subsequent statistical analyses, the researchers would use the actual physician-verified circumcision status. Inexplicably, this is not the case. The authors state the self-reported circumcision status was used for all analyses.

The authors give us a hint as to why they elected to use this strange practice of using wrong data when the correct data were known---it weakens their findings. They report the overall adjusted odds ratio for penile HPV infections according to self-reported circumcision was 0.37 (95 percent confidence interval, 0.16-0.85) whereas the adjusted odds ratio associated with clinician-assessed circumcision was 0.44 (95 percent confidence interval 0.17-1.13). Not only is the odds ratio less impressive using the real numbers, but the confidence interval for physician verified numbers includes 1.0; in other words, the result is no longer significant.

I find this to be intellectual dishonesty and a demonstration of bad faith on the part of the researchers. Using self-reported circumcision status might be forgiven if there were no alternative. However, when the actual circumcision status was known, the researchers were obligated to use the most correct data available in their analyses, in my opinion.

Could it make a difference to the findings? Absolutely. I will get back to this below.

Overall, out of 1913 couples in the study, 19.6% of the women with cervical cancer (194/977) had circumcised partners, whereas in those women without cervical cancer, only 18.8% (176/936) were circumcised. In other words, women with cervical cancer were slightly more likely to have a circumcised partner overall, before further statistical chicanery. Expressed in terms of uncircumcised (intact—I know, I know) partners, 80.1% of women with cervical cancer had uncircumcised partners whereas 81.2% of women without cervical cancer had uncircumcised partners. This difference is not significant by my calculation of Fishers exact test (odds ratio 1.07; confidence interval 0.85-1.34). This is using the self-reported (erroneous) data.

Here are the numbers for cervical cancer according to the self-reported circumcision status of the partner (from Table 3) (Fisher exact test calculated using Analyse-It for Excel):

Overall Cervical Cancer

Now, we know that 42 men in the above table are misclassified (i.e. in the wrong box above). Specifically, 5 need to move from the circumcised to the uncircumcised row and 37 need to move from the uncircumcised to the circumcised row. In order to see what the maximum impact of this misclassification could be, let us move 5 from circumcised to uncircumcised in the cervical cancer negative column and move 37 from uncircumcised to circumcised in the cervical cancer positive column (of course it is unlikely that the real data would be this skewed). This gives the following results:

Cervical Cancer

Now the odds ratio shows an even greater likelihood that a woman with a circumcised partner will have cervical cancer. The odds ratio becomes 1.385 and the confidence interval no longer includes 1, which means a significant result, i.e. women with circumcised partners are significantly more likely to have cervical cancer.

The validity of pulling numbers from other studies to test a hypothesis which they were not designed to test is questionable. I believe this is called post-hoc analysis (my stats courses were some time ago…). Well actually, it is not even that, since post-hoc analysis applies to testing for other hypotheses than the study was originally designed for in the case of one's own single study. Here we have an analysis of data pulled from seven other studies. The other studies were not designed to test the hypothesis that there is an association between circumcision status and cervical cancer. In addition, the countries studied had very different underlying rates of HPV infection ranging from 5.5% of the men studied who were positive in Philippines to 37.6% positive in Brazil. The majority of the circumcised men in the study happened to come from Philippines (65.4% of the circumcised men in the study) where the overall rate of HPV was the lowest, nearly half the rate of the second lowest country (Spain at 10.8% HPV positive overall). There may have been other studies done in other geographical locations with far different numbers, which were not included in the analysis. Since the authors have demonstrated their bad faith by knowingly analysing erroneous data, how are we to know that the particular studies chosen were not culled from a larger pool specifically to show a particular result?

The odds ratios were adjusted to account for other variables such as age, location, education, and so on. Usually these adjustments are applied when at first glance there seems to be a large correlation between two variables, in order to account for some of this apparent correlation. So, when the odds ratios are adjusted, an apparent correlation usually becomes smaller, and what at first appears to be a strong correlation may be entirely explained away by one or more of these co-factors, so that no real correlation between the two variablesunder study remains.

In the case of this study, we start out with no apparent correlation between circumcision status and cervical cancer, and in fact with a slightly greater risk for cervical cancer in the partners of circumcised men. The odds ratio is 1.07 (see table above). After applying an adjustment for the co-factors, the odds ratio shown in Table 3 for the same numbers is given as 0.72. The confidence interval is 0.49-1.04. We are not given the actual numbers or formula for the adjustment or the adjusted result. Thus, the correlation nearly reaches significance in the opposite direction (i.e. greater risk of cervical cancer in the partners of uncircumcised men) as the odds ratio applied to the actual numbers without adjustment. I am not sure that this is a valid way of trying to show correlation between two variables. It implies that by finding more and more co-factors (which may be strategically chosen), one could eventually show a correlation between to variables where none would seem to exist by simply looking at the overall correlation between them. If the data presented in the study were true, this would also imply that one or more of the co-factors varies strongly in the direction opposite circumcision with regard to cervical cancer (i.e. increases risk), yet also tends to vary in the same direction as circumcision (i.e. more common in circumcised), so that when its effect is not separated out, there appears to be no overall effect of circumcision on cervical cancer. This would be a three-way interaction, so that Factor X is more likely to be seen in circumcised men yet also more likely to be seen in couples where the women has cervical cancer. This could then explain the fact that without allowing for Factor X, there appears to be no association between circumcision and cervical cancer, but when the effect of Factor X is accounted for, the true relationship between circumcision and cervical cancer is revealed. What could Factor X be? It would have to be something more likely to be seen in the circumcised group, and also more likely to be associated with cervical cancer. In the study, the circumcised men were considered subjectively to have better genital hygiene overall, for example (apparently, smegma is pathological…). Could better genital hygiene be associated with a greater risk of cancer? If this were the case, it could in fact obscure the relationship between circumcision and cervical cancer. The circumcised men's better genital hygiene exposes their partners to a higher risk of cervical cancer, which was counterbalanced by the lower risk of cervical cancer due to their having been circumcised (still with me?) Should we start getting this message out to the population? Well, if this variable does not intuitively fit the bill for Factor X, the circumcised men also tended to be younger, more educated, to have started sex at a later age, had fewer sexual partners, less sex with prostitutes, and more condom use, especially with prostitutes. In other words, could we further reduce the risk for cervical cancer by advocating sex with more partners, preferably prostitutes, preferably without a condom, preferably beginning at a younger age, etc? This obviously seems counterintuitive.

My point is that there is no trend toward an association between circumcision and cervical cancer using the overall pooled data, but according to the researchers, there is a strong trend which becomes evident when applying an unspecified adjustment to the data to account for cofactors that intuitively would not seem to be likely candidates for increasing the risk of cervical cancer. It would be nice to have the actualdata here.

To be fair, there is another co-factor called genital washing after intercourse, which in fact does vary in the direction necessary for a putative Factor X--- it is much less common for circumcised men to wash after sex according to the data. So perhaps not washing after sex, as is more commonly the case for circumcised men also increases the risk for cervical cancer. (Sound bite: genital washing after intercourse has more effect on risk of cervical cancer than circumcision status). This might make sense intuitively, but it is hard to place much weight on the validity of this factor. Again, it is a self-reported variable, with two categories: Always versus never or occasionally. No attempt was made to quantify the meanings of the categories, or validate the couples' self-report. It is hard to believe that this cofactor could have such a powerful impact.

The other cofactor I have not mentioned in the context of a Factor X is study location, but this cofactor had the most significant interaction with circumcision status of all. It is not a continuous variable, like most of the others (e.g. more/less sexual partners, etc.), but obviously could have an impact. In order to reveal an association between circumcision status and cervical cancer that was not seen in the overall pooled data, it would be necessary for the likelihood of coming from Country Z to be greater both in the partners of women with cervical cancer and also among the circumcised men. Since the studies were case-controlled, each country contributed roughly equal numbers of women with and without cervical cancer. So Country Z would have to have contributed a higher proportion of women with cervical cancer having circumcised partners, in order to obscure the true relationship between circumcision and cervical cancer. This is not the case. Each country has a slightly higher percentage of uncircumcised partners of women with cervical cancer as opposed to controls.

So, overall, the pooled data revealed no relationship between circumcision status and cervical cancer whatsoever. I calculate an odds ratio of 1.07—about as close to no association as you can get. The study reports a strong trend toward less risk of cervical cancer in partners of circumcised men with an adjusted odds ratio of 0.72, very close to being significant (CI 0.49 to 1.04). In order for the adjustment to reveal a trend that was not there in the pooled data, the co-factors used to adjust the odds ratio would have to vary in the direction of the so-called Factor X discussed above, which does not seem intuitively possible.

The only finding which reaches significance is in a subgroup analysis. Taking only those women who had only one male sexual partner (purportedly to decrease variance caused by women possibly contracting HPV and cervical cancer from a previous partner), they further divided the couples into those where the male partner had sex with fewer than six other partners or six or more partners. They report an adjusted odds ratio of 0.42 (CI 0.23-0.79) for the group of men with >= 6 partners. However, when I plug in the numbers given in Table 4, I get an unadjusted odds ratio of 0.981 (CI 0.681-1.413), again just about as close to no relationship as you can get. The numbers from Table 4 for men with >=6 lifetime sexual partners are shown below.

Cervical cancer

So where does this adjusted odds ratio that is significant come from? Bear in mind that the numbers above still include some of the men known to have been misclassified. At this point, it is not stated how many of these men are left in this subgroup analysis, although with ever smaller numbers in each cell, switching a few men from one cell to another could have a huge impact, as I showed above. The adjustment for co-factors does not seem to have an obvious Factor X as described above that varies in a way so as to obscure a possible relationship between circumcision and cervical cancer.

So, we are not given data to support what is billed as the most significant finding of all, the one which was played up in the editorial and the media. The adjusted odds ratio for the subgroup appears as if out of thin air and we are supposed to trust the authors' calculations, when careful scrutiny of the co-factors used does not point in the direction of a co-factor varying in the manner it would have to in order to make sucha large change in the odds ratio.

This is only a beginning of the criticisms I believe can be levelled at this study, but I wanted to get at least a few of my concerns out there. Other issues would be claiming to show a causal relationship using correlational data (a statistical red flag), etc. Hopefully, someone with a good statistical background can point out to me where I went wrong in my analysis. Someone with a good statistical background can probably find other problems with the study that I have not noticed.

I would be in favor of trying to enlist a professional statistician's aid in bulletproofing our case and trying to exploit the weaknesses in this study to the maximum.

The Circumcision Information and Resource Pages are a not-for-profit educational resource and library. IntactiWiki hosts this website but is not responsible for the content of this site. CIRP makes documents available without charge, for informational purposes only. The contents of this site are not intended to replace the professional medical or legal advice of a licensed practitioner.

  © CIRP.org 1996-2024 | Filetree | Please visit our sponsor and host:  IntactiWiki.