For those unfamiliar with the Bechdel test (sometimes referred to as the Bechdel-Wallace test), it originates from an 1980s comic strip by Alison Bechdel, which posits three criteria for a movie to pass. The criteria are
This is, of course, not a rigorous scientific test, but it has gained mainstream traction as a measure of gender representation in film.
Given that the test is primarily about female representation, I was curious whether women actually rate Bechdel-passing movies higher. If representation matters to women, we might expect movies with stronger female characters and storylines to be rated higher by women. Equally interesting is the effect on male audiences. If increased female representation does enhance women’s perception of the movie, as reflected by the rating, does it come at the cost of men’s ratings?
While I couldn’t find any direct analysis of this particular question, there have been a few analyses of the Bechdel test in relation to various metrics.
In a FiveThirtyEight article they analyzed the relationship between movies passing the test and their profits and budgets. They found that women tend to be cast in lower budget movies and also evidence that passing the test has higher return on investment. I have a couple of problems with this analysis. First of all, while being careful to not state it directly, they heavily imply causation in their conclusion
“…while this might be a side effect of films with lower budgets tending to have higher returns on investment than films with higher budgets, it’s still a strong indicator that films with women in somewhat prominent roles are performing well.”
This is carefully worded and it is true, but there is no reason to believe that the movie performs better due to more women in the cast, based on this analysis. It could simply be a side effect of lower budget movies both performing well regardless of the number of women AND lower budget movies for some reason having more women in the cast.
They get this finding by running a regression on the return on investment, that is ROI = Revenue/Budget. They also state that they control for budget, but this is problematic since budget is used to calculate their dependent variable (ROI). Including budget as an independent variable when it appears in the denominator of ROI creates a mechanical relationship that can cause spurious correlations to appear. This is particularly problematic in this context because they also report that Bechdel-passing movies have 16% lower budgets on average. Since ROI is calculated as Revenue/Budget, movies with lower budgets will mechanically appear to have higher ROI even with identical revenue. In other words, their definition of the dependent variable mathematically biases the results in favor of finding that Bechdel-passing movies perform better. While I can’t say for sure, it is possible that their finding is entirely dependent on this methodological oversight.
Brian C. Keegan has also criticized FiveThirtyEight’s analysis for lack of methodological transparency, though the ROI-budget issue appears to have gone unnoticed.
In another article by Steven Follows, he analyzed whether movies passing the Bechdel test are rated higher by both audiences as well as critics. He used ratings from IMDb, Rotten Tomatoes and Metacritic and found that movies that do not pass the test are rated higher by both critics and audiences alike. He states that he doesn’t know the reason for this, but theorizes a bit.
While there might be a number of reasons, I think the major one is simply that the movies most likely to be rated highly are often about extreme conditions and life and death situations (war movies, westerns etc.), which have been, and largely still are, a predominantly male domain for mostly biological reasons.
There are a few things the analysis by Steven Follows does not account for, most importantly gender of the rater. For one, women generally rate movies higher than men, but they are also heavily outnumbered on sites such as IMDb, which could impact his findings. Since the Bechdel test is largely about representation, presumably for the benefit of women more so than men, I thought it would be interesting to see if women as a group rate movies that pass the test higher than ones that don’t, as well as how it affects the ratings by men.
Since IMDb breaks down user ratings in demographics by both gender and age, it will be the foundation for assessing how gender affects the rating. I used the IMDb API and decided to limit the movies by only going back to 1990 ending at 2020. I also limited it to movies with more than 10,000 votes and movies that listed English as a language.
This resulted in a total of 4,530 titles with an average rating of 6.44 and an average of 102,302 votes (82,572 for men and 19,730 for women). Women tend to rate movies higher than men, which is also the case for these titles, where the average for men is 6.44 and 6.62 for women.
For the Bechdel scores I downloaded the dataset from the site bechdeltest.com, which is a user driven aggregator that lists more than 8000 titles rated on a scale 0-3 on how many of the criteria a movie passes.
The final dataset after having joined the two consists of 3,221 titles (the difference being movies present in only one of the two datasets) with distribution of the Bechdel scores as such:
Figure: The distribution of the bechdel scores in the dataset.
Unfortunately the ratings on IMDb are not just the arithmetic means of user ratings. They use a formula to calculate the ratings and for whatever reason are very secretive about it. I use the ratings as they are, but depending how they are calculated, it might affect the result. I’ll assume that it does not have an effect.
It is well known that men on average rate movies on IMDb lower than women, but since I’m mainly interested in how women rate movies depending on how they do on the Bechdel test, I’m only comparing within group, so the difference in rating between male and female voters doesn’t really matter.
To assess whether passing the Bechdel test influences IMDb ratings, I use ordinary least squares regression with demographic-specific ratings (all ages) as the dependent variable, analyzed separately for male and female raters. The Bechdel test is coded as a binary variable: 1 for passing (rating of 3) and 0 for failing (ratings of 0, 1, or 2). I chose the binary approach because the pass/fail distinction is the primary point of interest. Treating them as individual steps on the scale does not change the outcome, but simply preserves some information about how much each step influences the IMDb rating, which we aren’t particularly interested in.
I control for potentially confounding variables including year of release, runtime, and genre. Since films can have multiple genres, I encode them as dummy variables (one-hot), allowing each film to be associated with multiple genre categories.
To test the robustness of the findings, I also do a bootstrap analysis with 1,000 iterations. For each iteration, I draw equal-sized samples (1,000 films each) from the Bechdel-passing and failing groups using sampling with replacement. This stratified sampling approach tests whether the observed relationships are stable across different sample compositions and not just a consequence of specific films included in the full dataset. For each bootstrap sample, I run the same regression and record the coefficient and p-value for the Bechdel variable. I then aggregated these results to assess the reliability of the results.
To get a sense of the data, I plotted it by age demographic as a violin plot as can be seen in the figure.
We can observe that women rate movies higher across all age demographics except for the under 18 group. Apparently teenage girls are harsh critics. I wouldn’t put too much stock into this since this age group has a very low vote count for many movies, and the difference is most likely not significant. We can also observe that for the other age groups, even for movies not passing any of the Bechdel test criteria, women still rate them higher on average. It is also observable that as the Bechdel score increases, very little happens to the mean rating of the movies.
| Demographic | Coefficient | Std. Error | P-value | 95% Confidence Interval | Significant |
|---|---|---|---|---|---|
| Female | 0.0470 | 0.026 | 0.076 | (-0.005, 0.099) | No |
| Male | -0.0706 | 0.029 | 0.013 | (-0.127, -0.015) | Yes |
The results show that for women, Bechdel-passing films are rated slightly higher (+0.047 points), but this effect is not statistically significant (p = 0.076). The confidence interval includes zero, reflecting uncertainty about the direction and magnitude of any effect. This indicates that there is no reliable association between Bechdel score and how women rate movies.
For men, the relationship is negative and statistically significant (p = 0.013). Bechdel-passing films receive ratings that are 0.071 points lower on average. Here the relationship is consistent, with the confidence interval not including zero.
Both effects are very small in magnitude, considering it is a 10-point scale. This is particularly evident when compared to genre effects, which is further discussed below.
As described in the method section, I did a bootstrapped robustness test by repeatedly sampling and running regressions on these samples. This resulted in the following:
| Demographic | Mean Coefficient | Std. Dev | 95% CI | % Significant | Direction Stability |
|---|---|---|---|---|---|
| Female | 0.0453 | 0.0207 | (0.0039, 0.0852) | 17.0% | 98.6% positive |
| Male | -0.0719 | 0.0222 | (-0.1148, -0.0295) | 51.4% | 100% negative |
The bootstrap analysis reveals that both effects are directionally stable but very small relative to sampling variability:
Both effects are near the threshold of reliable detection. Bechdel scores appear to have minimal influence on ratings, with the female effect particularly weak and the male effect small.
Both regressions have R-squared values of about 0.30, meaning the models explain roughly 30% of the variation in IMDb ratings. That is a meaningful amount, but it also leaves a large share of the variation unexplained, which is to be expected for noisy, user-generated ratings.
The lack of significant findings suggests that the Bechdel score does not influence how women rate movies. One possible explanation is that the Bechdel test sets a relatively low bar that doesn’t capture meaningful representation. A movie can pass by having two named female characters briefly discuss something as mundane as the weather, while still lacking well-developed female characters or compelling storylines. If women care more about the quality of representation rather than mere presence, the Bechdel test may fail to capture this.
The significant negative coefficients for male ratings imply that higher Bechdel scores correlate with lower ratings from the male demographic. This could indicate that male audiences are less inclined to rate movies positively when they have characteristics that typically pass the Bechdel test. It is of course also possible, as stated earlier, that the types of films men enjoy are not generally suited for a larger more prominent female cast. It’s also possible that Bechdel-passing movies differ systematically in ways unrelated to representation (genre conventions, narrative structure etc.) that affect their appeal to male audiences.
The bootstrap analysis reveals important nuances about the stability of these findings. While both effects are directionally consistent across resampled datasets, they are very small relative to sampling variability. The female effect is particularly weak—close to the noise floor with substantial variation in the estimated magnitude. The male effect is somewhat more stable but still modest. Neither effect is large enough to be reliably detected without favorable sampling conditions.
To put the magnitude of this effect on male ratings in perspective, the coefficient of -0.0706 indicates that Bechdel-passing films are rated 0.07 points lower by males—less than 1% of the 10-point scale. This is a very modest effect. Female representation, as measured by the Bechdel test, does not appear to meaningfully affect IMDb ratings when controlling for various confounders.
Genre effects were among the strongest predictors of ratings across both demographics, often substantially larger than the Bechdel effect itself. Animation films received approximately 0.75 points higher ratings from women and 0.70 points higher from men compared to the baseline, while Horror films were rated 0.45 points lower by women and 0.41 points lower by men (see Appendix for full genre coefficients). These effect sizes are much larger than the Bechdel coefficient of -0.07 for males, underscoring why controlling for genre is essential. Certain highly-rated genres such as Action, War, and Thriller are also less likely to pass the Bechdel test, which could partially explain why prior analyses that did not account for genre or rater demographics found that Bechdel-passing movies received lower ratings overall.
The major limitation of this analysis is the absence of budget data as a control variable. This is of course somewhat ironic given my criticism of the FiveThirtyEight study for their problematic use of budget, when I don’t even include it. Budget likely influences both movie ratings and the probability of passing the Bechdel test, so its absence may introduce confounding that affects the results.
Finally, the standard disclaimer about correlation not implying causation applies. This analysis is observational and can only establish correlations, not causal relationships. While we observe that higher Bechdel scores are associated with lower male ratings, we cannot conclude that female representation directly causes men to rate movies lower. There could be underlying confounding variables driving the effect, which is not unearthed here.
The analysis shows that the Bechdel score, despite being a measure intended to promote female representation, does not significantly impact how women rate movies. This finding is somewhat ironic, as the Bechdel test’s primary purpose is to encourage more inclusive, representative content that might be expected to resonate more with female viewers. Instead, the Bechdel score’s significant effects are observed only in the male demographic, where higher scores are associated with lower ratings.
I may redo the analysis at a later time where I include budget, but at this point collecting the data would be too much of a hassle, since there is no single readily available dataset to get it from.
R-squared: 0.303 Adj. R-squared: 0.298
| Independent Variable | Coefficient | Std. Error | t-value | P-value | 95% Confidence Interval |
|---|---|---|---|---|---|
| const | 18.7596 | 3.417 | 5.490 | 0.000 | (12.060, 25.459) |
| bechdel_pass | 0.0470 | 0.026 | 1.774 | 0.076 | (-0.005, 0.099) |
| year_x | -0.0068 | 0.002 | -4.002 | 0.000 | (-0.010, -0.003) |
| runtime | 0.0148 | 0.001 | 19.601 | 0.000 | (0.013, 0.016) |
| Action | -0.1979 | 0.038 | -5.184 | 0.000 | (-0.273, -0.123) |
| Adventure | -0.0165 | 0.041 | -0.399 | 0.690 | (-0.097, 0.064) |
| Animation | 0.7482 | 0.062 | 12.059 | 0.000 | (0.627, 0.870) |
| Biography | 0.2398 | 0.054 | 4.451 | 0.000 | (0.134, 0.345) |
| Comedy | -0.1107 | 0.039 | -2.855 | 0.004 | (-0.187, -0.035) |
| Crime | 0.0053 | 0.039 | 0.135 | 0.893 | (-0.071, 0.082) |
| Drama | 0.3149 | 0.037 | 8.621 | 0.000 | (0.243, 0.387) |
| Family | -0.0770 | 0.059 | -1.295 | 0.196 | (-0.194, 0.040) |
| Fantasy | -0.1261 | 0.048 | -2.653 | 0.008 | (-0.219, -0.033) |
| History | -0.1956 | 0.081 | -2.418 | 0.016 | (-0.354, -0.037) |
| Horror | -0.4536 | 0.050 | -9.136 | 0.000 | (-0.551, -0.356) |
| Music | 0.0176 | 0.082 | 0.214 | 0.830 | (-0.144, 0.179) |
| Musical | 0.0291 | 0.135 | 0.215 | 0.830 | (-0.236, 0.294) |
| Mystery | -0.1038 | 0.047 | -2.218 | 0.027 | (-0.196, -0.012) |
| Romance | -0.1412 | 0.042 | -3.361 | 0.001 | (-0.224, -0.059) |
| Sci-Fi | -0.0542 | 0.050 | -1.087 | 0.277 | (-0.152, 0.044) |
| Sport | -0.0200 | 0.109 | -0.184 | 0.854 | (-0.234, 0.194) |
| Thriller | -0.0347 | 0.043 | -0.808 | 0.419 | (-0.119, 0.049) |
| War | 0.1271 | 0.129 | 0.982 | 0.326 | (-0.127, 0.381) |
| Western | -0.0367 | 0.176 | -0.208 | 0.835 | (-0.382, 0.309) |
R-squared: 0.292 Adj. R-squared: 0.287
| Independent Variable | Coefficient | Std. Error | t-value | P-value | 95% Confidence Interval |
|---|---|---|---|---|---|
| const | 18.3336 | 3.681 | 4.981 | 0.000 | (11.117, 25.551) |
| bechdel_pass | -0.0706 | 0.029 | -2.476 | 0.013 | (-0.127, -0.015) |
| year_x | -0.0067 | 0.002 | -3.642 | 0.000 | (-0.010, -0.003) |
| runtime | 0.0158 | 0.001 | 19.400 | 0.000 | (0.014, 0.017) |
| Action | -0.2449 | 0.041 | -5.958 | 0.000 | (-0.326, -0.164) |
| Adventure | -0.1114 | 0.044 | -2.506 | 0.012 | (-0.199, -0.024) |
| Animation | 0.7050 | 0.067 | 10.548 | 0.000 | (0.574, 0.836) |
| Biography | 0.1627 | 0.058 | 2.803 | 0.005 | (0.049, 0.276) |
| Comedy | -0.1206 | 0.042 | -2.887 | 0.004 | (-0.203, -0.039) |
| Crime | 0.0348 | 0.042 | 0.826 | 0.409 | (-0.048, 0.117) |
| Drama | 0.3378 | 0.039 | 8.584 | 0.000 | (0.261, 0.415) |
| Family | -0.3666 | 0.064 | -5.723 | 0.000 | (-0.492, -0.241) |
| Fantasy | -0.2219 | 0.051 | -4.332 | 0.000 | (-0.322, -0.121) |
| History | -0.2395 | 0.087 | -2.749 | 0.006 | (-0.410, -0.069) |
| Horror | -0.4102 | 0.053 | -7.670 | 0.000 | (-0.515, -0.305) |
| Music | -0.1462 | 0.089 | -1.651 | 0.099 | (-0.320, 0.027) |
| Musical | -0.2207 | 0.146 | -1.514 | 0.130 | (-0.506, 0.065) |
| Mystery | -0.0971 | 0.050 | -1.926 | 0.054 | (-0.196, 0.002) |
| Romance | -0.2864 | 0.045 | -6.329 | 0.000 | (-0.375, -0.198) |
| Sci-Fi | -0.0166 | 0.054 | -0.310 | 0.757 | (-0.122, 0.089) |
| Sport | -0.0622 | 0.117 | -0.530 | 0.596 | (-0.292, 0.168) |
| Thriller | -0.0188 | 0.046 | -0.407 | 0.684 | (-0.109, 0.072) |
| War | 0.0453 | 0.139 | 0.325 | 0.745 | (-0.228, 0.319) |
| Western | 0.0117 | 0.190 | 0.062 | 0.951 | (-0.361, 0.384) |
