Most of the articles I have read for my literature review are quantitative. I am used to dealing with percentages, means, correlations and chi-square tests. I know that the result is significant when p is smaller than 0.05. I can handle these basics.

But I have a problem… When statistics get more sophisticated and terms like multiple regression, effect size, post hoc analysis and MANCOVA start appearing, I get lost.

Recently I found a meta-analysis by Bernard and colleagues (2009), and I read the following:

In addition, the linear association between treatment strength and effect size was significant (β_Regression[1, 73] = .09, p = .01, Q_Regression = 6.66, p = .01).

Ok… So… What does that mean? Seriously, what does that mean? It’s a single sentence, but I find it so hard to understand!

Here is what I know:

• There is a p, smaller than .01… Whatever they did, it was significant, important.
• Treatment strength is one of the variables they are considering. They are talking about an association (linear… how many types of associations are there, anyway?) between one variable and an effect size.

Here is what I do not know:

• Is the effect size another variable?
• What is  Q_Regression?
• What is β_Regression?
• If they are talking about one association, why is there a β_Regression and a Q_Regression?
• Why does β_Regression have numbers between brackets and Q_Regression does not?
• What are those numbers between the brackets (1, 73)?

And that is just one sentence! Now try reading a whole paragraph!

We found that the overall unadjusted average effect size of 0.10 was significantly different from zero, z(73) = 3.52, p < .001, and significantly heterogeneous, Q_T(73) = 209.86, p < .001. We then looked to see whether the variability might be explained by methodological quality. The scores on the methodological quality scale for these achievement data ranged from 6 to 14 and produced the frequency distribution of scale categories shown in Table 3. The scale categories significantly explained effect size, Q_B(8) = 30.30, p < .001 and, treated as an ordinal scale in regression, significantly predicted effect size (β_Regression[1, 73] =.08, p < .001, Q_Regression = 23.50, p < .001). Based on this analysis, we decided to classify the scale into three larger categories of methodological quality.

Do you understand that? Ok. Maybe I am being unfair. I am not providing you the context of the study. In my case, reading the whole study didn’t help much. I still don’t get it.

When I am reading a very quantitative study, like a meta-analysis, I usually skip the numerical babble and jump down directly to the end. I know that in the discussion and conclusions, the author will translate all those numbers and statistical terms into words that I can actually understand.

However, I am aware of the problems of doing this. Since I don’t understand half of the results, I have to trust in the correctness of the conclusions. Maybe there was a methodological problem. Maybe one of the assumptions of the studied variables was not met, making the results untrustworthy. Maybe the author stated something without having enough evidence to support it. Maybe. But I wouldn’t know.

I am a PhD student, and I don’t understand statistical studies.

Ok, I said it.

Funny enough, it is not easy to find training about this. I have taken several statistic courses… But they were not as specialized as what I need now. Fortunately, I still have two more years to figure this out and learn…

– Brenda Padilla

Reference

Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A. & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243-1289.