Coop média de Montréal
Journalisme indépendant
Can you support independent journalists with $5? donate today!
Lies, Damn Lies and La Presse Statistics
Blog posts reflect the views of their authors.
And crooked like La Presse poll results?
I support the student strike. I think other people should too. I'll argue with people who are in favour of tuition fee increases; I think it's a legitimate discussion though, and believe there are intelligent arguments in favor of the hike. I just don't agree with them.
I wanted to get that out of the way so there are no pretensions here. Each journalist has their opinion about the strike and fee increases, but we don't hear them because these opinions either a) are not supposed to exist or b) are not supposed to matter. I think there are ways to produce accurate and fair reporting even through your own bias; I try to do it myself.
But those biases are there, and we all have opinions. There is nothing more frustrating, though, than when opinions and biases are passed off as scientific fact.
And that appears to be what is splashed across pages two and three of La Presse today. While some La Presse columnists have been sympathetic to the strike, their editorial pages have not. This does not de facto deligitimize their coverage of the strike. But it does raise some serious questions.
Today's article states the the Quebec population are 60 per cent in favor of the tuition fee increase, while 60 per cent also believe the government should negotiate with students. Maybe those numbers are correct, and that would be disappointing, but that's fine. The issue, though, is that La Presse and polling agency CROP can't say whether or not the numbers are correct, and that is highly problematic.
In the fine print of the infographic they have showing the poll results, they state that:
Méthodologie: Le sondage a été mené en ligne auprès de 800 répondants les 28 et 29 mars derniers. Les résulats ont été pondérés afin de refléter la distribution de la population adulte du Québec selon le sexe, l'âge, la région et la langue maternelle des répondants. Compte tenu du chractère non probabiliste de l'échantillon, le calcul de la marge d'erreur ne s'applique pas. [emphasis mine]
This means that 800 people were surveyed online; the results were sorted to reflect the Quebec population, but it is a non-probabilistic sample. That's really important, because it means they cannot state the margin of error within their results (normally written along the lines of: these results are accurate within plus or minus 2%, 19 times out of 20). The reason they cannot state the margin of error is because the sample does not allow them to accurately state that these numbers are representative of the Quebec population's opinions. They can only say with any certainty that these numbers reflect the opinion of their sample group.
According to Statistics Canada:
Reliability cannot be measured in non-probability sampling; the only way to address data quality is to compare some of the survey results with available information about the population. Still, there is no assurance that the estimates will meet an acceptable level of error. Statisticians are reluctant to use these methods because there is no way to measure the precision of the resulting sample.
I wouldn't argue that we should place much faith in many polls; statistics are useful tools, but are often twisted. But in this case, La Presse is using numbers that cannot be accurately verified to pronounce upon what is one of the largest questions facing Quebec society right now.
I don't know the opinions of the two journalists who wrote this piece, and I don't adhere to the theory that editors are breathing down reporters' necks, ensuring that they publish stories towing the party line. But that doesn't eliminate the questions in my head about whether a similar survey presenting different results, but using the same method, would have either been minimized or even pulled.
Maybe that's extreme though, and leaning towards conspiracy theory. There's another reason to be concerned though: it's lazy, inaccurate and I'd argue irresponsible journalism. According to Statistics Canada, one of the reasons researchers use non-probabilistic sampling is because it is "quick, inexpensive and convenient." So there's just as strong a case to be made that this is a situation of a newspaper chasing a story, going for a scoop, and publishing numbers and a major article, the validity of which cannot be ascertained.
La Presse should pull the story, and agree to no longer use such studies when reporting on major policy issues.
About the poster
739 words
The site for the Montreal local of The Media Co-op has been archived and will no longer be updated. Please visit the main Media Co-op website to learn more about the organization.