Can you trust what you read about climate change?

In January 2016, the final temperature measurements from the previous year were coming in from around the world. “2015 was hottest year in historical record, scientists say” declared a headline in The New York Times. The very same week, a headline from Forbes read “2015 was not even close to hottest year on record.”

In a world where the answer to a yes or no question like “Was it hotter last year than ever before?” is up for debate, how can you know what to believe?

Emmanuel Vincent and colleagues at the Center for Climate Communication at UC Merced had an idea: What if scientists could review online articles about climate change for accuracy?

Their online project, Climate Feedback, has made this a reality, powered by a growing group of Ph.D.s from around the world who volunteer their time to review articles on climate change in the mainstream media. There are currently over 100 scientists in the ranks at Climate Feedback, hailing from institutions all over the globe, with expertise in climate science and related topics including public health, food security, forestry and marine biology.

Fixing climate journalism

At a time when portions of the the public are increasingly hostile to experts of all sorts, is throwing more scientists at the problem going to help?

Vincent is optimistic. A Yale study from 2011 found that more than 75 percent of Americans trust scientists on climate change, while less than 40 percent trust journalists or politicians.

“People really want to hear from scientists. People care about who is speaking,” he said. “Climate Feedback is opening a new channel of communication for scientists to engage on their own terms with authors and readers.”

The biggest challenge for Climate Feedback is scale: there’s so much misinformation and hyperbole being published daily that 100 scientists volunteering a fraction of their time can’t possibly keep up.

“We look for big news – things that have the power to influence public opinion,” said Vincent. “And we only provide feedbacks to articles that claim to be based in science.”

While some individual readers will use Climate Feedback to check the accuracy of articles they read, Vincent thinks the real power will come from the ability to encourage better reporting from journalists and help inform editorial decisions by publishers.

This is why Vincent and his colleagues settled on the word “feedback”: their goal is to improve the accuracy of science journalism, not to point fingers or further entrench sides on what has become a highly polarizing topic.

They’ve seen some success. After being contacted by Climate Feedback, The Telegraph issued a correction to a story entitled “Earth heading for ‘mini ice age’ within 15 years,” removing some of the more alarming and unsubstantiated claims in the story (but keeping the historic photo of policemen ice skating on the frozen Thames).

In another case, meteorologist and writer Eric Holthaus took his own Rolling Stone-published article “The point of no return: Climate change nightmares are already here” to Climate Feedback for a critique. Links to original research had been removed during the editing process, and wording had been changed in a way that overstated what was supported by the science. Rolling Stone took the feedback from Climate Feedback reviewers into account and added links back into the article.

In search of credibility

Each article that Climate Feedback takes on receives a “scientific credibility rating” based on scoring provided by reviewers, ranging from -2.0 (“very low”) to +2.0 (“very high”). In addition, the experts provide direct feedback using web annotation, so a reader can see comments in the context of the article and follow citations to see the basis for the criticisms.

With apps like Genius, internet users can add annotations to nearly any page on the internet – essentially comments for pages, even if they don’t allow comments. Climate Feedback builds on this approach, using the open-source web annotation software Hypothes.is, but the annotations only come from experts with academic credentials.

“It’s not just about facts, but also the reasoning,” said Vincent. “By reading scientists’ comments, readers can learn about scientific thinking.”

More than 95 percent of actively publishing climate scientists agree that human activities are directly influencing the climate, but that doesn’t mean there isn’t room for nuance and disagreement.

“In many cases there is consensus among the reviewers, in others not,” said Vincent. “Articles will often touch on several topics, and may be accurate in one area, but not in another. This is why having the annotations in the context of the article is so powerful.”

Both the New York Times and the Forbes stories on 2015 global temperatures were put to a panel of Climate Feedback experts around the world.

The New York Times story received a scientific credibility rating of “very high” (+1.9) from eight scientists. One reviewer described it as “A very nice, well-written, and factual article, including comment from some of the globe’s leading climate scientists,” and others echoed this sentiment.

The Forbes story received a scientific credibility rating of “very low” (-1.8) from ten scientists, who cited numerous inaccuracies, misleading statements that appeared intentional, and even derogatory name calling. “Clearly an attempt to discredit science,” said one reviewer.

Forbes did not respond to our request for comment and has not acted in response to Climate Feedback’s reviews.

Vincent pointed out that Climate Feedback reviewers have been critical of both articles that downplay the impacts of climate change and the ones that exaggerate them, when the claims are not supported by science. For instance, an article published by The Independent claimed that global warming was causing the jet stream to cross the equator with dire consequences. Scientists that reviewed the article found the claims to be overly alarmist and not defensible.

“What drives scientists is the motivation to promote more accurate coverage in their field of expertise and to help readers identify information they can trust that is consistent with current scientific knowledge,” said Vincent.

Patterns in the media

For many digital publishers today, web traffic and social media buzz trump accuracy. In the quest for viral web content, shocking and counterintuitive claims often outperform factual reporting.

In the same week of January, Forbes also published “Climate scientists report that 2015 was the hottest year on record” by a staff editor, mirroring the reporting by The New York Times. This article has been read just over 2,600 times at the time of writing. The article that asserted the opposite has been read nearly 50,000 times.

Another Forbes article from 2015, “Updated NASA data: Global warming not causing any polar ice retreat,” has been read over 1 million times. This article received the lowest possible rating (-2.0) from the nine scientists that reviewed the article on Climate Feedback.

As Climate Feedback has started building up a body of article reviews, patterns are emerging: Some news sources consistently produce accurate stories on climate change, while others are consistently inaccurate.

To help point readers to trustworthy publications and to incentivize better reporting by the mainstream media, Climate Feedback is working on a new feature called the “Scientific Trust Tracker”, aided by a recent successful crowdfunding campaign that raised $10,000 by the end of the first day that will allow Climate Feedback to roll out new features and take on more articles.

Beyond climate

Vincent has found that the articles taken on by Climate Feedback often stray into other topics, some that can be equally controversial and politically charged, like nuclear energy or genetically modified food crops.

“This is a model that can be applied to lots of other topics – anything that is confusing, hard to understand or controversial,” said Vincent. “The same approach could be adapted to renewable energy or vaccines, for example.”

A group of scientists led by Brittany Anderton from the Institute for Food and Agricultural Literacy at UC Davis are piloting a new site, Biotech Feedback, an attempt to help people make informed decisions in the face of often exaggerated and conflicting reporting on biotechnology in the media.

“This is adapted to the time we are living in,” said Vincent. “People find it very hard to know what is credible right now.”