“The news audiences’ choices are influential in the context of South Korea and can affect which issues get widely distributed and receive attention. Also their choices can affect journalistic production and editorial news selection,” she said.
For that reason, while Deepnews created part of its training set using journalism school students to assess articles based on the “journalism school” definition of quality, Choi focused on members of the public to see whether they may hold the same ideas. They read 1,500 articles from Naver.com and rated them both for quality (1–10) and for “journalistic values” such as believability, readability, depth and sensationalism. Choi wondered if you can use machine learning to predict the quality of an article that the model hadn’t been trained on given the journalistic values that it shows. It turns out you can, and that the abstract ideas of “values “ are better predictors than “formal features” of articles such as the number of named entities it has or the length.
“We found that the stronger predictor of audience-rated news quality is journalistic values. Among them, believability, depth, and diversity were regarded more important than others. It suggests that news audiences, like journalists and journalism scholars, regard journalistic values highly as substantial factors in news quality,” she said.
The value of “sensationalism” had the lowest predictiveness.
This research is of course a little different from what Deepnews has done, which is to use a larger collection of articles more than 50,000, labelled with a quality score to predict a quality score for other articles, rather than use one set of scores to predict another. However, it is further proof that there is a concept of journalistic quality that people understand and that it can be measured algorithmically.
Choi’s research also means something for the question at the beginning of this post, about how we get our news. If quality can be grasped algorithmically and people can understand it, then perhaps we should start expecting (or demanding) that news platforms incorporate the idea into their systems in addition to engagement.
“I think the primary logic of how the digital news platforms are operating should be revised by considering the quality of the news articles, considering democratic public discussions, the quality of public discussions,” Choi said. “I think it’s important to consider the news quality in their algorithm design.”
In the future different news sites and aggregators may be using a several algorithms to categorize different news articles according to different criteria such as quality but also others like those journalistic values, how scientific an article is or something else. Twitter CEO Jack Dorsey has fantasized about an “app store for social media algorithms” that would give people the choice over what kind of things they are being shown, rather than being pushed articles selected for them by a centralized algorithm that benefits one company.
Ten years from now, we may look back on using primarily clicks to categorize news online like someone trying to measure air quality using a thermometer.
P.S: If you’re interested in a piece with background on the situation in Australia, our very first blog post for Deepnews (September last year) spoke with Prof. Rob Nicholls of University of New South Wales on all things antitrust.