Civic IQ 2.0: Civility and Solutions
At Artefact, we’ve been thinking a lot about the state of online discussions. These days it’s almost a cliché to make fun of how bad how bad online commenting has become – like when John Oliver called on the “trolls of the internet” as if summoning demons from the depths of hell to flame the FCC’s discussion on Net Neutrality. But when we read studies that show that comments are not only bad on their own but also negatively distort people’s perception of issues, it becomes clear that this is no laughing matter.
So we’ve been asking ourselves, is it possible to make online commenting more civil and constructive? Or is it just human nature that it devolves into a vitriolic cesspool? Thanks to support from the Knight Foundation, we set about tackling this challenge by building upon Civic IQ 1.0, our previous idea for a more open government.
We started looking at multiple online discussions and by leveraging our prior research, we analyzed the underlying behavioral cues in these forums. This led to ideation of several concepts around different hypotheses about how to best support and encourage more civil discussions. We created simple paper prototypes that allowed us to elicit feedback on diverging ideas. Three intriguing insights emerged:
- Anonymity is the enemy of civility: Real names and profiles are powerful tools to encourage civility. Disclosing who you are makes you think twice before posting nasty comments.
- Familiar features increase engagement: Any time we tried to add more novel structure – such as a literal debate system or a branching mindmap of discussions – it introduced barriers to engagement. The key, we learned, was to keep it simple, carefully balancing familiarity with new features and structure.
- Language has power: We experimented with different labels and prompts for commenting categories and were encouraged to hear that the way we label different sections of the tool impacts people’s behavior. One of the comments we heard frequently was, “Just the fact that you labeled the commenting window a ‘thought’ made me think more deeply about my response.”
Just build it
Armed with these initial insights, and following an agile and iterative approach, we began building Civic IQ even as we continued to test and evolve the design. Today, the alpha Civic IQ media plug in has four key attributes:
- Behavioral nudges are integrated into Civic IQ to increase civility:
Mandatory sign-in with real-names via Facebook
Clearly stated ‘rules of conduct’
Civic IQ points for positive behavior
Tagging posts as helpful or unhelpful
Democratized spam filtering
- A lightweight structure for categorized comments gives users specific prompts to nudge them into the right mindset:
Ideas for sharing or discussing a solution
Questions for probing deeper into an issue and eliciting more relevant information
Thoughts for supporting general discussion but with more “thoughtful” language and perspective
- Discussion highlights bubble-up the most interesting and valuable contributions to increase engagement and provide an understanding of the overall discussion
- A Likert scale voting system and visualization shows the support or controversy around an idea. We encourage ideation by enabling idea improvements and alternative to proposed solutions.
Validation at scale
In order to properly test commenting at scale, we knew we had to efficiently engage hundreds of people as part of the discussion, but how? A normal focus group would be too slow, costly, and biased. And we couldn’t properly recruit an open public study with enough control. We devised a study using Amazon’s Mechanical Turk. We pit our tool against an industry standard commenting platform in an A/B test in which the participants were assigned, one of the two commenting solutions. After a brief pilot, we launched the test, in which 784 people participated, generating 897 comments. After the discussions were complete, a second group of Mechanical Turk reviewers categorized the data (stripped of their respective tools and formatting) to help us identify key differences across the nature of the comments in the two plugins. We also conducted an analysis of basic data about the discussions (number of votes, replies, and threads) to understand if, and how, they differed.
Based on the independent evaluation, Civic IQ performed above the industry standard across most key areas:
- Civility: Civic IQ comments contained fewer vulgarities or attacks on people or ideas.
- Constructive questions: Civic IQ comments were twice as likely to contain constructive questions that helped move the conversation forward and there were significantly more comments that recognized a new point of view in Civic IQ, including phrases like, “I hadn’t considered that before…”
- Engagement: We saw more voting activity per post and longer threads on Civic IQ.
Our hypothesis that Civic IQ will also encourage more ideation was initially unclear by the study. However, when we ran a follow-up pilot with a different article that more clearly defined the problem and implied a call to action, solution generation was 50% higher on Civic IQ. While encouraging, this finding indicates that as we continue to build Civic IQ, we need to consider tools that help with problem definition as well as ideation.
Realizing a vision
Why is this important? Imagine if this tool were available today on the FCC’s site for commenting on Net Neutrality – perhaps then a solution would be easier to find. Or, perhaps a tool like Civic IQ, with its potential to make comments more civil and constructive could convince Popular Science to reverse its decision to shut down commenting on their site. That in of itself is an ambitious goal, but there is an even bigger opportunity.
We’ve been inspired by studies which have demonstrated the problem-solving power of large, diverse groups of people. If we conservatively say that just 5% of online posts are serious, that’s still millions of valuable discussions currently being lost in the noise of the internet, never reaching any consensus or conclusion. What if as Civic IQ gets adopted by media, becomes the hub to gather the best of these conversations, problems, and ideas from around the web, and turn them into action?
In just a couple of months, we built a working prototype of a media plugin that is more civil and solution-oriented than the industry standard. But there is more work to be done. We have a lot of ideas how to further improve Civic IQ, especially around enabling discussions to identify and re-frame problems and support and encourage further iteration of ideas. Our goal is to refine the design and optimize the code to create a turnkey solution freely available to any site or anyone in need of a better commenting platform.