12.1 Wisdom of Crowds and Herd Thinking: Difference between revisions

From Sense & Sensibility & Science
(// Edit via Wikitext Extension for VSCode)
(// Edit via Wikitext Extension for VSCode)
Line 73: Line 73:


</tabber>
</tabber>
<restricted>


== Useful Resources ==
{{#restricted:{{12.1 Wisdom of Crowds and Herd Thinking}}}}{{NavCard|prev=11.2 When Is Science Suspect|next=12.2 Grill the Guest}}
 
<tabber>
 
|-|Lecture Video=
 
<br /><center><youtube>CngkRkOEVn4</youtube></center><br />
 
|-|Discussion Slides=
 
{{LinkCard
|url=https://docs.google.com/presentation/d/1bjUn2Pe7h3h2x9tBlFuH6hDY3KA4s5wyYcNr4XD7q-4/
|title=Discussion Slides Template
|description=The discussion slides for this lesson.
}}
<br />
 
|-|Handouts and Activities=
 
{{LinkCard
|url=https://docs.google.com/forms/d/1Q6EiPSUxdqShcn5ayhzlBy16Yo3_SbGZqzmRDgtzQjc/
|title=Estimates Survey Template
|description=Google form for the wisdom of crowds estimates activity.}}
{{LinkCard
|url=https://docs.google.com/spreadsheets/d/15VGm9-QMS998frqYhtXTfJ0dRJU0gbAyBvNt3JwDSSk/
|title=Estimates Spreadsheet Example
|description=Google sheet for the wisdom of crowds estimates activity.}}
{{LinkCardInternal
|url=:File:Kaplan and Miller - Reducing the Effects of Juror Bias.pdf
|title=Reducing the Effects of Juror Bias
|description=On jury bias and composition. Kaplan and Miller, 1978.}}
{{LinkCardInternal
|url=:File:Kerr et al. - 1985 - Effects of victim attractiveness, care and disfigu.pdf
|title=Effects of Victim Attractiveness, Care and Disfigurement on the Judgements of American and British Mock Jurors
|description=On jury bias and composition. Kerr, Bull, MacCoun and Rathborn, 1985.}}
{{LinkCardInternal
|url=:File:MacCoun - 1990 - The Emergence of Extralegal Bias During Jury Delib.pdf
|title=The Emergence of Extralegal Bias During Jury Deliberation
|description=On jury bias and composition. MacCoun, 1990.}}
{{LinkCardInternal
|url=:File:Kerr et al. - 1999 - Bias in Jurors vs Bias in Juries New Evidence fro.pdf
|title=Bias in Jurors vs Bias in Juries: New Evidence from the SDS Perspective
|description=On jury bias and composition. Kerr, Niedermeier and Kaplan, 1999.}}
<br />
 
|-|Readings and Assignments=
 
{{LinkCardInternal
|url=:File:The Social Psychology of the Wisdom of Crowds - Larrick.pdf
|title=The Social Psychology of the Wisdom of Crowds
|description=Richard Larrick, Albert Mannes, and Jack Soll on the wisdom of crowds.}}
<br />
 
</tabber>
 
== Recommended Outline ==
 
=== Before Class ===
 
Update the Google form in the [[#Warm-up Polls|warm-up polls]].
 
=== During Class ===
 
{| class="wikitable" style="margin-left: 0px; margin-right: auto;"
|5 Minutes
|Introduce the lesson and go over the plan for the day. Make sure people have groups, spokespeople, etc.
|-
|30 Minutes
|Have the class do the assorted [[#Warm-up Polls|warm-up polls]].
|-
|40 Minutes
|Walk the class through the various [[#Jury Discussion|jury discussions]].
|-
|5 Minutes
|Do the [[#Group Problem|group problem]] if time allows. This part is optional.
|}
 
== Lesson Content ==
 
=== Warm-up Polls ===
 
For this discussion section, students will start by filling out a Google form which asks for their best estimate of four quantities (should take 3-5 mins). The form should connect to a spreadsheet which shows how the class is or isn't aggregating live. Depending on the particular setup, this may take some spreadsheet finagling live during class. An example spreadsheet that was used in spring 2023 can be found below.
{{LinkCard
|url=https://docs.google.com/forms/d/1Q6EiPSUxdqShcn5ayhzlBy16Yo3_SbGZqzmRDgtzQjc/
|title=Estimates Survey Template
|description=Google form for the wisdom of crowds estimates activity.}}
{{LinkCard
|url=https://docs.google.com/spreadsheets/d/15VGm9-QMS998frqYhtXTfJ0dRJU0gbAyBvNt3JwDSSk/
|title=Estimates Spreadsheet Example
|description=Google sheet for the wisdom of crowds estimates activity.}}
==== Wisdom of Crowds Estimates ====
 
This first poll demonstrates the wisdom of crowds and what happens when there's a systematic bias in people's judgments. It has the following four numerical questions.
<ol start=1><li>What is the heaviest recorded weight for a feline as of 2013?</li></ol>
{{BoxAnswer|922 lbs}}
<ol start=2><li>What was the average weight for American men in 1960?</li></ol>
{{BoxAnswer|166 lbs}}
<ol start=3><li>What is the heaviest recorded weight for a rabbit as of 2013?</li></ol>
{{BoxAnswer|55 lbs}}
<ol start=4><li>What was the average weight for American women in 2010?</li></ol>
{{BoxAnswer|163 lbs}}
{{BoxTip|Make sure to copy the Google form and provide it to the students with a link and/or QR code.}}
{{BoxCaution|Make sure the students give their best guesses for each question ''without'' discussing with any classmates or looking any answers up. Don't reveal the answers until everyone's submitted their responses.}}
==== Famous Artist Poll ====
 
[[File:Mystery Painting.png|thumb|right|A painting made by... ''someone''.]]
This poll demonstrates the power of herd thinking in amplifying shared biases. Students are shown the painting on the right and asked which of the following artists created it.
# Claude Monet
# Andy Warhol
# {{Correct|Armand Guillaumin}}
# Eugene Boudin
{{BoxCaution|Note that the names are ordered this way so that Claude Monet gets asked before Armand Guillaumin. If the order is switched you may get an opposite effect where a slightly larger (or more confident) proportion of students voting for Armand Guillaumin inspires more students to herd with them. In this case, the power of expertise directing a group is demonstrated.}}
The poll is done in two rounds. In both cases, say the names in order and count how many students raise their hands for each one. Pause for a moment to give students a chance to get their hands up if they're slow (and to see what other people are doing).
# Ask the students to make their best guess, but don't require students to raise their hand.
# Force ''every'' student to make a guess. Tell the students that they ''all'' need to raise their hand for ''some'' choice even if they don't know the answer.
The correct answer is Armand Guillamin. However, it's likely the case that most people will pick Monet, because he is more widely known than the other options. This is an example of the ''negative'' outcome of herd thinking, where a group of people independently and collectively select the wrong answer because they all share the same prior bias (picking the famous painter). In the first case the bias ''only'' comes from people's tendency to pick familiar names. In the second, it may get amplified by seeing other more people raise their hands and then wanting to join in.
{{BoxCaution|This activity often does not work as intended. Students usually understand what it's intending to convey, but it's hard to get them to fall victim to it within the classroom.}}
==== Structure of Groups Poll ====
{{BoxCaution|Students may not know what a starling murmuration is unless you show/tell them.}}
The final warm-up is a quick poll (via hand raising is fine) to ask the students what the best way to answer each of the following questions is. Should it be done by deciding separately and averaging, by discussing as a group, or through some other means?
<ol start=1><li>How many starlings are in a murmuration?</li></ol>
{{BoxAnswer|Independent Guesses.}}
<ol start=2><li>Is the center of mass between the Moon and Earth inside or outside of the Earth?</li></ol>
{{BoxAnswer|Group Discussion. It's about a thousand miles below the surface of the Earth.}}
<ol start=3><li>A jury deliberation in a criminal trial?</li></ol>
{{BoxAnswer|It's complicated. We will go over this more in the [[#Jury Discussion|jury discussion]] next.}}
==== Takeaways ====
 
* The "wisdom of crowds" allows for the removal of statistical errors.
** When there's no "systematic bias" in the opinions of a group, then independent guesses without discussion can average out individuals' "statistical biases."
** Groups can come to surprising correct answers even ''without any expertise''.
** When there's expertise, but not systematic bias, often the "error bars" on the answer get smaller, but the average remains the same.
* The "wisdom of crowds" can become "herd thinking" in the case of systematic errors. But, it can also sometimes help.
** When there's no systematic bias, but the guesses aren't independent (i.e. the group discusses their answer), there's often "herding" around the earlier answers.
** When there's a systematic bias that is shared by many (but not all) members of the group, then discussion can ''sometimes'' dampen it.
** This depends a lot on the group dynamics we looked at in our lesson on conformity.
* But what about when expertise is required, the decision is multi-faceted, and there may also be biases present? In that case, things get more complicated...
 
=== Jury Discussion ===
 
==== What is a Jury? ====
 
Many of the students may be international or haven't served on a jury before. So, we start by giving them a quick reminder of juries' most important features (for the purposes of this course).
* It is meant to be a group of peers randomly selected from your community.
* A jury is charged with the responsibility of deciding whether, on the facts of the case, a person is guilty or not guilty of the offence for which he or she has been charged.
* The jury is supposed reach its verdict by considering only the evidence introduced in court and the directions of the judge.
* Attorneys can typically veto potential jurors that they think may have some bias against their side.
 
==== Jury Group Dynamics Discussion Questions ====
 
We start with some discussion questions about shared biases. These should be answerable given the takeaways from the [[#Warm-up Polls|previous activity]]. Remind the students that juries typically discuss the merits of the case as a group so as to come to a consensus on whether the defendant is guilty.
# What are the kinds of biases that might be present in a jury?
# In what situations might the biases in a jury get amplified by deliberation?
# In what circumstances might biases get averaged out?
Given the dynamics of conformity and obedience, the exact number of people and explicit decision making processes may also play a role in the decisions they make.
* A jury of 6 is constitutional, but a jury of 5 is not.
*: (''Williams v. Florida'', 1970) (''Ballew v. Georgia'', 1978)
* In a jury of 6, a decision must be unanimous.
*: (''Burch v. Louisiana'', 1979)
* In a jury of 12, a majority of 9:3 or 10:2 can be constitutional for non-capital offenses.
*: (''Johnson v. Louisiana'' and ''Apodaca v. Oregon'', 1972)
Given this, have the students answer the following questions.
# How might the dynamics of a jury of 6 with 1 dissenter (believes not guilty) be the same as or different from one of 12 with 2 dissenters?
# How might jury composition dampen or amplify shared or individual biases?
 
==== How Juries Help ====
 
[[File:How Juries Help.png|thumb|How juries help.]]
'''Kaplan & Miller (1978):''' Argued that deliberation helps juries reach a just evaluation of cases by averaging across different biases. The jury does this by arguing/debating the merits of the case. Strong/weak refers to the merits in the figure. The study varied obnoxiousness of attorney and which case was strong. Deliberation helps juries reach a just evaluation of cases by averaging across different biases. (Regardless of how obnoxious the attorney is, as long as they present the evidence.)
{{LinkCardInternal
|url=:File:Kaplan and Miller - Reducing the Effects of Juror Bias.pdf
|title=Reducing the Effects of Juror Bias
|description=On jury bias and composition. Kaplan and Miller, 1978.}}
==== How Juries Hurt ====
 
[[File:How Juries Hurt.png|thumb|How juries hurt. This figure is not actually from any of the papers linked on this page. It was kidnapped in cold blood from slides Rob MacCoun used in 2016.]]
'''Group Polarization:''' Group deliberation tends to exaggerate shared biases. For example…
 
'''MacCoun (1985, 1990):''' Juries tended to overweight a completely irrelevant consideration: the defendant's attractiveness.
 
Why did juries overweight attractiveness? Because it's a shared bias.
{{LinkCardInternal
|url=:File:Kerr et al. - 1985 - Effects of victim attractiveness, care and disfigu.pdf
|title=Effects of Victim Attractiveness, Care and Disfigurement on the Judgements of American and British Mock Jurors
|description=On jury bias and composition. Kerr, Bull, MacCoun and Rathborn, 1985.}}
{{LinkCardInternal
|url=:File:MacCoun - 1990 - The Emergence of Extralegal Bias During Jury Delib.pdf
|title=The Emergence of Extralegal Bias During Jury Deliberation
|description=On jury bias and composition. MacCoun, 1990.}}
==== How Juries Do Both ====
 
[[File:How Juries Do Both.png|thumb|How juries do both.]]
'''Kerr, Niedermeier, & Kaplan (1999):''' Groups magnify biases when evidence is ambiguous. Groups reduce bias when evidence is really strong or weak.
{{LinkCardInternal
|url=:File:Kerr et al. - 1999 - Bias in Jurors vs Bias in Juries New Evidence fro.pdf
|title=Bias in Jurors vs Bias in Juries: New Evidence from the SDS Perspective
|description=On jury bias and composition. Kerr, Niedermeier and Kaplan, 1999.}}
==== Jury Overview Discussion Questions ====
 
# What are the flaws of the jury system?
# Which components of the jury system are similar to herd thinking? What about wisdom of crowds?
# How do the flaws of the jury system diverge from the flaws of wisdom of crowds directly?
 
==== Building an Ideal Jury ====
 
Have the students discuss the following in small groups.
* Considering wisdom of crowds, herd thinking, and the other effects of group dynamics on decision making, how would you design a better jury system?
* Consider the whole process from selection of people (if you even have this?) to the final decision that's made. Assume there are no limits to your resources and control!
{{BoxAnswer|There's no one canonical answer to these questions. But, encourage the students to be ''really'' creative. Their solutions may include things like having multiple juries and judges debating in parallel, having the attorneys not know which side they're going to defend until the start of the trial, trying to find other ways to blind the jurors to irrelevant aspects of the case, etc.}}
=== Group Problem ===
 
The purpose of this is just to give one last final example of a case in which having group deliberation ''does'' help. Typically one person figures out the trick and everyone else understands once it's explained.
 
Jack is looking at Anne, but Anne is looking at George. Jack is married, but George is not. Is a married person looking at an unmarried person?
<ol style="list-style-type:lower-alpha">
    <li>{{Correct|Yes}}</li>
    <li>No</li>
    <li>Cannot be determined</li>
</ol>
{{BoxAnswer|title=Explanation|The correct answer is the first one. But, most people think it can't be determined. Consider the two cases. Either Anne is married or they're not. If Anne is married then Anne looking at George is the case of a married person looking at an unmarried person. If Anne is not married then Jack looking at Anne is the case of a married person looking at an unmarried person. Either way, it happens.}}<!-- == Overflow ==
 
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;">
<div style="font-weight:bold;line-height:1.6;">Extra content that's not currently part of the official lesson plan.</div>
<div class="mw-collapsible-content">
 
=== Fermi Problem ===
 
(Optional) As a final point, give the students an especially challenging Fermi problem to solve collectively. This is to emphasize that some problems ''do'' benefit from having deliberation at certain stages.
 
==== Instructions ====
 
# (11 min) Have the students work out step 1 as groups.
# (4 min) Have the students make their individual estimates and then average them together.
 
==== Step 1: The Tree ====
 
IKEA mainly produces two types of industrial bags. There are the blue bags that people can purchase and the yellow bags that are only used in the store. Besides the colors, these two types of bags are identical. As a group, figure out the quantities you need to estimate in order to determine the following:
 
# How many blue (purchasable) bags does IKEA produce per year?
# How many yellow (only for use in store) bags does IKEA produce per year?
# Are there more blue or yellow IKEA bags?
 
==== Step 2: Estimates ====
 
Now, using wisdom of crowds, have every person in the group estimate each quantity independently without discussion. Then, average the final values together to get a result that's (hopefully) better than if the estimates were just determined individually.
 
==== Discussion Question ====
 
# In terms of wisdom of crowds and herd thinking, why should the first step be collaborative, while the second step is individual?{{Answer|The first step involves detailed logical reasoning, and each person can contribute a small step in that process with little chance of bias. The second step involves the estimation of numbers, where an average of independent guesses is more accurate than a deliberated group consensus.}}
 
=== Changemaker ===
 
{{Changemaker|"Members of a homogeneous group rest somewhat assured that they will agree with one another; that they will understand one another's perspectives and beliefs; that they will be able to easily come to a consensus. But when members of a group notice that they are socially different from one another, they change their expectations. They anticipate differences of opinion and perspective. They assume they will need to work harder to come to a consensus. This logic helps to explain both the upside and the downside of social diversity: people work harder in diverse environments both cognitively and socially. They might not like it, but the hard work can lead to better outcomes."}}
 
{{Changemaker|How does homogeneity in groups contribute to herd thinking? What benefit does diversity therefore provide?}}
 
</div></div> --></restricted>{{NavCard|prev=11.2 When Is Science Suspect|next=12.2 Grill the Guest}}
[[Category:Lesson plans]]
[[Category:Lesson plans]]

Revision as of 00:14, 22 August 2023

Topic Cover - 12.1 Wisdom of Crowds and Herd Thinking.png

Explore ways that groups fall short of their optimal reasoning ability. There are better and worse ways to aggregate a group's knowledge.



The Lesson in Context

This lesson discusses when groups of people make better or worse judgments than individuals. It leads into the last part of the course, which revolves around group decision making, e.g. as a society.

Relation to Earlier Lessons
2.2 Systematic and Statistical UncertaintyTopic Icon - 2.2 Systematic and Statistical Uncertainty.png
  • Individual judgments can deviate from the truth due to systematic bias and random fluctuation. Reducing shared bias and increasing sample size are ways to improve the effects of Wisdom of Crowds.
9.2 BiasesTopic Icon - 9.2 Biases.png
  • Conformity to the group consensus, illustrated by the Asch experiment, and obedience to a perceived authority in a group, illustrated by the Milgram experiment, are psychological tendencies that affect group decision making, tending to increase herd thinking.
10.1 Confirmation BiasTopic Icon - 10.1 Confirmation Bias.png
  • Confirmation bias can exacerbate other biases in group decision making, such as the motivation to make judgments that conform to the group consensus or agree with the opinion of the authority figure - again, increasing problematic herd thinking.
Relation to Later Lessons
13.1 Denver Bullet StudyTopic Icon - 13.1 Denver Bullet Study.png
  • When making group decisions using the Denver Bullet Study method, it is important to survey each expert (on matters of fact) and each stakeholder (on matters of value) individually and independently so as to reduce "herd thinking" effects.
13.2 Deliberative PollingTopic Icon - 13.2 Deliberative Polling.png
  • In a somewhat opposite approach, deliberative polling encourages moderated discussion between all participants punctuated by dialogue with the relevant experts, before participants answer a poll to make informed decisions in society. Conformity with other participants as well as "obedience" to (in the sense of taking advice from) the expert panelists are essential features. The result tends to be a convergence of opinions towards moderation on divisive issues.


Takeaways

After this lesson, students should

  1. Not take for granted that consensus offers the best conclusions.
  2. Take seriously (but not as absolute!) the consensus of a group which has reasoned about a question in a careful, appropriate way.
  3. Take seriously (but not as absolute!) the average of a large group's independent estimates of a number, under appropriate conditions.
  4. Identify shared biases in a given group which may increase the odds of problematic herd thinking rather than helpful wisdom of crowds.

Wisdom of Crowds

Sometimes groups make better judgments than individuals. This happens when:
  • Judgments are genuinely independent, preventing herd thinking.
  • Members of the group do not share the same biases.
  • There are enough people in the group to balance out random biases or fluctuations (analogous to the need for an adequate sample size).
  • Works especially well when estimating a quantity, where errors may be large but are not systematic.

Herd Thinking

Sometimes groups make worse judgments than individuals. This happens when:
  • Judgments of individuals are influenced by the judgments of others, leading to groupthink and sometimes polarization.
  • Members of the group share biases, which can be exaggerated by discussion and cannot be decreased by averaging judgments.

Meta-analysis

A statistical analysis that aggregates the results from many independent studies addressing the same question. Each individual study is expected to have some error independent of the other studies. By combining these results this aggregate statistical error can hopefully be eliminated.

Family Dynamics

Within a group like a family, the parents may have chosen each other partly due to shared opinions and beliefs, and taught these to their children. The experience of strong consensus and absence of dissent within the family can cause the members to become more confident and more extreme in their beliefs (herd thinking), even if these opinions are quite different from a broader consensus in society. Moreover, it can be difficult for members to break out, since disagreeing with the group can be seen as disloyal, unethical, defiant, and ungrateful.

Congressional Committees

In Congress, most committees include both Republicans and Democrats. Ideally, this leads to groups whose members have different biases, helping to prevent herd thinking and leading to more considered policy proposals.

Scientific Community

In science, the ideal is a large community of people with varied biases who pursue investigations independently, and share their findings and ideas periodically. These features help the enterprise of science take advantage of the virtues of wisdom of crowds.

Insofar as the members of the scientific community share the same biases - for example, that they are not demographically representative of humanity at large - this can reduce the efficacy. For this reason, representation in science is important not only for ethical reasons of equity, but also for epistemic reasons.

Everyone agrees, so it must be true.

Sometimes everyone in a group agrees due to shared biases, sometimes combined with a charismatic or dominating figure who has convinced everyone else. Consensus can be a good indicator of accuracy, especially when biases are distributed and independent judgments are included, but it isn't a guarantee, especially when the group is missing key information.

I trust my guess about how many pages are in this book more than the average of my guess with other people's guesses, because even if I don't have more information than other people, it's MY guess and you should stick with your own guess.

If the goal is accuracy, more information is generally better. When other people's guesses don't share biases, as is typical with numeric guesses, averages of many independent guesses tend to be more accurate than any given randomly selected guess. Of course, if one person is an expert in some way, it may be worth weighting their answer more highly or even trusting their expertise.

Additional Content

You must be logged in to see this content.

⟨

Previous

11.2 When Is Science Suspect

⟩

Next

12.2 Grill the Guest

Retrieved from "https://sensibility.berkeley.edu/index.php?title=12.1_Wisdom_of_Crowds_and_Herd_Thinking&oldid=6303"