Meritocracy? Might want to re-think how you define merit.

Rock on!
You might think if you put together a lot of smart people, you’d get a smart group, but new research into group intelligence shows that’s not always the case. (For those of you who don’t have access to online journal subscriptions through your local library or university, there are more details in the Carnegie Mellon University press release.)

What we found is that the intelligence of the team members was not significantly related to the collective intelligence, either positively or negatively.


Our first observation and the one that surprised us the most was that the proportion of females in the group seemed to be strongly predictive of the collective intelligence of the group.

However, when they looked more closely they realised that it wasn’t the gender that mattered, but rather the social sensitivity of the group members (previous studies had shown that women tend to score more highly in social sensitivity).

It’s not the intelligence of the group members that matters; it’s their social sensitivity.

So the more your group members were socially sensitive, the better the group performed in measures of collective intelligence. The key here was that group members need to collaborate, and to do that they needed those social skills to help them work together. This includes some different conversational patterns: groups where one or two people dominated conversations exhibited low collective intelligence, while groups where more people contributed had higher collective intelligence.

This scientific research is potentially a big blow to the standard “meritocracy works” theory often espoused in open source and computing groups. Standard meritocracy rules say you do clever things and you get accepted, and this will make for perfectly good teams. But given that there’s often bias that dismisses “soft skills,” it turns out that folk may actually be using typical geek meritocracy rules to weed out some of the people we need to make the group most effective as a whole.

Some of my female colleagues would like to conclude that you simply just need to hire more women. While that might be easier, what it really suggests is that you need to pay attention to what people refer to as these “softer skills” and thinking about who’s going to be a good team player, not necessarily focused solely on individual achievement, individual accomplishments.

So if you want to claim that the best way to build tech teams is meritocracy… you might want to think more carefully about how you define merit.

Rock show DS

The quotes in this article are drawn from Bob McDonald’s conversation with Dr. Anita Williams Woolley, the lead author, on the Quirks and Quarks interview aired October 9. You can download the podcast of the segment on collective intelligence here.

23 thoughts on “Meritocracy? Might want to re-think how you define merit.

  1. Aoede

    Well, that’s only sensible. If the position is one in a team, then teamwork is part of the job description, and therefore demonstrated merit in teamwork ought to be a factor in hiring decisions. Different story if it’s more of a solo gig, of course, but I’m given to understand that groups are overwhelmingly more common.

  2. jon

    It’s a great study! I just blogged about it in Hold that thought … totally agree with your conclusion that’s a good reason to rethink how people typically define merit.

    The gender aspects are difficult to tease out. Based on ihatewasabi’s summary here, they found three separate things were significantly correlated with their measure of collective intelligence: the average social sensitivity of group members, as measured by the “Reading the mind in the eyes test”; equal participation in speaking; and the proportion of females in the group. Women in the sample score better on this measure of social sensitivity than men (consistent with other research), so after doing regression analysis the researchers conclude that this is the most important factor.

    That said, if women tend to be better at this key attribute for team success, then it seems to me that Dr. Woolley’s colleagues are right: hiring more women will lead to better team performance. She’s right too of course: paying attention to softer skills such as making sure everybody on the team gets to participate, and rethinking how much we value traditional ‘rock star’ individual excellence, is also vital. But the results are consistent with the underlying model in Scott Page’s work: diverse teams outperform if they can work together effectively. So I think it’s worth highlighting that aspect in its own right as well, and explicitly including diversity when we talk about “meritocracy”.


    1. Terri

      Just to restate here because I suspect it’s easy to mis-read:

      The proportion of women did NOT matter, once social sensitivity of participants was taken into account. Gender was not found to be a predictive factor by itself.

      (Jon grouped gender in with the two factors which actually were predictive: social sensitivity and equal participation in speaking. I just want to be clear that gender was NOT a relevant finding for this study, just an interesting artifact.)

      1. jon

        Here’s how the paper describes it, where “c” is the measure of collective intelligence:

        Finally, c was positively and significantly correlated with the proportion of females in the group (r = .23, P = .007). However, this result appears to be largely mediated by social sensitivity (Sobel z = 1.93, P = .03), since (consistent with previous research) women in our sample scored better on the social sensitivity measure than men; t(441) = 3.42, P = .001. In a regression analysis with the groups for which all three variables (social sensitivity, speaking turn variance, and percent female) were available, all had similar predictive power for c, though only social sensitivity reached statistical significance (β = .33, P = .05) (12).

        > gender was NOT a relevant finding for this study, just an interesting artifact

        I wouldn’t describe it that way. Their results show that the percentage of women in a group has predictive power for “collective intelligence” which in turn predicts group success at solving problems. I’m certainly not taking anything away from the importance of the other results, and obviously more follow-on work is needed (the researchers have said they weren’t looking at gender specifically and so hadn’t constructed the study to look at it) but when I sent the link to Scott Page his initial response was “wow” and that’s pretty much how I feel.


        1. Terri

          In the interview I linked, the lead researcher was very careful to note that although there was a correlation, it was not the gender that mattered so much as the social sensitivity. That is, if you compare men and women of equivalent social sensitivity, then the gender difference is not significant. It’s a subtle statistical point, but she really went out of her way to try to make it clear, so I felt I should note it as well.

          An example to try to explain:

          Suppose you’re looking at weather reports and the number of people playing tennis outdoors at your club, and you find that any time the temperature is below 20C, very few tennis players go out to play. So initially, you conclude that tennis players don’t like cooler days. But a bit of investigation over the period shows that most days where it was below 20C, it also rained. It turns out people can’t play safely on the wet courts, and what seemed to be outliers where many people played on cooler days are actually just the days it didn’t rain. So while it may be true that fewer tennis players play on cooler days, the real result is that tennis players don’t play in the rain, which happens to correspond with many cooler days.

          The fact that it’s the rain and not the temperature that matters isn’t a big deal in practical terms — you could probably still schedule fewer staff on weeks where the forecast is cold based on this knowledge, just as it’s probably “good enough” to suggest that ideal teams will have more women — but it does make a difference to a statistician or someone who’s concerned about properly representing their results.

  3. Carolyn

    Actually, I think it has less to do with social sensitivity and more to do with the diversity of tools that everyone brings to the group. It makes sense to me that if you have a group with more women, the group is probably more diverse in general.

    There’s a great video from 2007 of Scott Page discussing this topic in mathematical terms here:

    1. Terri

      Once again, the gender was NOT found to matter so much as the social sensitivity, and it seemed like higher in all members was better there which suggests participants who have more in common… and actually sort of runs counter to the suggestion that diversity is key factor here.

      I vaguely recall studies on diversity supporting your assertion, but that’s not a finding of this particular study.

      Edit: Another (clearer?) way to say this: If women have higher social sensitivity, on average, this study actually implies that an ideal team could be filled mostly with women. Not so diverse!

      1. Restructure!

        Thank you for re-emphasizing to the commenters that gender and social sensitivity are different. I hate it when people assume that I am better at dealing with people because I’m a/the woman, when people outside tech think of me as the opposite.

  4. Andrew

    Software is a fundamentally individual task. With a few exceptions (Extreme programming’s pair-programming exercise) it almost has to be. Other people in the room are a distraction, they disturb your mental picture of the computer. This is why programmers speak of “stuckage and flow”. Yes, developers have to talk to each other and coordinate but this tests technical communication far more than it tests the social sensitivity as described in the broadcast.

    This goes doubly so for the open source community. In a typical project, you might never meet the other developers in person. Your ability to read non-verbal cues (how they measured social sensitivity in the broadcast) doesn’t matter because on the mailing list and on IRC there are no nonverbal cues.

    Since these tests place the groups in the same room and require constant interaction, I think they are a poor indicator of what would work for writing software.

    One of the other biggest factors the broadcast mentioned that groups that were willing to take input from all of their members. This has long been part of hacker culture, specifically because hacker culture is so individualistic. Take a recent ESR blog post for example:
    The ideas of the young ESR, an unknown programmer at a small UNIX shop, where seriously considered by the authorities at the IETF.

    “If I were a egotist I’d claim to have saved the world, maybe. But what really mattered is that I threw my disruption into a roomful of hackers with an innate distrust of hierarchy and a strong allergy to system designs with single-point vulnerability. It was that shared culture that made the difference, more than me – it produced my objection, and it produced people ready to hear it.”

    I think that hacker culture’s value on technical meritocracy and this innate distrust of hierarchy are one and the same. By advocating for some type of social skills over technical talent you are advocating against this core value in the hacker community. Which also happens to be something that the authors of that study found to be quite valuable for group intelligence. The idea that anyone in the room, even the n00b, might have the idea that saves the world.

    1. Mary

      In a typical project, you might never meet the other developers in person. Your ability to read non-verbal cues (how they measured social sensitivity in the broadcast) doesn’t matter because on the mailing list and on IRC there are no nonverbal cues.

      This is only one model of Open Source development, and I’m honestly not sure how dominant it is. To my knowledge, the following projects heavily use periodic in person meetings and conferences:

      the Linux kernel (Kernel Summit, Plumbers, sometimes, undoubtedly others)
      many Python projects have sprints around the PyCon events, which is a big part of why there are so many PyCons
      the Canonical sponsored Open Source projects (Ubuntu, Launchpad, Bazaar) have important planning and collaboration meetings every 3 to 6 months

      There are definitely also projects where the developers seldom or never meet or communicate outside written mediums, but I call {citation needed} on this being “typical” given the high-profile complex successful projects that find in-person collaboration useful.

      1. Andrew

        I think my point stands. There’s a huge difference between a task where constant and continuous collaboration is required and the software engineering world as it stands today. Yes, these conferences exist and I presume that they are quite useful. But, these are not where the day-to-day work gets done. I think it’s largely the same way in corporate environments too. Everyone might meet up to decide on what to do next, but then everyone goes back to their office to get coding on their own.

        1. Mary

          I think my point stands.

          I don’t agree.

          OK, specific points where I am sympathetic to scepticism:
          * I agree that software engineering is not right at the constant collaboration all the time end of the spectrum
          * caution is generally called when attempting to generalise reults of psych studies. It’s almost certain that this finding doesn’t map perfectly onto software development, it may not even map well or at all.

          But arguing that the mapping should be rejected on the basis that software development and especially Open Source is very solitary isn’t a convincing argument. Software dev is not necessarily, and I think perhaps not even often, very solitary work, although solitary work is part of it and is one of the productive elements. Coding is, for many people, a solitary activity (although pair programming etc is popular for a reason, it’s not an exceptional case). The overall job/work of a software engineer, including in many Open Source projects is quite collaborative.

          Anecdata: I live with a professional Open Source developer, and I am doing a PhD project. He does far more collaboration than I do, including user support on IRC, mentoring other developers, reviewing patches, design arguments, discussions about technical tools, conferences etc etc. I meet with a supervisor for 15 minutes a week and give talks every six months. (I did corporate software dev for a time, and interacted with co-workers far more then than I do now in a research setting.)

      2. Skud

        There are also lots of non-verbal cues in online communication. It’s naive to think otherwise.

        Like, imagine this IRC conversation:

        person1: hey did you get the patch i sent?
        person2: yeah
        person1: are you gonna apply it?

        And then silence.

        What would you make of that? How would you respond? How long would you wait before you bugged them again? Would you do a /whois and check their idle time, see if they just went afk or whether they’re just not talking to *you*? When you next contact them, would it be on open channel, or in /msg, or maybe by email? Why might you choose one over the other?

        This whole “online communication is all 100% literal and doesn’t require any empathy or social skills” thing is bullshit.

        1. Andrew

          But, that’s not how this particular study measured ‘social sensitivity’. Their procedure involved showing the precipitants photograph of human faces. That being said, you have valid point, I will concede that there are some non-verbal cues in online communication. I’m just not convinced that the findings of this study necessarily follow to software development on any level.

        2. Terri

          Andrew also says, “But, that’s not how this particular study measured “social sensitivity’. Their procedure involved showing the precipitants photograph of human faces.

          I did wonder how long it would be before someone brought this up. So yes, good catch. And if you were willing to extrapolate some, you might suggest that this is a reason that open source software has fewer women than industry, as the facetime in industry is usually much higher then in open source. Be hard to prove, but an interesting thought.

          However, if you take a look into other studies of social sensitivity you’ll find that the social sensitivity measure they used (human facial empathy) is a measure that in previous other studies has been found to be predictive of other forms of empathy and social awareness. This is simply an easy predictive measure that saves time putting people through a larger battery of testing. Some of the things this measure predicts include reading of verbal and written social clues that would be relevant even in email or IRC.

          So yes, although on the surface facial empathy seems unlikely as a measure, this is actually a well-tested measure that does predict other types of social awareness and empathy, and is likely to be predictive for non-in-person interactions. I’m guessing most researchers would hypothesize similar results if the groups were to collaborate online, although it would likely make for a good study anyhow.

          (I actually debated putting a note about this in the original post, but didn’t have time to look for appropriate citations. However, this was covered in one of my 2nd year psych courses when I was an undergraduate, so I doubt it would be had to find if anyone’s curious.)

    2. Terri

      Andrew claims, “Software is a fundamentally individual task.

      I think that on a small personal scale, you’re totally right. But software isn’t something observed only on a small personal scale anymore: Larger scale architectures require interoperating parts, and the APIs, network protocols, standards, etc. that allow communication between these “software islands” are as much a part of software as the individual coder’s library. (You could claim that “software” ends at the code contributed by an individual while “architecture” or “systems” take into account the rest, but that’s pretty much a tautology and not really interesting.)

      So I guess what I’m saying is that software is both a fundamentally individual task and a fundamentally collaborative task, depending on how you look at it. Sometimes it’s useful to just build this library, sometimes it’s much more important to see how that fits into the picture, and interact with others to ensure that it fits well.

      As a related note, among many communities including the security folk and the usability folk, it’s considered a considerable sign of intellectual immaturity if you refuse take the larger systems and architecture into account when looking at software. I don’t mean this exactly as an insult, but more of a commentary of how cultural ways of viewing software differ depending upon your goals and intellectual background.

      1. jon

        Totally agree with Mary’s and Terri’s points about software engineering. One-person software projects and solitary programming are great, but pair programming is extremely effective — and most open-source and commercial software is developed by teams.

        Very helpful perspectives on the ‘social sensitivity’ measurement used in the study, especially for those of us who aren’t specialists in that area. I agree with the points about social cues in online discussions; it’d be very interesting to see where these other measurements do and don’t align. Anybody know if that research exists?

    3. spz

      The idea that the newbie (or unknown) will actually be heard in all hacker cultures / Open Source software projects does not coincide with my experience. Brushing off none too gently if the mail is not praises sung and flattery brought is not that much of an unknown.

      In my experience it’s the “one alpha male” projects that fail most egregiously at this, corroborating the studies findings. :)

  5. John

    I don’t have paywall access, so I’m only going by the writeups… but I’m wondering whether the test participants recruited would be typical (particularly in terms of individual intelligence) for software developers, who I think tend to be of significantly above-average individual intelligence. I can understand there being little correlation with the average ability of members for groups who average wasn’t very far from the population average (which is presumably the range psychology experimenters try to recruit from unless investigating a particular kind of person).
    It’s an interesting study, but I’d be careful about applying it unmodified to such an unusual situation as software development, which is one of the most intellectually complex tasks people do.
    It might also vary between cultures.

    1. Terri

      I haven’t looked in detail at their participants for this particular study, but for these studies the participants are often recruited through and associated with the universities involved, which indicates a certain level of academic success, and often a moderately affluent background — things which are also fairly common for software developers.

      Note that they also found the groups of high collective intelligence performed well across the tasks they gave them. There’s a good chance that some of those tasks would indeed be relevant to software development skills.

  6. John

    I’m sure social sensitivity can be learnt, but can it be taught?

    If it can, perhaps it should be taught as part of CS degrees.

Comments are closed.