Monday, April 14, 2014

Critical thinking and standardized tests

Lawyer Barry Arrington has posted a link to an article that compares scores on standardized tests (LSAT) with undergraduate discipline. It also looked at university GPA. That article is: The best prospective law students read Homer. Look at the chart below.

It's not a big shock to see that the average GPA of religion and classics students is higher than that of biology and engineering students (Y-axis). It's interesting that students specializing in biology perform slightly better than religion students on the LSAT (X-axis) but these differences aren't very significant.


Barry Arrington thinks they are significant. His post at Biology Students Score Below Religion and Classics Students on Test of Critical Thinking makes the following claim ...
One wonders why biology students do so poorly while classics and religion students do so well. One hypothesis: classics and religion students learn critical thinking skills while biology students are taught to parrot the central dogma. The chart is from a study of which undergraduate majors correlated most highly with success on the LSAT.
Barry is making the false assumption that scores on the LSAT correlate with the ability to think critically. I suppose it's natural for a lawyer to think like this.

My experience indicates that one of the serious downsides to teaching critical thinking is that it hurts the students' chances of doing well on standardized tests such as the LSAT, MCAT, and GRE. Those tests are usually set up to encourage memorization and regurgitation even though some of the questions look like "think" questions. That's why I tell students to always give the standard, expected, answer on the MCAT even if they know it's wrong or misleading.


12 comments :

  1. My score on the LSAT (the law school aptitude test) got me a full scholarship to law school, and I am here to tell you its relation to "critical thinking" is equivalent to or perhaps less than most children's puzzles on paper placemats in diners.

    But besides that, here's one of many significant items Arrington misses, either through lack of careful consideration (critical thinking) or on purpose: Classics in particular, and religion as well, are humanities majors. If these majors are followed up with a professional graduate education, law is the traditional one. On the other hand, a biology major is often an indication that the student intends to follow up with medical school. So among biology majors taking the law school aptitude test, there will be a fair number who did not have the grades or test scores to get into medical school. In other words, this is an assessment not of all biology majors, but of a cohort skewed toward those who didn't do well enough in school to attain their most desired career choice. The fact that even "failed" biology majors do almost as well on this test as people who have been pointing for law school for a long time says something quite uncomplimentary about the classics and religion majors.

    ReplyDelete
  2. Isn't Arrington's post good evidence against his claim, or at least against his critical thinking skills? He appears to give no thought to the units on his graph, or to either the magnitude or significance of the differences. And why should the GPAs of different majors even be considered comparable, much less indicative of critical thinking skill?

    By the way, I don't think the GRE or LSAT are at all about regurgitation and rote memorization. Why would you think so? The MCAT is to a fair degree except for the critical reading section.

    ReplyDelete
  3. He's also, amusingly, confounding dogma with the (tongue in cheek) Central Dogma of molecular biology.

    ReplyDelete
  4. There are not that many Catholic seminaries left in the US. You can get in with a 2.5 GPA. This is a lower score than Kansas wants for its prison guards.

    http://seminarygradschool.com/article/Busting-the-Myths-of-Applying-to-Seminary

    What's the honest answer to 'why did this person become a priest?'. Most of the time it's 'because they couldn't get higher than a C in high school'.

    Clearly there are some smart priests. But it's not something they select for.

    ReplyDelete
  5. Those tests are usually set up to encourage memorization and regurgitation even though some of the questions look like "think" questions.

    Yes. I remember that (at least in the early 1990s when I took it) the GRE had these "logic" questions -- "Bill lives in a green house; Joe lives next to a yellow house", etc. and when I first saw them on the practice tests I thought they really tested logic. But then I saw in a test-preparation booklet a nearly automatic way of just filling in a chart to solve them. Once you've seen that no thinking was really involved.

    ReplyDelete
    Replies
    1. Sure, no thinking is involved. But neither is it regurgitation. There's very little memorization involved in the GRE. The only thing I can think of would be the vocabulary questions.

      Delete
    2. There is an extremely instructive section in the book "None of the Above" about the SAT and "aptitude" tests more generally, written by a fellow named David Owen (a former editor of The Atlantic). It's a reading comprehension section of four questions drawn from an actual SAT test.

      It was trivial to get all four reading comprehension questions correct. Oh, but here's the kicker: The reading selection one is supposed to comprehend in order to answer the questions has been left out.

      It's really a fascinating illustration of the fact that a great deal of performance on such tests can be gained by picking up on implicit cues the question drafters have left regarding which answers are right and which are wrong. The more you share a certain background and way of thinking with the question drafters, the more obvious these implicit cues are. The less you share that background, the more of a handicap you start with. It may not be so stark as to make it more likely that those who don't share the background will get the question wrong. But people who don't have to waste time with two or three answers per question in hundreds of four-answer multiple choice questions on a timed test have an obvious time advantage.

      Delete
  6. Its all about identity. Classics/religion at this stage brings in those types of people interested in more intellectual things.
    The other subjects bring in those more motivated for prestiges jobs or jobs they really desire.
    its reasonable it taps into those demographics that grew up with a more intellectual background in their circles.
    so a little sharper possibly shown by these marks.
    Anyways it is all about memorization and that as the tool for faster better learning.
    The memorization coming from motivated people and so generally people from particular backgrounds.
    i don't think these things are a test of a roll of the dice about whether religious prople are smarter then biological students.
    So much more to it.

    ReplyDelete
  7. You comprehend that people who major in religion are not automatically religious, right? (Why am I even asking?)

    ReplyDelete
  8. Of course, Barry the lawyer apparently believes that all biology majors take the LSAT, not understanding that the biology majors with very high GPAs probably don't actually want to be lawyers. That sort of thinking is just too much for the guy that tried to blame Columbine of evolution.

    ReplyDelete
  9. This table shows SAT score and average IQ by college major
    http://www.statisticbrain.com/iq-estimates-by-intended-college-major/

    ReplyDelete
  10. In addition to most of the comments above, a few other things to thing about:

    1. How many colleges/universities have majors in biology?

    2. In classics?

    3. How many students are biology majors?

    4. How many students are classic majors?

    5. What's the average background of a biology major, in terms of family wealth, pre-college education, social class, etc.?

    6. Ditto for classics majors.

    I don't know where you'd get hard data on this, but I bet the answers are:


    1. How many colleges/universities have majors in biology?

    Virtually all of them, from Harvard down to community colleges.

    2. In classics?

    Mostly expensive liberal arts colleges and big-name institutions.

    3. How many students are biology majors?

    Tons.

    4. How many students are classic majors?

    A few.

    5. What's the average background of a biology major, in terms of family wealth, pre-college education, social class, etc.?

    Due to the huge population, biology majors on average will be a pretty average sample from the overall undergraduate population.

    6. Ditto for classics majors.

    I bet classic majors would be above average in having rich parents, privileged upbringing and education, attending expensive cools, and being confident/well-funded enough in their life prospects to get an "esoteric" degree.

    Which leads to question 7:

    7. What is Barry Arrington's ability to think critically?

    Not great.

    ReplyDelete