by, for, and about women in science, technology, engineering, & math

Clara wrote about the barriers to entering Computer Science at the college level, especially the CS101 “hidden prerequisite” of previous programing experience.

So you can imagine my reaction when I read Jeff Atwood’s Coding Horror blog post, Separating Programming Sheep from Non-Programming Goats, about creating additional – artificial – barriers. Not that it’s Atwood’s fault: he’s writing about a draft paper[pdf] by Professors Saeed Dehnadi and Richard Bornat where they propose a screening test for new programing students.

Clearly, Dehnahdi’s test is not a perfect divider of programming sheep from non-programming goats. Nevertheless, if it were used as an admissions barrier, and only those who scored consistently were admitted, the pass/fail statistics would be transformed. In the total population 32 out of 61 (52%) failed; in the first-test consistent group only 6 out of 27 (22%).

Well, that doesn’t sound so bad, right? In the context of a university course with such a high failure rate, something needs to be done. Let’s take a look at this test, administered – in this study – to students before their first day of instruction.

Read the following statements and tick the box next to the correct answer.

int a = 10;

int b = 20;

a = b;

The new values of a and b are:

[ ] a = 20 b = 0

[ ] a = 20 b = 20

[ ] a = 0 b = 10

[ ] a = 10 b = 10

[ ] a = 30 b = 20

[ ] a = 30 b = 0

[ ] a = 10 b = 30

[ ] a = 0 b = 30

[ ] a = 10 b = 20

[ ] a = 20 b = 10

Oh dear.

In fairness, from reading the paper, it does seem like the intent is not to test for the “right answer” but to look for adherence to a mental model between questions in the face of an ambiguous task. They’re trying to test for rule-consistent thinking, and intend to award points for “wrong but good” answers. So it’s not as terrible as it initially looks.

But aside from some hand-waving at the beginning of their methods section, it doesn’t seem like any attempt has been made to exclude students with previous programing experience, which would clearly be of enormous benefit in taking the test!

[T]he test was first administered to about 30 students on a further-education programming course at Barnet College before they had received any programming teaching whatsoever – that is, in week 0 of the course. Those students had no particular pattern of age, sex and educational background. Dehnahdi interviewed half of them before admission, and taught them all. We believe that none had any previous contact with programming, and that all had enough school mathematics to make the equality sign familiar. The test was anonymised, in that the students invented nicknames for themselves.

The same test was then administered to about 30 students in the first-year programming course at Middlesex University, once again before they had received any programming teaching. They were mostly male, aged about 18-20, from the middle range of educational attainment and from families towards the lower end of income distribution. Again the answers were anonymised. This time Dehnahdi tutored them but did not teach the course. We believe (without any interview evidence at all) that they had had no contact with programming but had received basic school mathematics teaching.

Um, really? The single most obvious way for any student to have a consistent mental model for this test is for that student to know Java or some other C-ish language. Much more explanation needs to be done about the exclusion of this confounding variable, beyond “we believe”.

But really, methodological discussion is beside my point: the whole enterprise is a perfect example of the prevalent confusion of prior experience for ability. Where Clara was writing about structural and institutional barriers to joining Computer Science in college, here we have a deliberate academic barrier in the form of a multiple choice test. If this thing were to become popular, it would institutionalize the requirements of pre-learning CS. What a disaster that would be: turning a sexist cultural norm into a deliberate collegiate impediment.