Jacob J. Walker's Blog

Scholarly Thoughts, Research, and Journalism for Informal Peer Review

Are international educational tests, like the PISA, asking the right questions (with the right answers)?

with one comment

You are viewing an old revision of this post, from December 27, 2013 @ 16:26:54. See below for differences between this version and the current revision.

My friend Ruth Ackerman cross-posted a video from the AFT on her Facebook page today, which questions some of the conclusions that some groups make from the mediocre U.S. performance on the PISA. (And I’m not going to dive into that politically charged question right now.)

I think we need to step back further, and ask if the PISA is measuring the “right” knowledge.  To try and answer this for myself, I found a set of sample questions for the PISA.  And I started to go through and answer the questions.   And I found one question where the PISA is wrong.

The question basically is about a science article where it talks about scientists that cloned a cow into five calves.  And then asks if the calves have the same genes, the same gender, and the same color.  The question appeared to want to see if students could infer the answer, as it didn’t explicitly give this information in the article that it had the student read.  And the question wanted to have the answer being “yes” for each one.

BUT, while it may seem to be a reasonable assumption to make that the answer would be “yes” for each, it isn’t necessarily the case for color of the cows.  I say this, because people who have cloned their cats have found that they often turn out to be a different color than their original cat, so I suspect this same thing could happen with cows.  So what if a student knew had this advanced knowledge and marked “no”…  Well, they would have been said to be wrong, when in fact the PISA is likely wrong.  (I say likely, as I am not sure that cat and cow clones would behave similarly when it comes to color, but it is no more or less a reasonable or unreasonable assumption as the one the PISA wants the student to make.)

Update: I emailed PISA this blog entry, and hopefully they will respond back, and I’ll share that on the blog also.

Post Revisions:

Changes:

December 27, 2013 @ 16:26:54Current Revision
Content
Unchanged: My friend Ruth Ackerman cross-posted <a href="http:// www.upworthy.com/5-myths- about-our-schools-that-fall- apart-when-you- look-closer- 6?g=2&amp;c=upw1">a video from the AFT</a> on her <a href="https:/ /www.facebook.com/ ruth.ackerman.5">Facebook page</a> today, which questions some of the conclusions that some groups make from the mediocre U.S. performance on the PISA. (And I'm not going to dive into that politically charged question right now.)Unchanged: My friend Ruth Ackerman cross-posted <a href="http:// www.upworthy.com/5-myths- about-our-schools-that-fall- apart-when-you- look-closer- 6?g=2&amp;c=upw1">a video from the AFT</a> on her <a href="https:/ /www.facebook.com/ ruth.ackerman.5">Facebook page</a> today, which questions some of the conclusions that some groups make from the mediocre U.S. performance on the PISA. (And I'm not going to dive into that politically charged question right now.)
Unchanged: I think we need to step back further, and ask if the PISA is measuring the "right" knowledge.  To try and answer this for myself, I found a <a href="http:// pisa-sq.acer.edu.au/">set of sample questions for the PISA</a>.  And I started to go through and answer the questions.   And I found <a href="http:// pisa-sq.acer.edu.au/ showQuestion.php?testId= 2298&amp;questionId=9">one question where the PISA is wrong</a>.Unchanged: I think we need to step back further, and ask if the PISA is measuring the "right" knowledge.  To try and answer this for myself, I found a <a href="http:// pisa-sq.acer.edu.au/">set of sample questions for the PISA</a>.  And I started to go through and answer the questions.   And I found <a href="http:// pisa-sq.acer.edu.au/ showQuestion.php?testId= 2298&amp;questionId=9">one question where the PISA is wrong</a>.
Unchanged: The question basically is about a science article where it talks about scientists that cloned a cow into five calves.  And then asks if the calves have the same genes, the same gender, and the same color.  The question appeared to want to see if students could infer the answer, as it didn't explicitly give this information in the article that it had the student read.  And the question wanted to have the answer being "yes" for each one.Unchanged: The question basically is about a science article where it talks about scientists that cloned a cow into five calves.  And then asks if the calves have the same genes, the same gender, and the same color.  The question appeared to want to see if students could infer the answer, as it didn't explicitly give this information in the article that it had the student read.  And the question wanted to have the answer being "yes" for each one.
Unchanged: BUT, while it may seem to be a reasonable assumption to make that the answer would be "yes" for each, it isn't necessarily the case for color of the cows.  I say this, because <a href="http:// drsophiayin.com/blog/entry/ cloning-cats- rainbow-and- cc-prove-that- cloning-wont- resurrect-your-pet">people who have cloned their cats have found that they often turn out to be a different color than their original cat</a>, so I suspect this same thing could happen with cows.  So what if a student knew had this advanced knowledge and marked "no"...  Well, they would have been said to be wrong, when in fact the PISA is likely wrong.  (I say <em>likely</em>, as I am not sure that cat and cow clones would behave similarly when it comes to color, but it is no more or less a reasonable or unreasonable assumption as the one the PISA wants the student to make.)Unchanged: BUT, while it may seem to be a reasonable assumption to make that the answer would be "yes" for each, it isn't necessarily the case for color of the cows.  I say this, because <a href="http:// drsophiayin.com/blog/entry/ cloning-cats- rainbow-and- cc-prove-that- cloning-wont- resurrect-your-pet">people who have cloned their cats have found that they often turn out to be a different color than their original cat</a>, so I suspect this same thing could happen with cows.  So what if a student knew had this advanced knowledge and marked "no"...  Well, they would have been said to be wrong, when in fact the PISA is likely wrong.  (I say <em>likely</em>, as I am not sure that cat and cow clones would behave similarly when it comes to color, but it is no more or less a reasonable or unreasonable assumption as the one the PISA wants the student to make.)
Deleted: <strong>Update</strong>: I emailed PISA this blog entry, and hopefully they will respond back, and I'll share that on the blog also. Added: <strong>Update</strong>: I emailed PISA (edu.pisa@oecd.org) this blog entry, but never heard back from them.

Note: Spaces may be added to comparison text to allow better line wrapping.

Written by Jacob Walker

December 27th, 2013 at 3:51 pm

One Response to 'Are international educational tests, like the PISA, asking the right questions (with the right answers)?'

Subscribe to comments with RSS or TrackBack to 'Are international educational tests, like the PISA, asking the right questions (with the right answers)?'.

  1. It seems to me we are talking about LOGIC VS FACTS…
    Apparently the answer to the question was false..but the PISA wanted a Logical Answer from the students?
    Looking for LOGIC?

    Carolyn Jette

    29 Dec 13 at 2:49 am

Leave a Reply

%d bloggers like this: