We Put ChatGPT Through the 2022 PSLE English Paper 2 & What We Found Might Shock You

A few months back, the Straits Times con­duct­ed an exper­i­ment to test the prowess of Open AI’s rev­o­lu­tion­ary chat­bot, Chat­G­PT. They fed Chat­G­PT with past year PSLE ques­tions and assessed whether Chat­G­PT was smarter than a PSLE stu­dent.  

The ver­dict? Chat­G­PT was “defeat­ed” by the PSLE. It was found to be “mis­er­able” at han­dling Math­e­mat­ics and Sci­ence and only “scraped though most of the Eng­lish com­pre­hen­sion ques­tions”. When the Straits Times pub­lished their find­ings on the papers, pub­lic respons­es were aplen­ty but mixed. Some expressed utter hor­ror and used these find­ings as fod­der for the overused argu­ment that the PSLE is sim­ply too dif­fi­cult for a 12-year-old. Oth­ers were smug and proud­ly pro­claimed how high the stan­dard of edu­ca­tion is in Sin­ga­pore: Chat­G­PT was able to pass the tough­est exams in the Unit­ed States includ­ing Wharton’s MBA exam, the New York Bar exam, and the US med­ical licens­ing exam, but it failed to sur­mount the hur­dle of Singapore’s PSLE.

Know­ing how pow­er­ful and smart Chat­G­PT is and can be, we were, to be very hon­est, shocked by the Straits Times’ report. To sat­is­fy our curios­i­ty, we decid­ed to test Chat­G­PT on our own. We did not test it with Math­e­mat­ics or Sci­ence because we want­ed a fair test. Open AI, the cre­ator of Chat­G­PT, has long acknowl­edged that the cur­rent ver­sion of Chat­G­PT is still not adept at inter­pret­ing images, dia­grams or graphs. Since ques­tions with images, dia­grams and graphs form a sig­nif­i­cant bulk of Math­e­mat­ics and Sci­ence ques­tions, Chat­G­PT would not do well for these two sub­jects. Hence, for our lit­tle exper­i­ment, we focussed on Eng­lish. We used the entire 2022 PSLE Eng­lish Paper 2, except for Visu­al Text Com­pre­hen­sion and Com­pre­hen­sion Open-End­ed.  

For the most part, our find­ings were to be expect­ed but there were some sur­pris­es! Here’s what we found:

PSLE 2022 Gram­mar MCQ: 6/10

We typed in all 10 Gram­mar MCQ ques­tions and asked Chat­G­PT for the answers as well as expla­na­tions. Chat­G­PT dis­ap­point­ing­ly gave us 4 wrong answers as well as 4 shock­ing­ly mis­di­rect­ed expla­na­tions. The worst part is that we are still befud­dled as to why and how Chat­G­PT could have made such glar­ing errors in mul­ti­ple-choice-ques­tions.   

Note: Ques­tions 5, 6, 9 and 10 were answered wrong­ly.

PSLE 2022 Vocab­u­lary MCQ: 5/5

We typed in all 5 vocab­u­lary MCQ ques­tions and asked GPT for the answers. Chat­G­PT was expect­ed­ly flaw­less.

PSLE 2022 Vocab­u­lary Cloze: 4/5

We typed the entire pas­sage onto Chat­G­PT and asked it what word the phras­es or words in the ques­tions have the clos­est mean­ing to. Chat­G­PT got the first ques­tion wrong. This ques­tion requires appre­ci­at­ing the con­text of the pas­sage and infer­ring what the fig­u­ra­tive phrase “feast­ing my eyes on” means. The fact that Chat­G­PT chose “observ­ing” as the answer shows that it only under­stood the lit­er­al mean­ing of “feast­ing my eyes on” as opposed to the inferred mean­ing of it in the pas­sage. This is evi­dence of a clear lim­i­ta­tion of Chat­G­PT.   

PSLE 2022 Gram­mar Cloze: 10/10

We typed the entire pas­sage onto Chat­G­PT togeth­er with the 10 ques­tions and 15 options. We told Chat­G­PT that it could only use one option once for each ques­tion. It mas­ter­ful­ly answered every ques­tion cor­rect­ly. 

PSLE 2022 Edit­ing: 12/12

We typed the entire pas­sage onto Chat­G­PT togeth­er with the 12 gram­mar and spelling mis­takes. We asked Chat­G­PT to cor­rect the spelling and/or gram­mat­i­cal mis­takes. Sur­pris­ing­ly, Chat­G­PT under­stood our instruc­tions and cor­rect­ed all 12 mis­takes.

PSLE 2022 Com­pre­hen­sion Cloze: 15/15

We typed the entire pas­sage onto Chat­G­PT togeth­er with the 15 blanks. We told Chat­G­PT to fill in every blank with an appro­pri­ate word. Chat­G­PT did not dis­ap­point.

PSLE 2022 Syn­the­sis & Trans­for­ma­tion: 10/10

We typed all 5 ques­tions onto Chat­G­PT and asked it to trans­form the orig­i­nal sen­tence to a new sen­tence based on the help­ing words pro­vid­ed in each ques­tion. Chat­G­PT under­stood the brief and gave us 5 cor­rect answers. Do note that ChatGPT’s answers to ques­tions 2 and 4 are dif­fer­ent from the answer sheet but we believe they are accept­able.

Con­clud­ing Remarks

Depend­ing on how high a pedestal you put Chat­G­PT on, 5 mis­takes out of a total of 67 ques­tions is not too bad. This remark­able per­for­mance of Chat­G­PT con­firms one thing: if used in the right way, it can be a very use­ful tool for self-direct­ed learn­ing. How­ev­er, as the peren­ni­al ques­tion goes, would Chat­G­PT be able to replace the role of a teacher? The short answer is no, at least not right now.

As demon­strat­ed, Chat­G­PT will, for the most part, be able to give you the right answers and expla­na­tions to ques­tions. But it is far from infal­li­ble. The rev­o­lu­tion­ary AI chat­bot is prone to mak­ing very fun­da­men­tal mis­takes. The oth­er major issue with Chat­G­PT is that it is not a flex­i­ble teach­ing tool because you must always ask it the right ques­tions. Chat­G­PT does not know you and can­not read your mind. It is thus only as effec­tive as the ques­tions you ask it. And there­in lies the prob­lem. Ask­ing the right ques­tions is the most dif­fi­cult part. From our expe­ri­ence teach­ing so many dif­fer­ent stu­dents from dif­fer­ent schools, we under­stand stu­dents by and large do not know what they do not know. When stu­dents do not know what they do not know, they would nat­u­ral­ly not know what to ask. As Chat­G­PT is unable to assess or diag­nose stu­dents, it is pow­er­less to pro­vide any form of mean­ing­ful assis­tance on this front.

Let’s com­pare this for a sec­ond to what we do at Think Teach Acad­e­my. Our stu­dents go through block assess­ments every 4 lessons under our tute­lage. Such assess­ments allow our teach­ers to track the pro­fi­cien­cy and progress of every stu­dent. Know­ing the strengths and weak­ness­es of our stu­dents means that we can bet­ter guide and coach them. Know­ing the gaps in their under­stand­ing means that we do not have to wait for our stu­dents to ask us ques­tions. We can pre-emp­tive­ly tell and teach them what they need to know. Such is the expe­ri­ence and exper­tise of a good teacher that arti­fi­cial intel­li­gence has still not yet fig­ured out how to repli­cate.

So, for now, we can safe­ly say that the job of the teacher is still well and secure. Things might of course change and be dif­fer­ent in the future, but until then, if you are going to use Chat­G­PT, pay heed to a wise latin say­ing, caveat util­i­tor (or user beware!). Be mind­ful that Chat­G­PT does not spout gospel truth so use it care­ful­ly and at your own risk!

This arti­cle was proud­ly writ­ten for you by TTA’s Eng­lish Team. Every year, we impact over a thou­sand stu­dents and empow­er them with our smart tech­niques to achieve exam excel­lence in their school exams as well as their PSLE. We want your child to be our next suc­cess sto­ry! Togeth­er with us, your child can strength­en his/her mas­tery of paper 2 con­tent by join­ing our Full Eng­lish Pro­gramme (FEP). Addi­tion­al­ly, your child can hone his/her com­po­si­tion writ­ing skills by par­tak­ing in our Junior Writ­ing Class­es (JWC) and Mas­ter Writ­ing Class­es (MWC).