ChatGPT and PSLE Science: A Worrisome Case of Wrong Answers & Faulty Explanations  

We all know by now that the Straits Times attempt­ed to put Chat­G­PT to the sword. It recent­ly chal­lenged Chat­G­PT to the PSLE Sci­ence Exam, and found that the chat­bot was “mis­er­able” at it, scor­ing an aver­age of 21/100 for the PSLE 2020 to PSLE 2022 Sci­ence exam. 

The rea­son for its fail­ure is sim­ple: the cur­rent pub­licly avail­able ver­sion of Chat­G­PT is not adept at inter­pret­ing images, dia­grams and graphs. When it comes to the PSLE Sci­ence, most mul­ti­ple-choice and the open-end­ed ques­tions are based on exper­i­ments and their results are record­ed in image, graph and table forms. Chat­G­PT would not be able to under­stand and han­dle such ques­tions. 

In fact, text-based ques­tions tra­di­tion­al­ly form around 20 to 25 out of 100 marks in the PSLE sci­ence exam, which explains why Chat­G­PT was only able to attain an aver­age over­all score of 21/100 for PSLE 2020 to PSLE 2022. 

With Open AI already dis­claim­ing that the cur­rent ver­sion of Chat­G­PT is unable to han­dle images, we did not find the Straits Times test to be a fair exper­i­ment (no pun intend­ed). So we decid­ed to give Chat­G­PT a sec­ond crack at PSLE Sci­ence. This time, we played to the chatbot’s strengths and capa­bil­i­ties by mak­ing sure to feed Chat­G­PT with only text-based mul­ti­ple choice ques­tions. 

By and large, with suf­fi­cient prompt­ing, the chat­bot was able to arrive at the right answers. How­ev­er, we found its expla­na­tions want­i­ng in many respects. In oth­er words, Chat­G­PT is a good stu­dent but not quite a good teacher. 

We sub­mit­ted 11 text-based and mul­ti­ple-choice ques­tions to Chat­G­PT, expect­ing accu­rate answers and expla­na­tions. Unfor­tu­nate­ly, we found that it pro­vid­ed incor­rect respons­es to 3 of the ques­tions. What was even more sur­pris­ing and wor­ry­ing was that the expla­na­tions giv­en were mis­lead­ing.

We will explain with our exper­i­men­tal find­ings.

PSLE 2022 MCQ: 7/11

In 2022, only 11 MCQs were text-based ques­tions or could be eas­i­ly con­vert­ed to text-based ques­tions. 

3 ques­tions were answered wrong­ly and 1 ques­tion, though answered cor­rect­ly, was explained inac­cu­rate­ly. They were ques­tions 7,8, 9 and 16. We repro­duce these ques­tions in the fol­low­ing screen­shots which show our con­ver­sa­tions with Chat­G­PT:

Ques­tion 7 

ChatGPT’s Answer: 3

To our amaze­ment, Chat­G­PT fell short in a rather straight­for­ward ques­tion. It answered wrong­ly in its first attempt but was able to cor­rect itself after we point­ed out its mis­take.

Ques­tion 8

Cor­rect Answer: C and D

ChatGPT’s Answer: C only 

Yet again, Chat­G­PT dis­ap­point­ed us with a wrong answer. It admit­ted an “over­sight” on its part and cor­rect­ed itself after we point­ed out what the right answer is. Its sav­ing grace, how­ev­er, is that it was able to pro­vide a cor­rect expla­na­tion for the answer giv­en. 

Ques­tion 9

Cor­rect Answer: 4

ChatGPT’s Answer: 4 but Wrong Expla­na­tion 

Chat­G­PT was able to iden­ti­fy that option (4) is the cor­rect answer.

How­ev­er, it fell short when asked to explain the answer to a pri­ma­ry school stu­dent. In Point 4, Chat­G­PT explained that the mass of seed leaves remains the same dur­ing ger­mi­na­tion. How­ev­er, a PSLE stu­dent would have learnt that the mass of seed leaves decreas­es since the seedling would use up the food stored in the seed leaf. When this mis­take was point­ed out to it, Chat­G­PT was able to apol­o­gise for its error and cor­rect itself.  

Ques­tion 16

Cor­rect Answer: 4

ChatGPT’s Answer: 3  

Final­ly, for ques­tion 16, Chat­G­PT gave the wrong answer once again.

How­ev­er, when its mis­take was point­ed out, Chat­G­PT was able to backpedal and answer the ques­tion cor­rect­ly. But the dam­age has already been done. If a stu­dent was not wise to the mis­take, he or she would have learnt some­thing wrong from Chat­G­PT. 

What we were able to con­clude from our lit­tle fun exper­i­ment was that it is essen­tial to exer­cise cau­tion when using AI-pow­ered chat­bots such as Chat­G­PT. Not only is it advanced enough to han­dle com­plex dia­grams or graphs, but it is also not 100% accu­rate all the time. The sim­ple rea­son for this is that smart chat­bots like Chat­G­PT learn from avail­able infor­ma­tion and data from the Inter­net. As we have all come to realise, not every­thing we see or hear is reli­able on the Inter­net.

Concluding Remarks

So is Chat­G­PT even use­ful to stu­dents when it comes to the sub­ject of Sci­ence? We will say yes, for old­er stu­dents, but will hes­i­tate using it for pri­ma­ry school stu­dents. ChatGPT’s poten­tial use­ful­ness is more evi­dent among ter­tiary-lev­el stu­dents who already pos­sess a sol­id foun­da­tion of Sci­ence knowl­edge and require the abil­i­ty to extract infor­ma­tion from a large vol­ume of data. We have to be cog­nizant that the infor­ma­tion pro­vid­ed by Chat­G­PT is not always aligned with the spe­cif­ic require­ments of the PSLE Sci­ence syl­labus in Sin­ga­pore. 

Chat­G­PT also has much room for improve­ment and occa­sion­al­ly makes con­cep­tu­al mis­takes. This pos­es a chal­lenge for pri­ma­ry stu­dents who seek accu­rate learn­ing resources tai­lored to the PSLE Sci­ence cur­ricu­lum. In addi­tion, Chat­G­PT does not equip pri­ma­ry stu­dents with the nec­es­sary writ­ing tech­niques or empha­sise the key­words and key phras­es that are cru­cial for answer­ing the Sci­ence exam­i­na­tion open end­ed ques­tions. 

The oth­er major issue with Chat­G­PT is that it is not a flex­i­ble teach­ing tool because you must always ask it the right ques­tions. Chat­G­PT does not know you and can­not read your mind. It is thus only as effec­tive as the ques­tions you ask it. And there­in lies the prob­lem. Ask­ing the right ques­tions is the most dif­fi­cult part. From our expe­ri­ence teach­ing so many dif­fer­ent stu­dents from dif­fer­ent schools, we under­stand stu­dents by and large do not know what they do not know. When stu­dents do not know what they do not know, they would nat­u­ral­ly not know what to ask. As Chat­G­PT is unable to assess or diag­nose stu­dents, it is pow­er­less to pro­vide any form of mean­ing­ful assis­tance on this front.

Con­sid­er­ing some of these lim­i­ta­tions, it becomes clear that an over­re­liance on Chat­G­PT is not advis­able. Instead, a more com­pre­hen­sive approach that incor­po­rates rel­e­vant and tar­get­ed edu­ca­tion­al resources should be adopt­ed to ensure pri­ma­ry stu­dents receive reli­able and tai­lored sup­port for their PSLE Sci­ence exam­i­na­tion prepa­ra­tion. 

This arti­cle was proud­ly writ­ten by TTA’s Sci­ence Team. We aim to help stu­dents achieve exam excel­lence in Sci­ence with our sig­na­ture tem­plat­ed answer­ing struc­tures which will help your child tack­le with ease and con­fi­dence for even the hard­est of sci­ence ques­tions.