Can ChatGPT4 Pass Cryptography Tests?

Three experts from the University of Siegen are researching how well ChatGPT4 performs in real testing situations. Their findings may surprise you.

The importance of artificial intelligence in our daily lives continues to grow. American tech giant Apple is planning to integrate ChatGPT, made by Open AI, directly into its iPhones. Yet, is ChatGPT also powerful enough to pass three complete cryptography tests? The experiment was launched at the University of Siegen by three researchers, Professor Bernhard Esslinger, Dr. Nils Kopal, and Dr. Vasily Mikhalev.

Professor Esslinger and Dr. Kopal are researchers in the field of IT security at the University of Siegen and belong to the “DECRYPT” project, which receives funding from the Swedish Research Council. They are also part of an international team that provides support for CrypTool, a free eLearning platform that allow students, schoolchildren, and other interested parties to familiarize themselves hand-on with cryptographic algorithms and encryptions. The team now hopes to determine whether artificial intelligence is capable of solving cryptography tests.

For their experiment, the researchers presented ChatGPT with three problems of varying difficulty: one oral examination at a bachelor's degree level; one written examination from the master's degree level; and a series of complex and complicated homework problems from a master’s degree program. When grading the tests, Professor Esslinger and his team applied the same criteria as they would have used for students. To ensure neutrality, the cryptologists used examination materials from various teaching chairs in Hessen and Baden-Württemberg.

“The new ChatGPT trains itself using the entire internet. We intentionally picked out oral and written examinations that have never been published online before. What we wanted to know: What kind of answers can ChatGPT really achieve using its logical processes?” Professor Esslinger explains.

The results were surprising: During the oral examination, which queried the fundamentals of cryptology, ChatGPT scored 202 out of 208 points, or 97%. During the more difficult master’s test, the AI also achieved a very good result (80.5 out of 90 points, corresponding to 89%). Yet one thing stood out: ChatGPT had a tendency to make calculation errors and false statements. “It's a general problem with artificial intelligence. They are very good at explaining know theory, but when they have to work with concrete numbers, they typically make mistakes. They are especially frequently wrong when it comes to complex calculations involving large numbers. Depending on how difficult a process is, they often answer: That is too complicated, more programming is now needed,” Dr. Kopal says.

This weakness was particularly evident in the final test: “Those are tasks where the people must do something actively, such as programming — and that’s where ChatGPT tends to have a problem,” Dr. Kopal explains. "For one task, AI simply solved the wrong problem. Here's the funny thing: Students do the same thing sometimes,” he says from personal experience. For the more complex tasks, ChatGPT4 still achieved a 75% score — a very good result, if one follows the grading key from the test's creator at the University of Mannheim.

So can artificial intelligence help students with their studies? “No,” the cryptology experts indicate. For good reason: “Even if students use ChatGPT during their bachelor’s degree, use it to pass tests or write their term papers — it will show up when it's time for their bachelor's thesis and I sit across from them for the oral exam. And if and when I discover that that’s what they did, then they won't pass the exam,” Professor Esslinger emphasizes. It’s crucial that students can still think for themselves and question any AI-generated solutions instead of trusting them blindly.

Still: the use of artificial intelligence like ChatGPT presents instructors with new challenges. For example, they make it harder to detect cheating. “With some students, you can tell that they copied off ChatGPT. The bot is extremely loquacious, chatty, and correct to a T,” Professor Esslinger notes. To prevent AI-based cheating, he has a simple recommendation: “We should do more oral examinations and assign less homework, because otherwise we can’t check what the students have done on their own.” Yet that’s not all there is to this story, he feels. “We have to put people in a position to use AI in a good and responsible manner.” Both Professor Esslinger and Dr. Kopel agree that the future of examinations necessitates test that are not harder per se, but which are designed with more originality, featuring practical applications and less questioning of fundamental knowledge.

The group published its findings as an academic paper, available for free, under the title “Evaluating GPT-4’s proficiency in addressing cryptography examinations”, in the industry journal Cryptologia. The research portion was made possible by Sweden’s “DECRYPT” project.

Contact:
Professor Bernhard Esslinger
Professor for Cryptology and IT Security (Fak. 3)
Email: bernhard.esslinger@uni-siegen.de

Professor Bernhard Esslinger and Dr. Niels Kopal have researched whether ChatGPT4 can solve exams in cryptology.