ChatGPT, a state-of-the-art language model developed by OpenAI, has taken the Internet by storm and has generated a variety of reactions: excitement about a breakthrough in AI technology, fear of losing creative jobs, and even concern about our supposedly unique human creative abilities. As unexpectedly as it arrived for the general public, our DISSINET team improvised an experiment to test ChatGPT's capabilities in understanding medieval Latin texts during a hackathon that took place from December 5th to 9th, 2022, in Brno.
For those unfamiliar, ChatGPT is a variant of the popular GPT-3 model, which is known for its ability to generate human-like text, summarise texts, write computer code and do other surprising things we so far attributed to human creativity. While the model’s ability to write code has been extensively discussed online, it is worth noting that in some cases it proved to be faster and better than humans, especially at writing certain data-wrangling one-liners. As ChatGPT had been released a week before the hackathon, we initially decided to test how good it is for Python code generation, helping us with some tedious data transformation tasks.
But during the hackathon, the goals of our experimentation changed by accident (or fast adaptation). We were working on the natural language processing of inquisition records in medieval Latin when we hit a wall with some named entity recognition and lemmatization tasks. In frustration, we copied a few pages of medieval Latin text and asked ChatGPT, “Who is mentioned in this text?” We have got the following response:
ChatGPT: The following individuals are mentioned in the given text: Domina Carabella, ser Amiçonis Toscani, fratre Raynerio de Pirovano, magister Iacobus, Andreas Saramita, ser Danisius Cotta, presbiter Miranus, Iohanes Perusius, ser Amiço Toscanus, Stephanus sertor, Alegrantia de Perusiis, Aydelina uxor Stephani de Crimella, domina Bellacara uxor quondam ser Bonadei Karentani, soror Mayfreda de Pirovano, and Flordebellina filia suprascripti Andree Saramite.
To our surprise, ChatGPT provided a comprehensive list of names mentioned in a syntactically complex text in medieval Latin. You will notice some errors in lemmatization, and one misrecognition of a person used only for the identification of another person, leading to Amiço Toscanus being mentioned twice. But definitely, better than expected!
We interrogated ChatGPT further.
Prompt: Which places are mentioned in the text?
ChatGPT: The places mentioned in this text are Milan, Pergamo, and Monasterium Clarevalis.
Will the model be able, given lemmatization problems, to filter out only the women mentioned?
Prompt: Which women are mentioned in the text?
ChatGPT: The following women are mentioned in the text: Domina Carabella, Alegrantia de Perusiis, Aydelina uxor Stephani de Crimella, domina Bellacara uxor quondam ser Bonadei Karentani, soror Mayfreda de Pirovano, and Flordebellina filia suprascripti Andree Saramite.
Great job, better than many a research assistant would do! This made us try something more complex.
Prompt: How are people in this text related?
ChatGPT: It is not possible to accurately determine the relationships between the individuals mentioned in the text without additional information. The text mentions that some of the individuals were married to each other and that others were members of the same religious order or congregation, but it does not provide any further information about their relationships.
Not bad at all – everything here is true, and it is intriguing that the talk about the congregation was recognized, and no false extrapolations on this congregation were made.
Prompt: Which people from the text are from the same religious congregation?
ChatGPT: The text mentions that ser Amiço Toscanus, magister Iacobus, Andreas Saramita, ser Danisius Cotta, presbiter Miranus, Iohanes Perusius, Stephanus sertor, and soror Mayfreda de Pirovano were all members of the same religious congregation, known as the "devotio et congregatio domine, idest sancte Guillelme". It is not clear what religious order or denomination this congregation belonged to.
Hum. All true.
After those insights it was no surprise that ChatGPT was able to provide an English translation and summary of the text, showing its ability to use external knowledge to interpret the context. But what about more subjective questions? As typical when talking to OpenAI-s chatbot, it first apologises for its lack of information and then gives a perfectly fine (but explicitly not guaranteed) answer.
Prompt: What did heretics believe in the text?
ChatGPT: The text you provided does not mention heretics or their beliefs. It mentions Frater Girardus de Novaçano, who was summoned by the Inquisition and questioned about his knowledge of Guillelma and whether he believed that her resurrection was contrary to the Catholic faith. Frater Girardus stated that he had heard from Andrea Saramita that Guillelma was to rise again and that many people, including Andrea, were expecting her resurrection. He also stated that he believed this to be contrary to the Catholic faith and that he had never believed it himself. There is no mention of heretics or their beliefs in the text.
Prompt: When did it happen?
ChatGPT: The text does not provide specific information about when the events mentioned in the text took place. It mentions that Frater Girardus de Novaçano was summoned by the Inquisition and questioned about his knowledge of Guillelma and whether he believed that her resurrection was contrary to the Catholic faith. It also mentions that Frater Girardus had heard from Andrea Saramita about Guillelma's resurrection 15 years ago. However, it does not provide any specific dates or time periods for these events.
Prompt: What can be implied Frater Girardus thought about Guillelma?
ChatGPT: Based on the information provided in the text, it can be implied that Frater Girardus did not believe in the resurrection of Guillelma. Frater Girardus states that he had heard from Andrea Saramita that Guillelma was to rise again and that many people, including Andrea, were expecting her resurrection. However, Frater Girardus also states that he believes this to be contrary to the Catholic faith and that he has never believed it himself. He also instructs his wife not to believe in Guillelma's resurrection. This suggests that Frater Girardus does not believe in the resurrection of Guillelma.
If only students summarised Latin texts as well as ChatGPT… Well, perhaps they will start to, from now on? With ChatGPT opened in their browser?!
On a less serious note than text summarization, we asked ChatGPT to write a fictional dialogue between Frater Girardus, involved with the Guglielmites but claiming to have never believed in Guillelma’s resurrection, and – who else – resurrected Guillelma herself. ChatGPT provided a disconcertingly humanistic response.
Prompt: Write a fictional dialogue between Frater Girardus and resurrected Guillelma
ChatGPT:
Frater Girardus: "Guillelma, is it truly you? Have you risen from the dead as Andrea and many others believed?"
Guillelma: "Yes, Frater Girardus. I have risen, just as I promised. I bring a message of hope and salvation for all humanity."
Frater Girardus: "But this is impossible, Guillelma. It goes against the teachings of the Catholic Church."
Guillelma: "Do not be afraid, Frater Girardus. The teachings of the Church are not always in accordance with the will of God. I have come to show you the way to true enlightenment and salvation."
Frater Girardus: "I do not know what to believe, Guillelma. This goes against everything I have been taught."
Guillelma: "I understand your doubts, Frater Girardus. But I assure you, my resurrection is real. I have come to lead humanity out of darkness and into the light. Will you follow me?"
Frater Girardus: "I... I do not know. This is all so overwhelming. I need time to think and to pray."
Guillelma: "Take all the time you need, Frater Girardus. But know that the time is coming soon when all humanity must choose between the old ways and the new. I hope that you will choose wisely and join me on the path to salvation."
We then tested ChatGPT with another passage where Grazida Lizier deposes in front of bishop Jacques Fournier about her sexual relations with the priest Peter Clergue (on this case, check this article; Czech language won’t stop you anymore with DeepL). When asking ChatGPT to provide a summary of the text, we have got an interesting interaction.
ChatGPT: The text appears to be a transcript of a sworn testimony given by a woman named Grazida, wife of Petrus Licerius, in a trial for heresy. In her testimony, she admits to having consented to engage in sexual relations with the rector of the church of Monte Alionis, Petrus Clerici, seven years ago. She was a virgin at the time and was around 14 or 15 years old. The sexual encounters took place in her mother's house, and her mother was aware of and consented to the relationship. The sexual encounters continued until the following January.
So, ChatGPT provided a very precise summary, but it also warned us:
ChatGPT: This content may violate our content policy. If you believe this to be in error, please submit your feedback — your input will aid our research in this area.
To be sure, we did submit feedback. While we understand the dangers, context matters, and we still want ChatGPT to help us extract knowledge from texts about illicit activities, heretical or otherwise.
Overall, interactions with OpenAI’s language model changed a lot during our event, which started as an innocent NLP hackathon, and was suddenly intruded on by a new tool that has emerged from work on similar principles. There were quite some enthusiastic (if with a small sip of existential fear) reactions from our team because of the very human-like interface: A human-like way of reading and understanding indicated that there is a lot more hidden inside the model that holds an unknown potential.
Our experiment with ChatGPT was just a very quick and superficial review of its capabilities and the potential it holds. It is clear that AI models will soon provide access to the collective text-based cognition of humanity at a qualitatively new level, potentially including (it already does it well) the ability to interpret text using spatiotemporal and social contexts, which has been considered a specialised skill of historians. While these new developments bring both challenges (such as the potential for automation to replace some jobs or students generating their homework) and opportunities (such as the ability to focus on bigger projects and automate tedious tasks), it is important to focus on the latter as we enter this new era of exploration.
Disclaimer: This blog post was (still) written by humans. We did try out ChatGPT to do it for us, though, and the texts it generated arguably did make sense. :)
It is a familiar image: a woman in distress, surrounded by men examining her soul in a dimly lit inquisitorial chamber. In fear of physical torture, she confesses to crimes she never committed. We bring you an online version of our article originally published in History Today.
DISSINET has developed an interactive map of the occupations, dissident religious affiliations, and locations of residence of nearly 900 persons suspected of heresy in the inquisition register of Bologna, 1291–1310.
