User:WeatherWriter/LLM Experiment 3
 | This page in a nutshell: Large language models (i.e. Chat-GPT) can be used to help identify sourced and unsourced information in a Wikipedia article, including from non-English sources. The experiment has 6/7 successful tests. |
This experiments purpose is to identify if large language models (i.e. Chat-GPT4) can be used to locate sourced and unsourced information within the following Wikipedia article: 22 March 2024 Russian strikes on Ukraine.
Previous experiments
[edit]Brief summary
[edit]The third Large Language Model experiment conducted by WeatherWriter was on the 22 March 2024 Russian strikes on Ukraine, an article which has since been merged and redirected. The experiment and associated discussion regarding the experiment occurred on that article talk page. Given the associated article no longer exists, the talk page discussion for the experiment was archived and has been copied and pasted below.
Experiment Discussion
[edit]An LLM verification check was conducted on this article. As of this message, the following sections were confirmed to be verifiable: Kharkiv, Zaporizhzhia, Kryvyi Rih, Strike on the Dnieper Hydroelectric Station, Sumy Oblast, and Poltava Oblast. The section Khmelnytskyi, was identified to have OR regarding casualties. No other sections were checked thus far by LLM for verification. The Weather Event Writer (Talk Page) 16:17, 22 March 2024 (UTC)
- @Oleg Yunakov: — I’m just doing a curiosity ping since you removed the OR tag and told me to go to the talk page. I don’t think you saw this message. LLM identified that the casualties were not directly stated by Meduza (the source), so it is original research. By chance, could you quote where it says those casualties so I can try to figure out why the LLM identified that as OR since you say the source states them? Thank you! The Weather Event Writer (Talk Page) 16:33, 22 March 2024 (UTC)
- Sorry, yes, I din't see this message when I undid the OR change. No, there is no original research as here is what is written by Meduza: "The Ministry of Internal Affairs of Ukraine also reports two deaths in Khmelnitsky". Thanks. With regards, Oleg Y. (talk) 16:37, 22 March 2024 (UTC)
- Are the injuries also directly stated? The A.I stated:
Regarding Khmelnytskyi, Meduza [1] notes damage to infrastructure and residential buildings and reports casualties among civilians following the Russian strike. However, Meduza does not provide specific numbers of dead and injured as mentioned in your text (two people killed and eight people injured).
The specific detail about the number of dead and injured in Khmelnytskyi (two killed and eight injured) is not explicitly stated by Meduza [1]. All other information regarding Sumy Oblast and Poltava Oblast is consistent with Meduza's reporting.
- Obviously it states two were killed so that was a true A.I. error. I’m wondering if the injuries is what tripped it. The Weather Event Writer (Talk Page) 16:43, 22 March 2024 (UTC)
- I see what you mean. The 8 was there from the beginning as can be seen from the addition of a colleague to the overall March timeline with the same source. The number 8 was taken by Meduza from this source (it's currently referenced as a link in that Meduza article). This source is an official channel of Ministry of Internal Affairs (Ukraine) who reposted message of a minister of internal affairs Ihor Klymenko and it states 8, so it's not an original research... With regards, Oleg Y. (talk) 16:57, 22 March 2024 (UTC)
- Thank you! Yep, not original research. The A.I. messed up. Cheers! The Weather Event Writer (Talk Page) 16:59, 22 March 2024 (UTC)
- I see what you mean. The 8 was there from the beginning as can be seen from the addition of a colleague to the overall March timeline with the same source. The number 8 was taken by Meduza from this source (it's currently referenced as a link in that Meduza article). This source is an official channel of Ministry of Internal Affairs (Ukraine) who reposted message of a minister of internal affairs Ihor Klymenko and it states 8, so it's not an original research... With regards, Oleg Y. (talk) 16:57, 22 March 2024 (UTC)
- Are the injuries also directly stated? The A.I stated:
- Sorry, yes, I din't see this message when I undid the OR change. No, there is no original research as here is what is written by Meduza: "The Ministry of Internal Affairs of Ukraine also reports two deaths in Khmelnitsky". Thanks. With regards, Oleg Y. (talk) 16:37, 22 March 2024 (UTC)
Conclusion
[edit]To conclude this experiment, it appears six of the seven individual tests were successful. However, as discussed with Oleg Yunakov above, one of the tests returned a completely false answer. Wikipedia editors should always be mindful of using Artificial Intelligence. While A.I. can be helpful and save time, all work done through or checked with LLMs should be rechecked individually by the user.