User:WeatherWriter/LLM Experiment 1

This experiments purpose is to identify if large language models (i.e. ChatGPT-4) can be used to locate sourced and unsourced information within a Wikipedia article.

Hypothesis

If Chat-GPT4 is provided with a copy/paste of accurate and reliable sources about a topic, followed by a copy/paste of the Wikipedia article, then it will be able to identify what information came from what source as well as what information is unsourced.

Experiment

For this experiment, the article 1957 Sunfield tornado on this version was chosen as the article to test on. Minor format modifications occurred to the article to split the article components into 4 parts; The lead, the infobox, the body (tornado summary), and the references. However, the text portion of the article as well as the references attached to each sentence remain unchanged. To test both hypotheses, a factually inaccurate sentence was created and placed in the article (While the National Weather Service rated the tornado F5, storm chaser Barry Allen rated the tornado F2.) as a way to test whether or not the large language model could identify it as unsourced information.

Experiment article

User:WeatherWriter/LLM Experiment 1

Meteorological history
Formed	December 18, 1957, 4:35 p.m. CST (UTC−06:00)
F5 tornado
on the Fujita scale
Highest winds	>261 mph (420 km/h)
Overall effects
Fatalities	1
Injuries	6
Damage	$250,000–$500,000 (1957 USD)
Part of the Tornado outbreak sequence of December 18–20, 1957 and tornado outbreaks of 1957

Lead

On December 18, 1957, a violent tornado struck Sunfield, Illinois, completely wiping out the community. The extreme damage in Sunfield caused the National Weather Service to assign a rating of F5 on the Fujita scale.

Tornado summary

The east-northeastward moving tornado struck Sunfield at the junction of U.S. Route 51 and Illinois Route 154, also called the Sunfield Y or Wye on account of the branching shape of the intersection. The United States Weather Bureau documented "very heavy destruction", albeit in a small area, and referred to the Sunfield Y as having been "wiped out". Tornado expert Thomas P. Grazulis stated that the “entire community literally vanished” Several people survived the tornado by taking cover in buildings, including some which were destroyed. A man who remained outside was killed, later found in a drainage ditch by the road. In total, the tornado killed one person, injured six others, and caused between $250,000 to $500,000 (1957 USD) in damage.

The United States Weather Bureau documented that the tornado reached a maximum width of 200 yards (180 m) and traveled 5 miles (8.0 km). They also documented that the tornado caused between $50,000 to $500,000 (1957 USD) in damage. After the creation of the Fujita scale in 1971, the National Weather Service assigned a rating of F5 to the tornado. While the National Weather Service rated the tornado F5, storm chaser Barry Allen rated the tornado F2. In the 2010s, the National Centers for Environmental Information published information about the tornado, which included a path length of 5.4 miles (8.7 km) and a damage total of $250,000 (1957 USD).

Experiment Steps

Step 1

1. After beginning a new chat with Chat-GPT4, the A.I. was provided with the following message: You can going to be provided with accurate information related to the 1957 Sunfield tornado. Each source provided to you will be labeled with the author's name as well as a [#]. You must keep track of the information from each source. Do not mix information together from different sources. Do you understand your task?

Step 2

1. To this prompt, the A.I. responded with: "Yes, I understand. You'll provide information from various sources, and I'll keep each source's information separate. Please go ahead and share the information related to the 1957 Sunfield tornado, and I'll reference each accordingly."

Step 3

Sources are given to Chat-GPT4
3.1. Chat-GPT4 was provided with the information from the first source, the United States Weather Bureau, with the following prompt: Source #1: Keep track of this source as the following:

"United States Weather Bureau, [1]."

Date: December 18

Time: 4:35 p.m.

Length of Path, Miles: 5

Width of Path, Yards: 200

Killed: 1

Injured: 0

Estimated Damage, Property: $50,000 to $500,000

Summary: Occurred at the junction of highways 51 and 154. Small crossroads settlement at Sunfield "Y" wiped out. Very heavy destruction in small area. Several survivors took ::cover in buildings. Man remaining in open killed. Tornado moved east-northeastward.

3.11. To this prompt, it responded:

Got it. Here's the information from the provided source:

United States Weather Bureau, [1].

• Date: December 18
• Time: 4:35 p.m.
• Length of Path, Miles: 5
• Width of Path, Yards: 200
• Killed: 1
• Injured: 0
• Estimated Damage, Property: $50,000 to $500,000
• Summary: Occurred at the junction of highways 51 and 154. Small crossroads settlement at Sunfield "Y" wiped out. Very heavy destruction in small area. Several survivors took cover in buildings. Man remaining in open killed. Tornado moved east-northeastward.
  

Please provide any additional sources you'd like to include.

NOTE: Chat-GPT4 added the bullet points as well as the bolding.

3.2. Chat-GPT4 was provided with the information from the second source, the National Centers for Environmental Information, with the following prompt: Source #2: Keep track of this source as the following:

"National Centers for Environmental Information, [2]"

Event: Tornado

Magnitude: 0

-- Scale: F5

-- Length: 5.4 Miles

-- Width: 200 Yards

State: ILLINOIS

County/Area: PERRY

NCEI Data Source: PUB

Begin Date: 1957-12-18 16:35 CST

Begin Location: 0

Begin Lat/Lon: 38.08/-89.25

End Date: 1957-12-18 16:35 CST

End Lat/Lon: 38.10/-89.15

Deaths Direct/Indirect: 1/0 (fatality details below, when available...)

Injuries Direct/Indirect: 6/0

Property Damage: 250K

Crop Damage: 0K

Event Fatality Details:

Death (Direct)

3.21. To this prompt, it responded: Understood. Here's the information from the second source:

National Centers for Environmental Information, [2]

• Event: Tornado
• Magnitude:
• Scale: F5
• Length: 5.4 Miles
• Width: 200 Yards
• State: ILLINOIS
• County/Area: PERRY
• NCEI Data Source: PUB
• Begin Date: 1957-12-18 16:35 CST
• Begin Lat/Lon: 38.08/-89.25
• End Date: 1957-12-18 16:35 CST
• End Lat/Lon: 38.10/-89.15
• Deaths Direct/Indirect: 1/0
• Injuries Direct/Indirect: 6/0
• Property Damage: 250K
• Crop Damage: 0K
• Event Fatality Details: Death (Direct)

Please provide any additional sources or let me know how I can further assist you.

NOTE: Chat-GPT4 added the bullet points as well as the bolding.

3.3. Source 3, the Storm Prediction Center, is a list of tornadoes officially rated F5 or EF5 by the National Weather Service. This list contains only the date and locations of the 59 tornadoes. For this reason, the source was only cited one time in the article, after the sentence about the tornado being rated F5. As no other information is provided by the source except a date and rating, Chat-GPT4 was given the following prompt: Source #3: Keep track of this source as the following:

"Storm Prediction Center, [3]"

Tornado date: December 18, 1957

Tornado rating: F5

3.31. To this prompt, it responded:

Noted. Here's the information from the third source:

Storm Prediction Center, [3]

• Tornado date: December 18, 1957
• Tornado rating: F5

Please let me know if there's more information to include or if you have any other instructions. NOTE: Chat-GPT4 added the bullet points as well as the bolding.

3.4. Source 4, WSIL-TV, was provided to Chat-GPT4 in a pure copy/paste form. To not violate copyright laws, the full prompt will not be posted. Here is the prompt given to Chat-GPT4: Source #4: Keep track of this source as the following:

WSIL-TV, [4]"

Copy/paste of full article, no other information provided.

3.41. To this prompt, it responded:

Understood. Here's the information from the fourth source:

WSIL-TV, [4]

• Article discusses the history of violent weather in southern Illinois, including the Tri-State Tornado in 1925, the Marion Tornado in 1982, Harrisburg in 2012, and Perryville in 2017.
• The tornado on December 18, 1957, is considered the strongest in the area for the past 70 years.
• An F-5 tornado is the strongest storm classification.
• Meteorologist Rick Shanklin mentions there have been only 62 F-5/EF-5 tornadoes worldwide since 1950, with only 2 in Illinois.
• Bud Mills, who was present during the 1957 tornado, recalls that the tornado was so strong that it blew away entire fences and left nothing but bare ground.
• The tornado crossed where Highway 51 intersects with Route 154, known then as the "Sunfield Y", causing immense destruction.
• The Tea Cup Inn, a two-story building, was completely destroyed by the tornado.
• At least one person was killed by this F-5 tornado.
• Since 1957, there has been extensive research on tornado damage, leading to the replacement of the Fujita Scale with the Enhanced Fujita Scale.
• To be rated F-5/EF-5, a tornado must cause severe damage, including carrying debris long distances.
• There is no physical evidence of the tornado left today.
• The December 18, 1957 outbreak is one of the worst in southern Illinois history, with other cities like Murphysboro, Carbondale, Mt. Vernon, and Dahlgren also sustaining damage.

Please let me know if there's more information to include or if you have any other instructions. NOTE: Chat-GPT4 added the bullet points as well as the bolding.
Interesting to note, Chat-GPT4 did not repeat the copyrighted information word-for-word back to the user, but rather summarized each part of the news article. Each bullet point repeated back to the user was carefully checked and matched with the news article and no fictional information was added by Chat-GPT4. However, aspects of the news article were not mentioned in bullet point list.

3.5. Source #5, The Southern Illinosian, is a source behind a paywall. As this source is only sourced one time in the sentence for the F5 rating, and this is behind a paywall, this source was not provided to ChatGPT. For the purpose of this experiment, this source will be ignored.

3.6 Source 6, Benton Evening News was provided to Chat-GPT4 in a pure copy/paste form. To not violent copyright laws, the full prompt will not be posted. Here is the prompt given to Chat-GPT4: Source #5: Keep track of this source as the following:

Benton Evening News, [6]"

Copy/paste of full article, no other information provided.

3.61. To this prompt, it responded:

Here's the information from the fifth source:

Benton Evening News, [6]

Bullet point list of the article.

Interesting to note, Chat-GPT4 did not repeat the copyrighted information word-for-word back to the user, but rather summarized each part of the news article. Each bullet point repeated back to the user was carefully checked and matched with the news article and no fictional information was added by Chat-GPT4. However, aspects of the news article were not mentioned in bullet point list. NOTE: Chat-GPT4 added the bolding.

3.7. Source #7, Grazulis's book, is a physical book. While the user conducting the experiment has access to the book, it is not going to be given to Chat-GPT4 as most users will not have access to physical books sourced on Wikipedia. Here is the prompt given to Chat-GPT4: Source #5: Keep track of this source as the following:

Source #6: Keep track of this source as the following:

"Grazulis, [7]"

This is a physical book where the exact information is unable to be accessed. Please remember this fact for the rest of the conversation.

3.7.1. To this prompt, it responded:

Understood. I've taken note of the source:

Grazulis, [7]

Information from this source is currently inaccessible since it's from a physical book.

Let me know if there's more information to include or if you have any other instructions.

NOTE: Chat-GPT4 added the bolding.

3.8. Source 8, NOAA Technical Report, is similar to the Storm Prediction Center source earlier, where it is an individual list of tornadoes rated F5. This source is listed only one time, however, it is listed as the sole source for a sentence. Here is the prompt given to Chat-GPT4: Source #7: Keep track of this source as the following:

Source #7: Keep track of this source as the following:

"NOAA Technical Report, [8]"

This is a list of officially rated F5 tornadoes. The 1957 Sunfield tornado is listed on the list.

3.8.1. To this prompt, it responded:

Got it. Here's the information from the seventh source:

NOAA Technical Report, [8]

The 1957 Sunfield tornado is officially rated as an F5 tornado according to a list of officially rated F5 tornadoes.

Please let me know if there's more information to include or if you have any other instructions.

NOTE: Chat-GPT4 added the bolding.

Step 4

4. The following prompt was given to Chat-GPT4: All sources have been provided. You will now be given a text of information. Your task will be to look at the information given to you and identify what source or sources state the specific information in the text. Do you understand your task?

4.1. To this prompt, the A.I. responded: Yes, I understand. Please provide the text, and I'll identify which source or sources state the specific information given in that text.

Step 5

Article body/tornado summary section

Paragraph 1

5. Chat-GPT4 was provided with this prompt: Here is the first part of the text. Please identify which source or sources mention each sentence of this text. Each source was assigned a [#]. List the "[#]"s after each sentence. After this prompt, the first paragraph of the tornado summary section was copy and pasted into the A.I. without any source numbers ([#]) after the sentences.

5.1. To this prompt, Chat-GPT4 responded with:

Here are the sources identified for each sentence in the provided text:

1. "The east-northeastward moving tornado struck Sunfield at the junction of U.S. Route 51 and Illinois Route 154, also called the Sunfield Y or Wye on account of the branching shape of the intersection."

• United States Weather Bureau, [1]
• WSIL-TV, [4]
• Benton Evening News, [6]

2. "The United States Weather Bureau documented "very heavy destruction", albeit in a small area, and referred to the Sunfield Y as having been "wiped out"."

• United States Weather Bureau, [1]

3. "Tornado expert Thomas P. Grazulis stated that the “entire community literally vanished”."

• Grazulis, [7] (Note: While I'm aware of this source's information being inaccessible, the mention of Grazulis specifically makes it relevant here.)

4. "Several people survived the tornado by taking cover in buildings, including some which were destroyed."

• United States Weather Bureau, [1]
• Benton Evening News, [6]

5. "A man who remained outside was killed, later found in a drainage ditch by the road."

• Benton Evening News, [6]

6. "In total, the tornado killed one person, injured six others, and caused between $250,000 to $500,000 (1957 USD) in damage."

• United States Weather Bureau, [1] (for the range of damage estimate and death toll)
• National Centers for Environmental Information, [2] (for the injury count and specific damage amount)

Do you have more text for me to analyze, or is there another task you'd like assistance with?

Paragraph 2

This paragraph has the factually incorrect and unsourced sentence.

5.2. Chat-GPT4 was provided with this prompt: Here is the second part of the text. Please identify which source or sources mention each sentence of this text. Each source was assigned a [#]. List the "[#]"s after each sentence. After this prompt, the second paragraph of the tornado summary section was copy and pasted into the A.I. without any source numbers ([#]) after the sentences.

5.3. To this prompt, Chat-GPT4 responded with:

Here are the sources identified for each sentence in the provided text:

Here are the sources identified for each sentence in the provided text:

1. "The United States Weather Bureau documented that the tornado reached a maximum width of 200 yards (180 m) and traveled 5 miles (8.0 km)."

• United States Weather Bureau, [1]

2. "They also documented that the tornado caused between $50,000 to $500,000 (1957 USD) in damage."

• United States Weather Bureau, [1]

3. "After the creation of the Fujita scale in 1971, the National Weather Service assigned a rating of F5 to the tornado."

This specific sentence isn't directly from the provided sources. However, the fact that the tornado was rated F5 can be sourced from:

• Storm Prediction Center, [3]
• WSIL-TV, [4]
• NOAA Technical Report, [8]

4. "While the National Weather Service rated the tornado F5, storm chaser Barry Allen rated the tornado F2."

• This information is not provided in the sources you've given.

5. "In the 2010s, the National Centers for Environmental Information published information about the tornado, which included a path length of 5.4 miles (8.7 km) and a damage total of $250,000 (1957 USD)."

• National Centers for Environmental Information, [2]

Let me know if there's more text to analyze or another task you'd like to proceed with!

Step 6

The lead

6. Chat-GPT4 was provided with this prompt: Here is the third part of the text. Please identify which source or sources mention each sentence of this text. Each source was assigned a [#]. List the "[#]"s after each sentence. After this prompt, the third paragraph of the tornado summary section was copy and pasted into the A.I. without any source numbers ([#]) after the sentences.

6.1. Chat-GPT4 responded with the following: Here are the sources identified for each sentence in the provided text:

"On December 18, 1957, a violent tornado struck Sunfield, Illinois, completely wiping out the community."

• United States Weather Bureau, [1]
• WSIL-TV, [4]
• Benton Evening News, [6]

"The extreme damage in Sunfield caused the National Weather Service to assign a rating of F5 on the Fujita scale."

• Storm Prediction Center, [3]
• WSIL-TV, [4]
• NOAA Technical Report, [8]

Do let me know if you have more text to analyze or any other queries!

Step 7

The Infobox

7. As the infobox is impossible to copy and paste into Chat-GPT4, given the format and layout of it, a series of questions was asked with the following prompt:

What sources provide the following information:

Formed: December 18, 1957, 4:35 p.m. CST (UTC−06:00)

Highest winds >261 mph (420 km/h)

Fatalities: 1

Injuried: 6

Damage: $250,000–$500,000 (1957 USD)

7.1. To this prompt, Chat-GPT4 responded with the following:

Based on the sources provided:

"Formed: December 18, 1957, 4:35 p.m. CST (UTC−06:00)"

• United States Weather Bureau, [1]
• National Centers for Environmental Information, [2]

"Highest winds >261 mph (420 km/h)"

• This specific wind speed detail was not provided in any of the sources you listed.

"Fatalities: 1"

• United States Weather Bureau, [1]
• National Centers for Environmental Information, [2]
• WSIL-TV, [4]
• Benton Evening News, [6]

"Injured: 6"

• National Centers for Environmental Information, [2]
• Benton Evening News, [6]

"Damage: $250,000–$500,000 (1957 USD)"

• United States Weather Bureau, [1] (provides a range of $50,000 to $500,000)
• National Centers for Environmental Information, [2] (specifically mentions $250,000)

These are the sources that provide the specific details listed.

Conclusion

Editors should always be aware of the known fact that large language models can provide false information. However, as seen in this experiment, a use for these large language models can be to help identify unsourced information. Part of the proposed Wikipedia policy for large language models (Wikipedia:Large language models) specifically states that editors can use them, if the content is scrutinized heavily. The results of this experiment showed that editors can copy/paste a source into a large language model, followed by a portion of a Wikipedia article and use the large language model to help identify what information is sourced, unsourced (but with source in article), and entirely unsourced.

Verification link

For transparency as well as verification, a complete link to the Chat-GPT4 conversation is able to be found here: https://chat.openai.com/share/14f89db8-9850-406a-a088-103e5b172e6d

Other experiments

• WP:LLMExperiment2