Can ChatGPT Write Your Next Scientific Paper?
Some of you may have heard about the new A.I. chatbot called ChatGPT, which was launched in November 2022. It's generating a lot of excitement and consternation because of its ability to answer questions, troubleshoot computer code, and write college essays that are surprisingly good. There are already dozens of YouTube videos touting its various uses, including aiding in scientific research and writing. Hi. I'm Karen McKee and I thought I would test out ChatGPT and see what it can and cannot do with respect to scientific writing. You can read about ChatGPT at openai.com. And you can try out ChatGPT at chat.openai.com. It s free to use at the moment and millions of people have already tested it. Registration requires an email address. This is the simple interface where you can check out some examples of the output and read about its capabilities and limitations. To the left are listed all your chats. Below are buttons to clear all conversations. Select light or dark mode. Link to Discord, which is another chat app by open ai. Read updates and frequently asked questions. And log out. And here is where you can type in your questions. I started with a question related to my own research. What is the effect of sea level rise on coastal wetlands? And in seconds it spits out an answer. Sea level rise can have a significant impact on coastal wetlands. As sea levels rise, saltwater can intrude into freshwater wetlands, causing a process called saltwater intrusion. This can lead to the loss of freshwater wetlands, as the saltwater can make the environment inhospitable for many of the plants and animals that typically live there. It goes on to talk about flooding impacts and loss of ecosystem services. And ends with a caveat that the effects of sea level rise will not be the same in all places. This is a surprisingly good answer to the question. I then tried to trip it up by asking what is the effect of sea level rise on beagles.
And it provided a sensible answer, pointing out that sea level rise would not likely have a direct effect on beagles as they are not normally found in coastal wetlands. But then talks about indirect effects via storms and flooding which could affect pets such as beagles and their owners. Pretty impressive. I next tried a variation on the sea level rise question, asking about responses of specific types of coastal wetlands. And again, I got another pretty good answer with three suggested mechanisms whereby coastal wetlands might accommodate sea level rise. The chatbot nicely bookends this information by initially pointing out that these wetlands can accommodate small rates of sea level rise and ends by saying that they may be negatively impacted by more rapid or extreme changes in sea level and that changes in sediment and water flows may make it more difficult for these ecosystems to survive sea level rise. Although I would word some things differently and modify the part about how vegetation aids in adjustment to sea level rise, in general, this response is reasonable. But, there s something missing. Citations for the sources of this information. When I asked for sources, I got this response. Which is not a good sign. Scientific papers must reference all sources of information. They need to be relevant and up-to-date. But let s see if this chatbot can help with coming up with a title for a paper. Here s a paper reporting on the effect of sea level rise on salt marshes and mangroves. I ll copy the abstract and paste it into ChatGPT. And ask it to suggest 5 titles for this abstract. I paste in that abstract and hit return. It s listed five titles, some not bad. Some not so great. But they represent five very different variations on a theme, which would get you thinking about some possible titles and lead to one that best represents the paper.Let's see if it can get us started on the introduction.
Write an introduction to a paper with this abstract. Coastal wetlands, such as mangroves and salt marshes, play a vital role in protecting coastal areas from the impacts of sea-level rise. These ecosystems are able to build soil surfaces vertically through the accumulation of inorganic sediment and organic matter, helping to keep the wetland surface above water. However, at climatic boundaries where mangroves are expanding and replacing salt marshes, the wetland s capacity to respond to sea level rise may change. Okay. That's not bad. It's taken the information in the abstract and the previous response to our question about sea level rise effects on coastal wetlands and changed the wording so that it doesn't sound too repetitive of the abstract. But there needs to be more information about mangroves and salt marshes and how they may differ in response to rising sea level. So, let's ask can you expand upon the characteristics of mangroves versus salt marshes in response to sea level rise? Okay. Both live in areas subject to tidal flooding but have some differences. One key difference is that mangroves have specialized roots allowing them to grow upwards whereas salt marshes do not. Well, that's not accurate. Both accumulate root matter contributing to soil volume and upward expansion of the soil surface. Next is salinity tolerance. No. both mangroves and salt marshes can grow in full strength seawater. Some mangrove species are better at tolerating hypersaline conditions but sea-level rise is not necessarily going to cause hypersalinity. So this answer shows that ChatGPT is relying on some information that is not exactly accurate or at least requires a more nuanced explanation. Let s see what happens when I question the accuracy of this response. I say that this information is not entirely accurate and briefly explain why. And the response is to accept the correction, even though I provided no evidence to support my criticism.
And it also modified the salt tolerance statements. Well, I think this exchange shows that you must be cautious in accepting what the chatbot gives you. I wonder how the chatbot will do with writing a discussion when given a set of results? I'll go back to the published paper and copy the results section which is fairly short. And ask it to write a discussion based on these results. And paste those in. As you can see, it's basically restated the results using slightly different wording and without any interpretation or comparison with previous work. What happens if I ask for a comparison with previous work? It replies that this cannot be done without more specific information. Okay, let s be more specific. Still not able to make a comparison. So, this exchange shows a serious limitation because of lack of access to such external sources. I would need to provide the details of the Perry and Mendelssohn study. Which is what we did in our paper's discussion and concluded that our results were similar. If you are a novice at writing, ChatGPT may be able to help you polish your paper. Let s ask it to improve a passage using active voice. Three mangrove species were subjected to treatment with freshwater or saltwater. Leaf number was negatively affected by salinity, but some species were more affected than others. This result was found in both greenhouse and field experiments. And we get, We subjected three mangrove species to treatment with freshwater or saltwater and found that salinity negatively affected leaf number. Some species were more affected than others, and this result was consistent across both greenhouse and field experiments. That flows better and is about the same number of words. Let s try another one. Can you improve this passage. The nitrogen concentration was high in January, a bit lower in February, even lower in March, and reached the lowest point in mid-July, after which the concentration increased from August to December.
And we get, the nitrogen concentration fluctuated throughout the year, with a high in January, a decrease in February, and a further decrease in March. The concentration reached its lowest point in mid-July, before increasing from august to December. That s better but not what I was looking for. I'll be more specific. Can you be more concise with fewer monthly details? The nitrogen concentration fluctuated throughout the year, reaching a high in January and a low in mid-July. It then increased from August to December. That's better. Let's try one more and ask it to make this paragraph more cohesive. You can pause the video to read it and the response. The paragraph now flows better and is not as choppy as the original. I could continue with more examples, but I think you get the idea. If you have a clunky passage, Chat GPT may be able to reword it for you. This may be especially helpful for those trying to write in a second language. What else might ChatGPT do? Can you suggest 3 directions for future research on this topic? And it suggests studies into the mechanisms underlying differences in elevation and other factors. Effect of differences on health and functioning of these ecosystems. And management interventions. All reasonable ideas that might be mentioned in the conclusions section of a paper reporting these results. What about help with social media and making your work more visible to the scientific community and beyond? Can you suggest 3 tweets based on this research? Well those are pretty good and it even suggests some hashtags. How about can you explain this research to a 12 year old? And we get. Sure. So, this research was all about how well different types of wetland plants, like mangroves and salt marsh grasses, can adapt to rising sea levels. When the sea level goes up, it can be hard for these plants to survive because they can get flooded by the water.
So, the researchers wanted to see how well different types of wetland plants can cope with this kind of change. To do this, they set up a bunch of plots with different types of plants in them and then measured how well the plants were doing over time. This wording might come in handy if you are interviewed by the news media or members of Congress. Especially if you struggle to put your research into everyday language. Okay. Let me wrap up by going over what I see as some pluses and minuses to using ChatGPT to write a scientific paper. On the positive side, ChatGPT can help you rewrite problematic passages in your manuscript, summarize results, or make suggestions for titles of your paper or future research directions.
And it provided a sensible answer, pointing out that sea level rise would not likely have a direct effect on beagles as they are not normally found in coastal wetlands. But then talks about indirect effects via storms and flooding which could affect pets such as beagles and their owners. Pretty impressive. I next tried a variation on the sea level rise question, asking about responses of specific types of coastal wetlands. And again, I got another pretty good answer with three suggested mechanisms whereby coastal wetlands might accommodate sea level rise. The chatbot nicely bookends this information by initially pointing out that these wetlands can accommodate small rates of sea level rise and ends by saying that they may be negatively impacted by more rapid or extreme changes in sea level and that changes in sediment and water flows may make it more difficult for these ecosystems to survive sea level rise. Although I would word some things differently and modify the part about how vegetation aids in adjustment to sea level rise, in general, this response is reasonable. But, there s something missing. Citations for the sources of this information. When I asked for sources, I got this response. Which is not a good sign. Scientific papers must reference all sources of information. They need to be relevant and up-to-date. But let s see if this chatbot can help with coming up with a title for a paper. Here s a paper reporting on the effect of sea level rise on salt marshes and mangroves. I ll copy the abstract and paste it into ChatGPT. And ask it to suggest 5 titles for this abstract. I paste in that abstract and hit return. It s listed five titles, some not bad. Some not so great. But they represent five very different variations on a theme, which would get you thinking about some possible titles and lead to one that best represents the paper.Let's see if it can get us started on the introduction.
Write an introduction to a paper with this abstract. Coastal wetlands, such as mangroves and salt marshes, play a vital role in protecting coastal areas from the impacts of sea-level rise. These ecosystems are able to build soil surfaces vertically through the accumulation of inorganic sediment and organic matter, helping to keep the wetland surface above water. However, at climatic boundaries where mangroves are expanding and replacing salt marshes, the wetland s capacity to respond to sea level rise may change. Okay. That's not bad. It's taken the information in the abstract and the previous response to our question about sea level rise effects on coastal wetlands and changed the wording so that it doesn't sound too repetitive of the abstract. But there needs to be more information about mangroves and salt marshes and how they may differ in response to rising sea level. So, let's ask can you expand upon the characteristics of mangroves versus salt marshes in response to sea level rise? Okay. Both live in areas subject to tidal flooding but have some differences. One key difference is that mangroves have specialized roots allowing them to grow upwards whereas salt marshes do not. Well, that's not accurate. Both accumulate root matter contributing to soil volume and upward expansion of the soil surface. Next is salinity tolerance. No. both mangroves and salt marshes can grow in full strength seawater. Some mangrove species are better at tolerating hypersaline conditions but sea-level rise is not necessarily going to cause hypersalinity. So this answer shows that ChatGPT is relying on some information that is not exactly accurate or at least requires a more nuanced explanation. Let s see what happens when I question the accuracy of this response. I say that this information is not entirely accurate and briefly explain why. And the response is to accept the correction, even though I provided no evidence to support my criticism.
And it also modified the salt tolerance statements. Well, I think this exchange shows that you must be cautious in accepting what the chatbot gives you. I wonder how the chatbot will do with writing a discussion when given a set of results? I'll go back to the published paper and copy the results section which is fairly short. And ask it to write a discussion based on these results. And paste those in. As you can see, it's basically restated the results using slightly different wording and without any interpretation or comparison with previous work. What happens if I ask for a comparison with previous work? It replies that this cannot be done without more specific information. Okay, let s be more specific. Still not able to make a comparison. So, this exchange shows a serious limitation because of lack of access to such external sources. I would need to provide the details of the Perry and Mendelssohn study. Which is what we did in our paper's discussion and concluded that our results were similar. If you are a novice at writing, ChatGPT may be able to help you polish your paper. Let s ask it to improve a passage using active voice. Three mangrove species were subjected to treatment with freshwater or saltwater. Leaf number was negatively affected by salinity, but some species were more affected than others. This result was found in both greenhouse and field experiments. And we get, We subjected three mangrove species to treatment with freshwater or saltwater and found that salinity negatively affected leaf number. Some species were more affected than others, and this result was consistent across both greenhouse and field experiments. That flows better and is about the same number of words. Let s try another one. Can you improve this passage. The nitrogen concentration was high in January, a bit lower in February, even lower in March, and reached the lowest point in mid-July, after which the concentration increased from August to December.
And we get, the nitrogen concentration fluctuated throughout the year, with a high in January, a decrease in February, and a further decrease in March. The concentration reached its lowest point in mid-July, before increasing from august to December. That s better but not what I was looking for. I'll be more specific. Can you be more concise with fewer monthly details? The nitrogen concentration fluctuated throughout the year, reaching a high in January and a low in mid-July. It then increased from August to December. That's better. Let's try one more and ask it to make this paragraph more cohesive. You can pause the video to read it and the response. The paragraph now flows better and is not as choppy as the original. I could continue with more examples, but I think you get the idea. If you have a clunky passage, Chat GPT may be able to reword it for you. This may be especially helpful for those trying to write in a second language. What else might ChatGPT do? Can you suggest 3 directions for future research on this topic? And it suggests studies into the mechanisms underlying differences in elevation and other factors. Effect of differences on health and functioning of these ecosystems. And management interventions. All reasonable ideas that might be mentioned in the conclusions section of a paper reporting these results. What about help with social media and making your work more visible to the scientific community and beyond? Can you suggest 3 tweets based on this research? Well those are pretty good and it even suggests some hashtags. How about can you explain this research to a 12 year old? And we get. Sure. So, this research was all about how well different types of wetland plants, like mangroves and salt marsh grasses, can adapt to rising sea levels. When the sea level goes up, it can be hard for these plants to survive because they can get flooded by the water.
So, the researchers wanted to see how well different types of wetland plants can cope with this kind of change. To do this, they set up a bunch of plots with different types of plants in them and then measured how well the plants were doing over time. This wording might come in handy if you are interviewed by the news media or members of Congress. Especially if you struggle to put your research into everyday language. Okay. Let me wrap up by going over what I see as some pluses and minuses to using ChatGPT to write a scientific paper. On the positive side, ChatGPT can help you rewrite problematic passages in your manuscript, summarize results, or make suggestions for titles of your paper or future research directions.