Investigating allegations of inappropriate GAI tool use in the conduct and reporting of research can be frustrating or even futile. I describe a scenario about an allegation of GAI tool use in a published article to illustrate the challenges.
Open science practices that help researchers to authenticate their work might help to tackle this problem.
The scenario: An allegation of inappropriate GAI use in an article.
A reader writes to you telling you that your journal recently published an article that is written using generative AI (GAI) tools . You read the article. It is a narrative review of already published research so there is no novel data. The authors could have generated the entire article using GAI tools. Perhaps the reader is right. The reader used a publicly available tool to assess the article. You can’t find any information about whether or how the tool was validated. You don’t know whether it can be trusted.
You decide to ask the authors whether they used any AI tools in the conduct and reporting of their research. The senior author responds that they merely used tools to check the grammar, nothing more.
What do you do? Believe the authors or the reader and the tool they used? If you believe the authors and you are wrong, you may have a fabricated article published in your journal which may be cited and referred to by other researchers. This would be concerning because the article ends with a list of clinical recommendations. If you believe the reader and are wrong, any actions you take might harm the reputation of innocent authors and open you and your journal to the risk of legal action.
You decide that you cannot dismiss the reader’s concerns but you can’t take any action based solely on the output of an unvalidated tool. You want evidence that the research reported in the article was carried out in the real world by human researchers and not entirely fabricated by a GAI tool.
The investigation
The usual next step would be to ask the author’s for proof that they conducted the research. This might take the form of asking for raw data or original images, or going directly to the authors’ institution to ask for a formal investigation. Even with non-AI generated content cases, asking for raw data is fraught with difficulties because it is easy to generate a table of ‘raw data’ or fabricate an ‘original’ image. In this case, there is no data to ask for.
You ask the authors whether they can provide evidence of its authenticity such as a first draft of the manuscript and the revisions that were made to it (although these can be fabricated too). The senior author responds that they cannot provide previous versions of their article. They were stored locally on an old computer which they no longer have.
You have the option of asking the authors’ institution for help. What, exactly, are you going to ask the institution to do? A vague request for an ‘investigation’ will produce a vague response. You have to be specific about your concerns. You think through what is bothering you about the article. It is not the use of AI tools in itself that’s the problem, it’s the inappropriate use of these tools and the possibility that the entire article was fabricated.
What is inappropriate use?
You decide on the following:
| It is OK to use AI based tools to: | It is not OK to use AI based tools to: |
|---|---|
| Check spelling, grammar and formatting. | Draw conclusions from the literature. |
| Search the literature. | Make clinical recommendations. |
| Summarise the literature to help the authors decide which articles to include and exclude. | Generate tables, graphs and figures without any underlying ‘real-world’ data. |
| Create an image that represents data collected by the authors | Write the entire manuscript with no human input. |
Ultimately, you want to know to what extent the authors used their own human clinical experience and judgement to analyse and synthesise the research they found into clinical recommendations.
How can the application of human experience and judgement be investigated and proven? The institution may be in no better position to investigate that than you are, especially if the authors maintain that they have lost previous versions or their work. Nevertheless, you contact the institution to ask whether they are able to investigate. You don’t get any response, despite weeks of chasing. Meanwhile, the reader has been chasing you for updates on how your investigation is going and when you will take action.
The conclusion
You accept that you cannot prove the extent to which the conclusions and recommendations in the article were generated by the authors themselves. The only option left to you is to publish an ‘Expression of Concern’ to alert readers about potential inappropriate GAI tool use. However, when you inform the authors of your decision, they deny any inappropriate use of GAI tools and threaten legal action.
You have to involve your publisher, and with their lawyers agree on a version of wording for the Expression of Concern that will mitigate the risks of a legal backlash. You regret that it no longer fully conveys what your concerns were but also feel uncomfortable about the possibility that you may be wrong and the authors entirely innocent.
You are left feeling jaded and unlikely to pursue a similar concern in the same way, if at all, again.
None of this scenario is uncommon or exaggerated.
A solution?
With the use of GAI tools likely to increase, along with tools to detect their use, it is reasonable to expect that concerns about GAI content in submitted and published scholarly works will be raised more frequently. As the scenario illustrates, it is challenging to prove the veracity of a manuscript or article retrospectively if the authors did not maintain an accessible record of their research process.
For journals, adopting policies that encourage open science practices, such as requiring authors to show time-stamped documentation of the history of their research (via tools such as the Open Science Framework) will become an increasingly important way to authenticate their work.
Focusing on a manuscript’s provenance, as well as screening its content, might be the only way to prevent the literature from becoming flooded with Expressions of Concern that encompass the works of innocent and guilty researchers alike.
The text for this blog post was written without the use of GAI tools.. The image was generated using Microsoft Copilot.