Perhaps the most challenging part of designing AI detection user experience is how to effectively convey detection results. When a paragraph of content is marked as "written by AI", the user will then ask: Why?

Our detection model outputs a numerical confidence score for each prediction category (mainly “AI,” “Human,” or “Mixed”). However, it is much harder to explain why texts are classified as AI-generated or human-written.

First, our model is able to learn the complex relationships between phrases in a document and use these relationships to distinguish AI-generated and human-written texts. But these internal mechanisms do not necessarily correspond to intuitive concepts such as AI writing styles (such as "clichés") that we usually understand.

In addition, AI-generated text is not always completely different from human-written text, nor is it completely mutually exclusive label. For example, if you type in ChatGPT: "Recite the first paragraph of the Declaration of Independence", it will output that famous text intact. So, does it mean that the founding literature of the United States should be classified as AI-generated content now?

Make AI detection visible

Despite the inherent difficulties, it is still our responsibility to help users understand the “reasons” behind AI detection as much as possible. Over the past year, we have launched a series of tools to achieve this. For example, our AI Vocabulary feature highlights words and phrases that are more likely to appear in text generated by large language models. In addition, we recently added Natural Language Interpretation (NLE) functionality to our core detection model. NLE describes the similarities between the scanned text and the AI text in the training set in an easy-to-understand English.

GPTZero's latest attempt to improve the intelligibility of AI detection is to launch a new feature: Compare with AI Text (internal codenamed LLM Parrot ), a tool that allows users to compare their suspected AI text with known AI generated text obtained directly from the OpenAI API. Two paragraphs of text are displayed side by side, and similar words will be highlighted. This comparison method provides users with personalized analysis results, not only covering common AI expression patterns (such as " play a significant role in shaping "), but also providing context adaptation based on the type of content analyzed.

The "Compare with AI Text" tool interface used in the paper on "The Impact of Gulfstream on European Climate"

To use the "Compare with AI Text" function, you can write prompts manually or select "Auto Generate", and the system will automatically create a targeted prompt of reasonable length based on the text you entered. For example, if you submit a paper about the construction process of the National Television Tower in Canada, the system may generate a prompt like "Writing a 500-word paper about the construction process of the CN tower." The prompt is then sent to ChatGPT, and the response it returns is directly retrieved from the OpenAI server and presented.

Why develop this feature?

Before this, some users have tried to perform similar operations manually: copy their text into ChatGPT, require similar content to be generated, and then compare line by line to find the overlapping part. This process is time-consuming and tedious, and it is easy to miss some matching content. Compare with AI Text is designed to automate this process, while also identifying close-match fragments that are difficult to detect in humans.

Once the full response of ChatGPT is downloaded, the system highlights similar parts of the two texts and adds a number next to the matching paragraphs to identify the corresponding relationship. For example, in the screenshot, the expression "the eastern coast of the United States" appears in both the text explicitly generated by ChatGPT and the text box submitted by the user (in this case, the latter is actually generated by ChatGPT).

Although the example above contains a large number of completely consistent text matches, we use a fuzzy string matching algorithm to compare OpenAI generated text with the original user input text, so that we can also identify similar text pairs. The algorithm is based on the improved version of Levenshtein distance , looking for text fragments that require only a small amount of editing (such as replacing individual words) to be exactly the same.

Using this feature is very simple: start an advanced scan (or switch to the advanced scan view through the right sidebar), click Compare with AI Text , enter a prompt (or select Automatically generate), and then click "Generate AI Text". After the ChatGPT response is completed, the matching result will automatically appear.

What are the actual meanings of these matching results

This feature is an important step in helping GPTZero users visualize "AI similarity". It runs independently of our core AI detection model and provides an external, independent reference for signs of AI usage. This means that even if a document is judged by our core model as "probably generated for AI", it is not necessarily possible to find matching content to ChatGPT (or other LLM) in this feature.

Vice versa, a document classified as “human-written” by GPTZero may also appear in comparison multiple matches to AI-generated texts (recalling the aforementioned example of the Declaration of Independence, whose output is essentially “human-created”). In addition, the large language model is essentially a probabilistic system, and the content generated is slightly different each time, so the matching results will also change accordingly.

Nevertheless, according to empirical observations, AI-generated texts usually produce significantly more matches than human writing. Taking this article as an example, under the following automatically generated prompt, no match was found in the Compare with AI Text function:

"Writing a detailed product description, introducing a new AI detection tool feature that enhances users' understanding of AI content classification by comparing user-submitted text with AI-generated text."

By simplifying the comparison process between user text and reasonable AI-generated text, we further expand the overall interpretability tool set, providing users with unprecedented transparency and confidence in our AI detection results.

The above is the detailed content of Inside Our New 'Compare with AI Text' Tool. For more information, please follow other related articles on the PHP Chinese website!