摘要

SoapFL是一个基于大模型的项目级软件缺陷自动定位系统，其结合的大模型的代码理解能力和测试覆盖信息，可以依据“原因分析-错误位置搜索-错误确认”的工作流进行缺陷自动定位。

相关工作已发表于 TSE 期刊 “SoapFL: A Standard Operating Procedure for LLM-based Method-Level Fault Localization”。

环境

Defects4J-V1.4.0 注意：V1.4.0 中的 buggy items 与 V1.2.0 完全相同。我们使用 V1.4.0 是为了避免 V1.2.0 中的一些问题。
Defects4J-V2.0.0
Python 版本 >= 3.8.5
Defects4J Mod

在运行 SoapFL 之前，请将 Defects4J_mod 目录下的文件应用到你的 Defects4J V1.4.0/V2.0.0 中，以修改 Defects4J。

运行 SoapFL

在 camel/model_backend.py 中设置你自己的 OpenAI API key。

可以使用以下命令轻松运行 SoapFL 来定位某个 bug：

python3 run.py --config <CONFIG_DIR> --version <D4J_VERSION> --project <PROJECT> --bugID <BUG_ID> --model <GPT_MODEL_NAME>

例如：

python3 run.py --config Default --version 1.4.0 --project Closure --bugID 26 --model GPT_3_5_TURBO

更多配置可以在 Config 目录下查看。

结果

我们在在线仓库中发布了 SoapFL 的所有结果，包括在 Defects4J V1.4.0/V2.0.0 上的评估结果，以及消融实验结果。

在每个 bug 的结果中，我们记录了所有 prompt、response 和中间输出。

人工评估结果可以在以下文件中找到：

EvaluationResult/DebugResult_d4j140_GPT35_human.xlsx

Agents 的 System Prompt

Test Code Reviewer:

You are a Test Code Reviewer. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You can examine the test code and the initialized classes to analyze the similar behavior of the failed tests within the test suite. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

Source Code Reviewer

You are a Source Code Reviewer. we are both working at DebugDev. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. Your main responsibilities is to generate a comment for each covered method base on the method call relationship. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

Software Test Engineer

You are a Software Test Engineer. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You main responsibilities include examining the information of the failed tests to analyze the possible causes of the test failures, and determining the method that need to be fixed. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.

Software Architect

You are a Software Architect. We share a common interest in collaborating to successfully locate the buggy code that cause the test suite to fail. You are very familiar with the architecture of the software, the functions of each class and method in the software. You main responsibilities include examining the given information to locate the possible buggy classes and buggy methods. To locate the bug, you must write a response that appropriately solves the requested instruction based on your expertise.