FineTrace-Bench

Version: 1.0.0
Status: Stable/Development
License: MIT

A benchmark framework for fine-grained requirements traceability analysis using Large Language Models (LLMs).

Overview

FineTrace-Bench is an automated tool that analyzes software requirements and source code to establish fine-grained traceability links. It uses LLMs to decompose coarse-grained requirements into fine-grained use cases and map them to specific code implementations.

Features

Automated Requirements Analysis: Decompose high-level requirements into fine-grained use cases
Code-Requirement Mapping: Establish traceability links between requirements and code files
Multi-Dataset Support: Process multiple software projects (eANCI, eTour, iTrust, LibEST)
LLM Integration: Support for Claude and Qwen models via API
Iterative Refinement: Automatic validation and correction of traceability links
CNL-B Format: Generate Controlled Natural Language use case descriptions

Project Structure

FineTrace-Bench-代码版/
├── main.py                  # Main entry point and workflow orchestration
├── dataset_manager.py       # Dataset loading and management
├── claude_llm_client.py     # Claude API client
├── qwenLLMClient.py        # Qwen API client
├── common_fun.py           # Utility functions (JSON/CSV handling, parsing)
├── datasets/               # Benchmark datasets
│   ├── eANCI/             # 55 requirements, 55 code files
│   ├── eTour/             # 116 requirements, 116 code files
│   ├── iTrust/            # 226 requirements, 226 code files
│   └── LibEST/            # 14 requirements, 14 code files
└── prompts/               # LLM prompt templates
    ├── system_prompt.txt
    ├── FineGrainedUseCase_Analyzer.txt
    ├── create_use_cases_step1.txt
    ├── create_use_cases_step2.txt
    └── note.txt

Configuration

Edit the LLM client configuration in qwenLLMClient.py or claude_llm_client.py:

api_base = "https://apis.iflow.cn"  # API endpoint
api_key = "your-api-key-here"       # Your API key
model_name = "qwen3-max"            # Model name
max_tokens = 20000                  # Maximum tokens
temperature = 0.1                   # Temperature setting

Update dataset paths in dataset_manager.py:

DATASETS_ROOT = r"D:-Traceability\FineTrace-Bench\datasets"
CURRENT_DATASET = "eANCI"  # Choose: eANCI, eTour, iTrust, LibEST

Usage

Basic Workflow

from dataset_manager import DatasetManager
from qwenLLMClient import qwenLLMClient

# Initialize dataset manager
dm = DatasetManager(name="eANCI")

# Load dataset structure
dataset_structure = dm.load_dataset_structure_data()

# Initialize LLM client
llm_client = qwenLLMClient()

# Run analysis (see main.py for complete workflow)

Run Complete Analysis

python main.py

Dataset Structure

Each dataset contains:

req/ - Requirement documents (text files)
code/ - Source code files (Java, etc.)
links.txt - Ground truth traceability links
all_req_filenames.txt - List of requirement files
all_code_filenames.txt - List of code files
README.txt - Dataset-specific information

Output Files

The tool generates:

{dataset}_analysis_results.json - Complete analysis results
{dataset}_structured_data.json - Structured traceability data
llm_response_step1.json - Initial LLM responses
llm_response_step2.json - Refined LLM responses
use_case/*.txt - Individual CNL-B use case files

Installation

Python 3.11.14 pip install -r requirements.txt

Key Components

DatasetManager

Manages dataset loading, filtering, and validation:

dm = DatasetManager(name="eANCI")
dm.load_dataset()  # Load requirements and code
dm.load_gold_standard()  # Load ground truth links
dm.filter_functional_requirements()  # Filter non-functional requirements

LLM Clients

Support for multiple LLM providers:

qwenLLMClient - Qwen model integration
ClaudeLLMClient - Claude model integration

Both support streaming and non-streaming modes.

Analysis Functions

analyze_requirement_with_retry() - Iterative requirement analysis with validation
validate_traceability() - Check completeness of traceability links
format_cnl_use_case() - Generate CNL-B format descriptions

API Reference

DatasetManager

load_dataset() - Load requirements and code files
load_gold_standard() - Load ground truth traceability links
filter_functional_requirements() - Filter functional requirements
load_dataset_structure_data() - Load complete dataset structure

LLM Client

generate(prompt) - Synchronous LLM generation
query_llm(prompt, system_prompt) - Asynchronous LLM query
load__Analyzer_prompt() - Load analysis prompt template

Utility Functions

save_to_json(data, file_path) - Save data to JSON
save_to_csv(data, file_path) - Save data to CSV
parse_llm_response(response_text) - Parse LLM JSON responses

Supported Datasets

Dataset	Requirements	Code Files	Status
eANCI	55	55	✅ OK
eTour	117	116	✅ OK
iTrust	226	226	✅ OK
LibEST	14	14	✅ OK

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contact

For questions, issues, or contributions, please contact:

Email: shiluoyuan1995@126.com