CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases

AI-Powered Repository Documentation Generation • Multi-Language Support • Architecture-Aware Analysis

Generate holistic, structured documentation for large-scale codebases • Cross-module interactions • Visual artifacts and diagrams

Quick Start • CLI Commands • Output Structure • Paper

Quick Start

1. Install CodeWiki

# Install from source
pip install git+https://github.com/FSoft-AI4Code/CodeWiki.git

# Verify installation
codewiki --version

2. Configure Your Environment

CodeWiki supports multiple models via an OpenAI-compatible SDK layer.

codewiki config set \
  --api-key YOUR_API_KEY \
  --base-url https://api.anthropic.com \
  --main-model claude-sonnet-4 \
  --cluster-model claude-sonnet-4 \
  --fallback-model glm-4p5

3. Generate Documentation

# Navigate to your project
cd /path/to/your/project

# Generate documentation
codewiki generate

# Generate with HTML viewer for GitHub Pages
codewiki generate --github-pages --create-branch

That's it! Your documentation will be generated in ./docs/ with comprehensive repository-level analysis.

Usage Example

What is CodeWiki?

CodeWiki is an open-source framework for automated repository-level documentation across seven programming languages. It generates holistic, architecture-aware documentation that captures not only individual functions but also their cross-file, cross-module, and system-level interactions.

Key Innovations

Innovation	Description	Impact
Hierarchical Decomposition	Dynamic programming-inspired strategy that preserves architectural context	Handles codebases of arbitrary size (86K-1.4M LOC tested)
Recursive Agentic System	Adaptive multi-agent processing with dynamic delegation capabilities	Maintains quality while scaling to repository-level scope
Multi-Modal Synthesis	Generates textual documentation, architecture diagrams, data flows, and sequence diagrams	Comprehensive understanding from multiple perspectives

Supported Languages

🐍 Python • ☕ Java • 🟨 JavaScript • 🔷 TypeScript • ⚙️ C • 🔧 C++ • 🪟 C#

CLI Commands

Configuration Management

# Set up your API configuration
codewiki config set \
  --api-key <your-api-key> \
  --base-url <provider-url> \
  --main-model <model-name> \
  --cluster-model <model-name> \
  --fallback-model <model-name>

# Show current configuration
codewiki config show

# Validate your configuration
codewiki config validate

Documentation Generation

# Basic generation
codewiki generate

# Custom output directory
codewiki generate --output ./documentation

# Create git branch for documentation
codewiki generate --create-branch

# Generate HTML viewer for GitHub Pages
codewiki generate --github-pages

# Enable verbose logging
codewiki generate --verbose

# Full-featured generation
codewiki generate --create-branch --github-pages --verbose

Customization Options

CodeWiki supports customization for language-specific projects and documentation styles:

# C# project: only analyze .cs files, exclude test directories
codewiki generate --include "*.cs" --exclude "Tests,Specs,*.test.cs"

# Focus on specific modules with architecture-style docs
codewiki generate --focus "src/core,src/api" --doc-type architecture

# Add custom instructions for the AI agent
codewiki generate --instructions "Focus on public APIs and include usage examples"

Pattern Behavior (Important!)

--include: When specified, ONLY these patterns are used (replaces defaults completely)
- Example: --include "*.cs" will analyze ONLY .cs files
- If omitted, all supported file types are analyzed
- Supports glob patterns: *.py, src/**/*.ts, *.{js,jsx}
--exclude: When specified, patterns are MERGED with default ignore patterns
- Example: --exclude "Tests,Specs" will exclude these directories AND still exclude .git, __pycache__, node_modules, etc.
- Default patterns include: .git, node_modules, __pycache__, *.pyc, bin/, dist/, and many more
- Supports multiple formats:
  - Exact names: Tests, .env, config.local
  - Glob patterns: *.test.js, *_test.py, *.min.*
  - Directory patterns: build/, dist/, coverage/

Setting Persistent Defaults

Save your preferred settings as defaults:

# Set include patterns for C# projects
codewiki config agent --include "*.cs"

# Exclude test projects by default (merged with default excludes)
codewiki config agent --exclude "Tests,Specs,*.test.cs"

# Set focus modules
codewiki config agent --focus "src/core,src/api"

# Set default documentation type
codewiki config agent --doc-type architecture

# View current agent settings
codewiki config agent

# Clear all agent settings
codewiki config agent --clear

Option	Description	Behavior	Example
`--include`	File patterns to include	Replaces defaults	`.cs`, `.py`, `src/*/.ts`
`--exclude`	Patterns to exclude	Merges with defaults	`Tests,Specs`, `*.test.js`, `build/`
`--focus`	Modules to document in detail	Standalone option	`src/core,src/api`
`--doc-type`	Documentation style	Standalone option	`api`, `architecture`, `user-guide`, `developer`
`--instructions`	Custom agent instructions	Standalone option	Free-form text

Configuration Storage

API keys: Securely stored in system keychain (macOS Keychain, Windows Credential Manager, Linux Secret Service)
Settings & Agent Instructions: ~/.codewiki/config.json

Documentation Output

Generated documentation includes both textual descriptions and visual artifacts for comprehensive understanding.

Textual Documentation

Repository overview with architecture guide
Module-level documentation with API references
Usage examples and implementation patterns
Cross-module interaction analysis

Visual Artifacts

System architecture diagrams (Mermaid)
Data flow visualizations
Dependency graphs and module relationships
Sequence diagrams for complex interactions

Output Structure

./docs/
├── overview.md              # Repository overview (start here!)
├── module1.md               # Module documentation
├── module2.md               # Additional modules...
├── module_tree.json         # Hierarchical module structure
├── first_module_tree.json   # Initial clustering result
├── metadata.json            # Generation metadata
└── index.html               # Interactive viewer (with --github-pages)

Experimental Results

CodeWiki has been evaluated on CodeWikiBench, the first benchmark specifically designed for repository-level documentation quality assessment.

Performance by Language Category

Language Category	CodeWiki (Sonnet-4)	DeepWiki	Improvement
High-Level (Python, JS, TS)	79.14%	68.67%	+10.47%
Managed (C#, Java)	68.84%	64.80%	+4.04%
Systems (C, C++)	53.24%	56.39%	-3.15%
Overall Average	68.79%	64.06%	+4.73%

Results on Representative Repositories

Repository	Language	LOC	CodeWiki-Sonnet-4	DeepWiki	Improvement
All-Hands-AI--OpenHands	Python	229K	82.45%	73.04%	+9.41%
puppeteer--puppeteer	TypeScript	136K	83.00%	64.46%	+18.54%
sveltejs--svelte	JavaScript	125K	71.96%	68.51%	+3.45%
Unity-Technologies--ml-agents	C#	86K	79.78%	74.80%	+4.98%
elastic--logstash	Java	117K	57.90%	54.80%	+3.10%

View comprehensive results: See paper for complete evaluation on 21 repositories spanning all supported languages.

How It Works

Architecture Overview

CodeWiki employs a three-stage process for comprehensive documentation generation:

Hierarchical Decomposition: Uses dynamic programming-inspired algorithms to partition repositories into coherent modules while preserving architectural context across multiple granularity levels.
Recursive Multi-Agent Processing: Implements adaptive multi-agent processing with dynamic task delegation, allowing the system to handle complex modules at scale while maintaining quality.
Multi-Modal Synthesis: Integrates textual descriptions with visual artifacts including architecture diagrams, data-flow representations, and sequence diagrams for comprehensive understanding.

Data Flow

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Codebase      │───▶│  Hierarchical    │───▶│  Multi-Agent    │
│   Analysis      │    │  Decomposition   │    │  Processing     │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │                        │
                                ▼                        ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Visual        │◀───│  Multi-Modal     │◀───│  Structured     │
│   Artifacts     │    │  Synthesis       │    │  Content        │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Requirements

Python 3.12+
Node.js (for Mermaid diagram validation)
LLM API access (Anthropic Claude, OpenAI, etc.)
Git (for branch creation features)

Additional Resources

Documentation & Guides

Docker Deployment - Containerized deployment instructions
Development Guide - Project structure, architecture, and contributing guidelines
CodeWikiBench - Repository-level documentation benchmark
Live Demo - Interactive demo and examples

Academic Resources

Paper - Full research paper with detailed methodology and results
Citation - How to cite CodeWiki in your research

Citation

If you use CodeWiki in your research, please cite:

@misc{hoang2025codewikievaluatingaisability,
      title={CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases},
      author={Anh Nguyen Hoang and Minh Le-Anh and Bach Le and Nghi D. Q. Bui},
      year={2025},
      eprint={2510.24428},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2510.24428},
}

Star History

License

This project is licensed under the MIT License.

Name		Name	Last commit message	Last commit date
Latest commit History 75 Commits
codewiki		codewiki
docker		docker
docs		docs
img		img
.DS_Store		.DS_Store
.gitignore		.gitignore
DEVELOPMENT.md		DEVELOPMENT.md
README.md		README.md
index.html		index.html
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

FSoft-AI4Code/CodeWiki

Folders and files

Latest commit

History

Repository files navigation