TNH Scholar TODO List¶

Roadmap tracking the highest-priority TNH Scholar tasks and release blockers.

Last Updated: 2026-02-08 (Updated ADR-CF02 prompt discovery status) Version: 0.3.1 (Alpha) Status: Active Development - Bootstrap path complete, production hardening phase

Style Note: Tasks use descriptive headers (not numbered items) to avoid renumbering churn when reorganizing. Use #### (h4) for task headers within priority sections.

Progress Summary¶

Bootstrap Path Status: ✅ COMPLETE — VS Code integration working, AI-assisted development enabled.

Next Steps:

🔮 JVB VS Code Parallel Viewer (P1, design phase) — ADR-JVB02 strategy + UI-UX design
🔮 Finish yt-dlp reliability suite + monthly ops trigger (P1, reliability)
🔮 Finish ytt-fetch robustness hardening (P1, reliability)
🔮 Add --prompt-dir Global Flag to tnh-gen (P1, minor)
🚧 GenAIService Final Polish - promote policy_applied typing (P1, minor)
🚧 Prompt Catalog Safety - error handling, validation (P2, critical infrastructure)
🚧 Knowledge Base Implementation (P2, design complete)
🚧 Expand Test Coverage to 50%+ (P2)

For completed items: See Archive section at end.

Priority Roadmap¶

This section organizes work into three priority levels based on criticality for production readiness.

Priority 1: VS Code Integration Enablement (Bootstrap Path)¶

Goal: Enable AI-assisted development of TNH Scholar itself via VS Code extension. Prioritizes foundational work for tnh-gen + extension integration.

Status: Foundation Complete (tnh-gen CLI ✅, Registry System ✅)

✅ tnh-gen CLI Implementation — See Archive¶

✅ File-Based Registry System (ADR-A14) — See Archive¶

✅ VS Code Extension Walking Skeleton — See Archive¶

✅ Pattern→Prompt Migration — See Archive¶

✅ Provenance Format Refactor (YAML Frontmatter) — See Archive¶

🔮 JVB VS Code Parallel Viewer (ADR-JVB02)¶

Status: NOT STARTED (Design Phase)
Priority: HIGH (flagship feature, builds on VS Code integration foundation)
Context: The JVB (Journal of Vietnamese Buddhism) parallel viewer enables scholars to view scanned historical journal pages alongside OCR text and English translations. v1 was a bespoke browser-based prototype; v2 will integrate into the tnh-scholar VS Code extension.
Project Paused: This work was on hold while VS Code integration and tnh-gen were developed. Now that the walking skeleton is complete, we can resume with fresh design.

Related Documentation:

v1 As-Built: ADR-JVB01 — Browser-based prototype architecture
v2 Strategy (Draft): JVB Viewer V2 Strategy — Pre-ADR strategy note (good foundations, needs formalization)
VS Code Platform Strategy: VS Code as UI Platform — Overall UI-UX direction
VS Code Integration: ADR-VSC01 — CLI-first extension strategy (implemented)

Proposed ADR Structure:

docs/architecture/jvb-viewer/adr/
├── adr-jvb01_as-built_jvb_viewer_v1.md              # ✅ Exists
├── adr-jvb02-vscode-parallel-viewer-strategy.md     # 🆕 Main strategy ADR
├── adr-jvb02.1-ui-ux-design.md                      # 🆕 Mockups, pane layout, workflows
├── adr-jvb02.2-data-model-api-contract.md           # 🆕 JSON schema, extension↔backend API
└── adr-jvb02.3-implementation-guide.md              # 🆕 Phase-by-phase implementation

Key Design Decisions Needed:

VS Code Pane Architecture: Which panes for scan overlay, text views, reconciliation controls, navigation?
Webview vs Custom Editor: Custom editor for .jvb.json files or webview panel approach?
Backend Integration: Python service via CLI (tnh-gen patterns) or dedicated HTTP service?
Data Model: Refine per-page JSON schema from v2 strategy, define API contract
Dual OCR Reconciliation UI: How users choose between Google OCR vs AI vision sources

Deliverables:

ADR-JVB02: Main strategy ADR (formalize v2 strategy, VS Code integration focus)
ADR-JVB02.1: UI-UX design with mockups/screen visualizations
ADR-JVB02.2: Data model and API contract specification
ADR-JVB02.3: Implementation guide with milestones
M0 Prototype: Static HTML mockup in VS Code webview (validate approach)

Implementation Milestones (from v2 strategy, to be refined):

M0: Static prototype — HTML showing page image, word bboxes, selectable sentences
M1: VS Code extension — load/save per-page JSON, overlay modes, section breadcrumb
M2: Dual-source UI — GOCR vs AI diff chooser, batch adoption, "reviewed" status
M3: Structure cues — columns, heading levels, emphasis flags captured and rendered
M4: Beta — section-level navigation, export HTML, light theming

🔮 Add `--prompt-dir` Global Flag to tnh-gen¶

Status: NOT STARTED
Priority: HIGH (improves tnh-gen UX for one-off operations and testing)
Estimate: 1-2 hours
Context: Users need convenient way to override prompt catalog directory for one-off CLI calls without setting environment variables or creating temp config files
ADR: ADR-TG01 Addendum 2026-01-02
Why Important: Enables clean one-off operations (tnh-gen --prompt-dir ./test-prompts list) for testing, CI/CD, and development workflows
Current Workarounds:
Environment variable: TNH_PROMPT_DIR=/path tnh-gen list (awkward)
Temp config file: tnh-gen --config /tmp/config.yaml list (verbose)
Deliverables:
Add --prompt-dir flag to cli_callback() in src/tnh_scholar/cli_tools/tnh_gen/tnh_gen.py:26
Update config_loader.py to handle prompt directory override at CLI precedence level
Update ConfigData type to accept prompt_catalog_dir override
Add unit tests for flag precedence (CLI flag > workspace > user > env)
Update help text and CLI reference documentation
Update docs/cli-reference/tnh-gen.md global flags section
Files to Modify:
src/tnh_scholar/cli_tools/tnh_gen/tnh_gen.py (add flag)
src/tnh_scholar/cli_tools/tnh_gen/config_loader.py (precedence handling)
src/tnh_scholar/cli_tools/tnh_gen/types.py (type definitions)
tests/cli_tools/test_tnh_gen.py (unit tests)
docs/cli-reference/tnh-gen.md (documentation)
Testing: Verify --prompt-dir flag overrides all other config sources (workspace, user, env)

🔮 Full-Coverage yt-dlp Test Suite + Monthly Ops Trigger¶

🔮 Patch ytt-fetch Robustness¶

Status: IN PROGRESS
Priority: HIGH (frequent breakage path)
Goal: Make ytt-fetch resilient to upstream changes and failures.
Test URL: https://youtu.be/iqNzfK4_meQ
Deliverables:
Add runtime preflight + yt-dlp runtime option injection
Verify transcript fetch on test URL (manual + test)
Add retries / improved error reporting
Ensure metadata embed + output path handling remain stable
Update docs and CLI reference if flags or behaviors change

🚧 GenAIService Core Components - Final Polish¶

⏸️ GenAIService Thread Safety and Rate Limiting (ADR-A15)¶

Status: DEFERRED - Not needed for VS Code integration (process isolation)
Priority: MEDIUM (revisit when building Python batch pipelines)
Issue: #22
ADR: ADR-A15: Thread Safety and Rate Limiting
Why Deferred: VS Code extension uses process isolation (each tnh-gen call = separate GenAIService instance). Thread safety only matters for Python-native batch pipelines.
When to Revisit: When implementing concurrent corpus processing loops or batch translation pipelines
Estimate: 3-6 hours (Phase 1: 1-2 hours, Phase 2: 2-4 hours)
Quick Summary: Add thread-safe retry state, locked cache, and optional rate limiting for high-throughput scenarios

Priority 2: Production Hardening (Post-Bootstrap)¶

Goal: Harden TNH Scholar for production use after VS Code integration enables AI-assisted development. Focuses on reliability, test coverage, and type safety.

🚧 OpenAI SDK 2.15.0 Validation (High Priority)¶

Status: NOT STARTED
Why: SDK bump impacts OpenAI adapter. (Codex harness suspended — see ADR-OA03.2 addendum)
Tasks:
Revalidate OpenAI adapter request/response mappings against 2.15.0
Update compatibility notes/docs if schema drift is found

🚧 Audio-Transcribe Service-Layer Refactor (P2)¶

Status: NOT STARTED
Goal: Align audio-transcribe with object-service pattern and ytt-fetch robustness.
Tasks:
Introduce typed service orchestrator + protocols (CLI becomes thin wrapper)
Extract audio source resolution into a typed resolver (yt_url/CSV/local file)
Replace dict options with Pydantic models (transcription + diarization params)
Add runtime preflight (yt-dlp inspector + ffmpeg availability); keep version checks ops-only
Migrate CLI to Typer with minimal surface (smoke tests only)
Add service-layer tests for all audio-transcribe use cases

⏸️ Agent Orchestration - Codex Runner (ADR-OA03.2)¶

Status: TABLED (2026-01-25)
ADR: ADR-OA03.2
Why Tabled:
Scope: Spike revealed that a proper Codex harness requires implementing full client-side agent orchestration (the VS Code extension uses a proprietary app server, not raw API calls)
Cost-benefit: Current human-in-the-loop workflow with Claude Code + VS Code Codex extension is effective and cost-efficient for project needs
No compelling need: Investment not justified when manual workflow works well
Findings: Codex Harness Spike Findings
Preserved Artifacts: src/tnh_scholar/agent_orchestration/codex_harness/, src/tnh_scholar/cli_tools/tnh_codex_harness/
Conditions for Resumption: Further insight or clear business need that justifies full agent orchestration investment

🚧 Expand Test Coverage¶

🚧 Consolidate Environment Loading¶

Status: NOT STARTED
Problem: Multiple modules call load_dotenv() at import time
https://github.com/aaronksolomon/tnh-scholar/blob/main/src/tnh_scholar/ai_text_processing/prompts.py
https://github.com/aaronksolomon/tnh-scholar/blob/main/src/tnh_scholar/audio_processing/diarization/pyannote_client.py
Tasks:
Create single startup hook for dotenv loading
Use Pydantic Settings consistently
Pass configuration objects instead of os.getenv() calls
Remove import-time side effects

🚧 Configuration Tech Debt — Migrate to ADR-CF01/CF02 Three-Layer Model¶

Status: PHASES 1-3 COMPLETE, Phase 4-5 NOT STARTED
Priority: MEDIUM (foundational, not blocking current work)
ADRs:
ADR-CF01: Runtime Context & Configuration Strategy
ADR-CF02: Prompt Catalog Discovery Strategy (status: accepted)
Related: ADR-A08: Config/Params/Policy Taxonomy

Migration Phases:

Success Criteria: - [x] No module-level config Path constants in __init__.py - [x] Prompt path discovery flows through TNHContext - [x] Prompt directories follow three-layer precedence (workspace → user → built-in) - [ ] At least tnh-gen and audio-transcribe share config loader pattern

🚧 Clean Up CLI Tool Versions¶

Status: PARTIAL (old versions removed, utilities pending)
Location: cli_tools/audio_transcribe/
Tasks:
Remove audio_transcribe0.py
Remove audio_transcribe1.py
Remove audio_transcribe2.py
Keep only current version
Create shared utilities (argument parsing, environment validation, logging)

✅ Documentation Reorganization (ADR-DD01 & ADR-DD02) — See Archive¶

Phase 1 COMPLETE - Remaining Phase 2 tasks:

Doc metadata validation script (check_doc_metadata.py) - validate front matter
Docstring coverage (interrogate) - threshold on src/tnh_scholar
Archive index + legacy ADR migration to docs/archive/**
Backlog: populate docs/docs-ops/roadmap.md with missing topics
User guides for new features, architecture component diagrams

🚧 Type System Improvements¶

Status: PARTIAL
Current: 58 errors across 16 files
High Priority: Fix audio processing boundary types, core text processing types, function redefinitions
Medium Priority: Add missing type annotations, fix Pattern class type issues
Low Priority: Clean up Any return types, standardize type usage

🚧 Prompt Catalog Safety¶

Status: NOT STARTED
Priority: HIGH (critical infrastructure)
Problem: Adapter doesn't handle missing keys or invalid front-matter gracefully
Tasks:
Add manifest validation
Implement caching
Better error messages (unknown prompt, hash mismatch)
Front-matter validation
Document prompt schema

🚧 Knowledge Base Implementation¶

Status: DESIGN COMPLETE
ADR: ADR-K01
Tasks:
Implement Supabase integration
Vector search functionality
Query capabilities
Semantic similarity search

🚧 Configuration & Data Layout¶

Status: NOT STARTED
Priority: HIGH (blocks pip install)
Problem: src/tnh_scholar/__init__.py raises FileNotFoundError when repo layout missing
Tasks:
Package pattern assets as resources
Make patterns directory optional
Move directory checks to CLI entry points only
Ensure installed wheels work without patterns/ directory

🚧 Logging System Scope¶

Location: src/tnh_scholar/logging_config.py
Problem: Modules call setup_logging individually
Tasks:
Define single application bootstrap
Document logger acquisition pattern (get_logger only)
Create shared CLI bootstrap helper

🚧 Comprehensive CLI Reference Documentation¶

Status: IN PROGRESS (tnh-gen complete ✅, other CLIs pending)
Tasks:
Update user-guide examples to use tnh-gen
Document other CLI tools (audio-transcribe, ytt-fetch, nfmt, etc.)
Consider automation for CLI reference generation

🔮 Shared CLI UI Module (tnh_cli_ui)¶

Status: NOT STARTED (Research/Exploration)
Priority: MEDIUM (UX consistency across CLI tools)
ADR: ADR-ST01.1: tnh-setup UI Design
Context: The tnh-setup UI redesign (Rich library) could be extracted into a shared module for consistent styling across all tnh-scholar CLI tools.
Research Questions:
Survey CLI tools for shared UI patterns (headers, status indicators, progress, tables)
Evaluate Rich vs alternatives (click-extra, questionary, etc.)
Design minimal API surface for common operations
Consider Typer + Rich integration patterns
Potential Scope:
Styled section headers with step progress
Standardized status indicators (✓/⚠/✗/○/•) with color vocabulary
Spinner wrappers for async operations
Summary table generators
Banner/header utilities
Affected Tools: tnh-setup, tnh-gen, ytt-fetch, audio-transcribe, nfmt, token-count, tnh-tree

🚧 Document Success Cases¶

Status: NOT STARTED
Goal: Document TNH Scholar's successful real-world applications
Cases: Deer Park Cooking Course (SRTs), 1950s JVB Translation (OCR), Dharma Talk Transcriptions, Sr. Dang Nhiem's talks
Tasks:
Create docs/case-studies/ directory structure
Document each case with context, tools, challenges, outcomes

🚧 Notebook System Overhaul¶

Status: NOT STARTED
Priority: HIGH
Goal: Transform notebooks from exploratory/testing to production-quality examples
Tasks:
Audit & categorize all notebooks
Polish core example notebooks
Convert testing notebooks to pytest
Archive legacy notebooks with context notes

Priority 3: Future Work & Advanced Features¶

Goal: Long-term sustainability, advanced features, and nice-to-have improvements. Address after bootstrap loop is working.

🚧 Refactor Monolithic Modules¶

Status: NOT STARTED
Targets:
https://github.com/aaronksolomon/tnh-scholar/blob/main/src/tnh_scholar/ai_text_processing/prompts.py (34KB)
- Break into: prompt model, repository manager, git helpers, lock helpers
- Add docstrings and tests for each unit
- Document front-matter schema
https://github.com/aaronksolomon/tnh-scholar/blob/main/src/tnh_scholar/journal_processing/journal_process.py (28KB)
- Identify focused units
- Extract reusable components

🚧 Complete Provider Abstraction¶

Status: NOT STARTED
Tasks:
Implement Anthropic adapter
Add provider-specific error handling
Test fallback/retry across providers
Provider capability discovery
Multi-provider cost optimization

🚧 Developer Experience Improvements¶

Status: PARTIAL (hooks and Makefile exist, automation pending)
Tasks:
Add pre-commit hooks (Ruff, notebook prep)
Create Makefile for common tasks (lint, test, docs, format, setup)
Add MyPy to pre-commit hooks
Add contribution templates (issue/PR templates)
CONTRIBUTING.md exists and documented
Release automation
Changelog automation

🚧 Historical ADR Status Audit¶

Status: NOT STARTED
Context: 25 ADRs marked with status: current from pre-markdown-standards migration
Tasks:
Review each ADR to determine actual status (implemented/superseded/rejected)
Update status field in YAML frontmatter
Cross-reference with newer ADRs for superseded decisions

🚧 Package API Definition¶

Status: Deferred during prototyping
Tasks:
Review and document all intended public exports
Implement __all__ in key __init__.py files
Verify exports match documentation

🚧 Repo Hygiene¶

Problem: Generated artifacts in repo (build/, dist/, site/, *.txt)
Tasks:
Add to .gitignore
Document regeneration process
Rely on release pipelines for builds

🚧 Notebook & Research Management¶

Location: notebooks/, docs/research/
Problem: Valuable but not curated exploratory work
Tasks:
Adopt naming/linting convention
Publish vetted analyses to docs/research via nbconvert
Archive obsolete notebooks

Recently Completed Tasks (Archive)¶

tnh-gen CLI Implementation ✅¶

Completed: 2025-12-27
ADR: ADR-TG01, ADR-TG01.1
What: Protocol-driven CLI replacing tnh-fab, dual modes (human-friendly default, --api for machine consumption)
Documentation: tnh-gen CLI Reference (661 lines)

File-Based Registry System (ADR-A14) ✅¶

Completed: 2026-01-01 (PR #24)
ADR: ADR-A14, ADR-A14.1
What: JSONC-based registry with multi-tier pricing, TNHContext path resolution, staleness detection
Key Deliverables: openai.jsonc registry, RegistryLoader, Pydantic schemas, JSON Schema for VS Code, refactored model_router.py and safety_gate.py, 264 tests passing

VS Code Extension Walking Skeleton ✅¶

Completed: 2026-01-07
ADR: ADR-VSC01, ADR-VSC02
What: TypeScript extension enabling "Run Prompt on Active File" workflow
Capabilities: QuickPick prompt selector, dynamic variable input, tnh-gen run subprocess execution, split-pane output, unit/integration tests
Validation: Proves bootstrapping concept - extension ready to accelerate TNH Scholar development

Pattern→Prompt Migration ✅¶

Completed: 2026-01-19
ADR: ADR-PT04
What: Pattern→Prompt terminology migration and directory restructuring
Key Changes: patterns/ → prompts/ (standalone tnh-prompts repo), TNH_PATTERN_DIR → TNH_PROMPT_DIR, removed legacy tnh-fab CLI
Breaking: TNH_PATTERN_DIR env var removed, tnh-fab CLI removed

Provenance Format Refactor ✅¶

Completed: 2026-01-19
ADR: ADR-TG01 Addendum 2025-12-28
What: Switched tnh-gen from HTML comments to YAML frontmatter for provenance metadata
Files Modified: provenance.py, test_tnh_gen.py, tnh-gen.md

OpenAI Client Unification ✅¶

Completed: 2025-12-10
ADR: ADR-A13
What: Migrated from legacy openai_interface/ to modern gen_ai_service/providers/ architecture (6 phases)

Core Stubs Implementation ✅¶

Completed: 2025-12-10
What: Implemented params_policy, model_router, safety_gate, completion_mapper with strong typing
Grade: A- (92/100) - Production ready with minor polish

Documentation Reorganization Phase 1 ✅¶

Completed: 2025-12-05
ADR: ADR-DD01, ADR-DD02
What: Absolute links, MkDocs strict mode, filesystem-driven nav, lychee link checking

Packaging & CI Infrastructure ✅¶

Completed: 2025-11-20
What: pytest in CI, runtime dependencies declared, pre-commit hooks, Makefile targets

Remove Library sys.exit() Calls ✅¶

Completed: 2025-11-15
What: Library code raises ConfigurationError instead of exiting process

Convert Documentation Links to Absolute Paths ✅¶

Completed: 2025-12-05 (PR #14)
What: Converted 964 links to absolute paths, enabled MkDocs strict link validation, integrated link verification

NumberedText Section Boundary Validation ✅¶

Completed: 2025-12-12
ADR: ADR-AT03.2 (status: accepted → should be implemented)
What: Implemented validate_section_boundaries() and get_coverage_report() methods for robust section management
Commits: cf99375 (docs), 798a552 (refactor unused methods)

TextObject Robustness Improvements ✅¶

Completed: 2025-12-14
ADR: ADR-AT03.3 (status: accepted → should be implemented)
What: Implemented merge_metadata() with MergeStrategy enum, validate_sections() with fail-fast, converted to Pydantic v2, added structured exception hierarchy
Commits: 096e528 (implementation), 03654fe (../../docstrings)