QuestFoundry: Say Goodbye To V3 Runtime!

Hey there, QuestFoundry enthusiasts! We've got some exciting news that's all about streamlining our platform and making your experience even better. You might have heard whispers about a shift from the older v3 runtime and MyST domain compilation pipeline to our newer, more efficient v4 JSON-based runtime. Well, it's time to make it official: we're embarking on a mission to completely remove v3 artifacts. This isn't just about tweaking things; it's a comprehensive cleanup designed to reduce confusion and lighten our maintenance load, allowing us to focus on delivering even more awesome features.

The Evolution to v4: A Cleaner, Faster Future

To truly understand the significance of this cleanup, let's quickly recap where we've been and where we're headed. For a while now, our QuestFoundry landscape has been a bit of a mix. We've had the v3 runtime, which relied on code generated from domain/*.md files via the MyST parser. Think of it as a build-your-own-adventure approach where steps were compiled before they could be used. On the other hand, the v4 runtime represents a significant leap forward. It's designed to load domain-v4/studio.json directly. This means no more compilation step, leading to faster load times and a simpler architecture overall. Currently, the command-line interface (CLI) for QuestFoundry still defaults to using the v3 runtime, --runtime v3. This is a legacy setting that we're eager to phase out as we move fully into the v4 era.

Understanding the Architectural Shift: V3 vs. V4 Pipelines

To visualize this transition, let's break down the architectural impact. The V3 pipeline was a multi-step process: the domain/*.md files were fed into a compiler, which then generated .py files. These generated files were then used by the orchestrator.py, and finally, interacted with the CLI. It looked something like this:

domain/*.md → compiler → generated/*.py → orchestrator.py → CLI

This involved several transformations and intermediate steps, each with its own potential for complications and maintenance overhead. Now, let's look at the V4 pipeline, which is our current and future focus:

domain-v4/studio.json → runtime/domain/loader → orchestrator_v4.py → CLI

As you can see, the v4 pipeline is much more direct. It loads the studio.json file from the domain-v4 directory straight into the runtime's domain loader. This simplified path bypasses the compiler and generated code entirely, making the entire process more efficient and easier to manage. This architectural simplification is at the core of our decision to fully remove the v3 runtime. By eliminating the old pipeline, we're not just tidying up; we're creating a more robust and maintainable foundation for QuestFoundry's future development. This means faster iterations, fewer bugs, and more resources dedicated to building the features you love.

The Grand Removal Checklist: A Phased Approach to V3 Elimination

We've mapped out a clear, phased approach to ensure this transition is as smooth as possible. Each phase focuses on systematically removing specific v3 artifacts. Our goal is to be thorough, leaving no trace of the old runtime behind.

Phase 1: Archiving Domain Source

Our first step is to gently archive the old src/questfoundry/domain/ directory. We'll be moving it to a new location, _archive/domain/. This ensures that the original domain source files are preserved for historical reference or if any unforeseen need arises, without cluttering our active development space. Crucially, we'll update any internal references that might still point to the old path to reflect this new location. This might seem like a small step, but it's vital for keeping our codebase clean and organized.

Phase 2: Removing Generated Code

Next, we tackle the src/questfoundry/generated/ directory. This directory contains code that was automatically compiled from the v3 domain sources. Since we're moving away from the v3 compilation pipeline, this generated code is no longer necessary. We will delete this directory entirely. Alongside this, we'll also remove related files like .claude/hooks/block-generated-edits.sh and .claude/rules/generated-code.md, which are specific to managing this generated code. Finally, we'll give CLAUDE.md a quick update to remove any lingering references to these generated artifacts, ensuring our documentation is up-to-date with the new structure.

Phase 3: Eradicating V3 Runtime Files

This phase involves removing the core files that constitute the v3 runtime. We'll be deleting src/questfoundry/runtime/orchestrator.py (a substantial 556 lines!), src/questfoundry/runtime/roles.py (which handled RoleAgentPool for RoleIR), and src/questfoundry/runtime/prompts.py (responsible for RoleIR prompting). If there are any remaining v3 imports in runtime/__init__.py, those will be removed as well. This is a significant step towards a v3-free runtime environment.

Phase 4: Deleting the Compiler (V3-Only Artifacts)

With the v3 runtime files gone, we can now address the compiler itself. We'll delete src/questfoundry/compiler/models/ir.py, which contained models like RoleIR, LoopIR, and ArtifactTypeIR. Following this, we'll perform a crucial evaluation: can the entire compiler/ directory be removed, or are there parts of it that are still used by other, non-v3 related functionalities? Our aim is to remove it entirely if possible. We'll also remove the qf compile command from the CLI, as it was exclusively tied to the v3 pipeline.

Phase 5: Updating the Command-Line Interface (CLI)

The CLI is our primary interface with QuestFoundry, so it needs a thorough update. We'll remove the v3-specific branch in cli/main.py (lines 482-495). The most impactful change here will be changing the default --runtime from "v3" to "v4". We are even considering removing the --runtime flag altogether if v4 is truly the only option moving forward, simplifying the command structure. Additionally, we'll update the qf doctor and qf roles commands, as they currently import ALL_ROLES from the now-deleted generated files. Finally, any other imports of these generated artifact models within the CLI will be purged.

Phase 6: Revisiting and Purging Tests

Our commitment to quality means rigorous testing. We'll begin by deleting the tests/integration/test_orchestrator_e2e.py file, which was a specific end-to-end test for the v3 runtime. Before proceeding with removal, we'll run all existing tests to establish a baseline. After each phase of removal, we'll re-run the test suite to catch any issues early. Once v3 is fully removed, we'll conduct a comprehensive test suite run and perform manual testing using qf ask --runtime v4 ... to ensure everything functions as expected with the new default.

Phase 7: Documentation Refresh

Finally, to ensure clarity for everyone, we'll update our documentation. This includes removing all v3 references from ARCHITECTURE.md, making any necessary adjustments in AGENTS.md, and updating any other README files that might still mention the v3 runtime or compilation pipeline. Clear and accurate documentation is key to a smooth user experience.

Files Marked for Removal: A Clear Overview

To give you a consolidated view, here’s a quick rundown of the primary files and directories slated for removal, along with their estimated line counts and purpose:

Dependencies to Scrutinize

Before we pull the plug on these v3 components, we'll perform a careful check to ensure no other parts of our system still depend on them. Specifically, we'll be looking for any references to:

• from questfoundry.generated.roles import ALL_ROLES
• from questfoundry.generated.roles import DOMAIN_VERSION
• from questfoundry.compiler.models import RoleIR
• from questfoundry.runtime.orchestrator import Orchestrator

Verifying these dependencies will be a critical step in ensuring a clean and complete removal.

Our Testing Plan: Ensuring a Smooth Transition

Our testing strategy is designed to provide confidence at every step of this removal process:

1. Baseline: We'll start by running the entire test suite to establish a clear baseline of expected behavior.
2. Phased Testing: After each phase of the removal process (as outlined above), we will re-run the relevant parts of the test suite to catch any regressions early.
3. Full Verification: Once all v3 components are removed, we will execute the full test suite one last time.
4. Manual Validation: Finally, we'll conduct manual tests, specifically using commands like qf ask --runtime v4 ..., to ensure the system behaves as expected in a real-world scenario.

Important Considerations and Notes

It's important to note a few key points regarding this transition. The _archive/ directory will serve as a historical repository for the v3 domain source, ensuring that valuable information isn't lost, but it won't be part of the active build process. The v4 runtime is production-ready and has undergone extensive testing, making it a reliable successor. Please be aware that this is a breaking change. There will be no backward compatibility with v3 checkpoints, so ensure you are using the v4 runtime for any new work or saved states.

This cleanup marks a significant step forward for QuestFoundry, enabling us to build a more focused, efficient, and powerful platform for all your creative endeavors. We appreciate your understanding and support as we make these important improvements!

For more insights into QuestFoundry's architecture and development, you can check out the official QuestFoundry Documentation and dive deeper into the evolving codebase on QuestFoundry GitHub.