Production Lines
WO-001 Dry Run Report — Production Line Validation
WO-001 Dry Run Report — Production Line Validation
Date: 2026-03-18 Product: Enterprise Architecture Agent for STM Reference implementation: C:/Projects/agent-ea/ Production line: digital-talent (Stages 4-7) Purpose: Stress-test the production line, not redesign agent-ea
Executive Summary
The production line stages are structurally sound — the right activities happen in the right order with the right owners. But when applied to a real product, significant operational gaps emerge: missing templates, undefined handoff formats, ambiguous instructions, and no automation for verification. The biggest theme: the stages describe what to do but not how to do it, leaving each factory worker to reinvent the wheel.
Verdict: The line can produce a digital talent, but it would take 2x the estimated time because Pablo/Quinn/Diego/Dana would constantly need to improvise. The fixes below would cut that overhead significantly.
Stage 4 (Build) — Gap/Lessons List
What Works
- Assembly.md phases A-H cover the right ground
- The phase ordering is correct (repo → config → skills → templates → refs → orchestration → docs → verify)
- Agent-ea passes most structural checks — it was well-built
Gaps Found
| # | Gap | Severity | Fix |
|---|---|---|---|
| B-01 | No CLAUDE.md template. Assembly says "write CLAUDE.md with required sections" but provides no starter template. Pablo must invent formatting each time. | HIGH | Create production-lines/digital-talent/templates/CLAUDE-template.md with all 9 required sections as fillable stubs |
| B-02 | No per-request folder structure guidance. Work order lists demandes-ae/ as a deliverable, but assembly.md never mentions creating request folder structures. |
HIGH | Add Phase A.2: "Create request folder structure per work order" with standard pattern |
| B-03 | No skill template file. Assembly says "each skill follows the standard skill template" but the template doesn't exist in the production line. | HIGH | Create production-lines/digital-talent/templates/skill-template.md with frontmatter + required sections |
| B-04 | No documentation templates. Phase G says "user guide, config guide, skill reference card" but provides no templates. Remy's collaboration model undefined. | HIGH | Create doc templates in production-lines/digital-talent/templates/ |
| B-05 | Quick-start cheat sheet missing from Phase G. Stage 7 expects it as a deliverable but Phase G doesn't mention it. | MEDIUM | Add to Phase G list |
| B-06 | Assembly checklist is too generic. Final checklist is just 8 checkboxes. Needs to be a fill-in form with file counts, skill counts, test results. | MEDIUM | Create detailed assembly checklist template |
| B-07 | No consistency check tooling. CLAUDE.md-to-commands/ sync is manual. Agent-ea has 5 undocumented skills + 2 phantom references. | MEDIUM | Add a verification script or at minimum a documented manual procedure |
| B-08 | content-in/ subdirectory guidance vague. Assembly says "adjust based on domain" but doesn't give examples. | LOW | Add a table of common domain → subdirectory mappings |
| B-09 | Reference material format requirements missing. Agent-ea has .jpg, .doc, .pdf mixed with .md. No guidance on what formats agents can actually read. | LOW | Add note: prefer markdown, document fallback for binary files |
| B-10 | Scope creep handling undefined. Agent-ea grew from 15 → 20 skills during development. Assembly doesn't address how to handle extra skills that emerge. | LOW | Add note in Phase C: document extras in assembly checklist, update WO |
| B-11 | settings.json vs settings.local.json ambiguity. Assembly says one, agent-ea uses the other. | LOW | Clarify which file and why |
| B-12 | No guidance on .claude/skills/ directory. Agent-ea has dev-only skills there (skill-creator, improve-agent) that are separate from product skills. | LOW | Address in Phase A or Phase H (strip dev skills before handoff) |
Agent-ea Quick Fixes (Side Effects)
| # | Fix | Effort |
|---|---|---|
| A-01 | Remove /ea-note and /ea-validate phantom references from CLAUDE.md workflow text (they're aliases that don't resolve) |
5 min |
| A-02 | Document the 5 extra skills in CLAUDE.md skills table (ae-creer-diagramme-hautniveau, bmad-add-creer-note, bmad-add-revoir-note, do-work, ea-archi-extraction) OR remove them if they're dev artifacts | 15 min |
| A-03 | Remove bmad-v6 reference from CLAUDE.md line 4-5 ("Source de verite: bmad-v6") — factory-internal path that shouldn't be in client product | 5 min |
| A-04 | Strip "WORKSPACE GELE" freeze notice from CLAUDE.md — this is a dev state, not a product feature | 5 min |
Stage 5 (QA) — Gap/Lessons List
What Works
- Test categories are correct (structural, functional, edge case, documentation)
- Scoring formula is clear and the thresholds are reasonable
- WO Section 8 provides excellent product-specific test cases
Gaps Found
| # | Gap | Severity | Fix |
|---|---|---|---|
| Q-01 | No test execution protocol. Does Quinn run each skill manually in a fresh Claude session? With which model? In what order? No guidance on test environment setup. | HIGH | Add "Test Environment Setup" section: fresh session, specified model, clean test folder |
| Q-02 | No test data specification. Stage says Quinn gets test data from Pablo but doesn't define what constitutes adequate test data. F7 (repository validation) needs synthetic data with known planted gaps. | HIGH | Define test data requirements per test type. Add to Pablo's Stage 4 handoff checklist. |
| Q-03 | No QA certification template. Stage says "QA certification" output but provides no template for the document. | HIGH | Create production-lines/digital-talent/templates/qa-certification-template.md |
| Q-04 | Isolation vs integration testing undefined. F4 (solution design) and F8 (orchestrator) require multi-step pipelines as prerequisites. Stage doesn't say whether Quinn tests skills in isolation or runs the full pipeline. | MEDIUM | Add guidance: test skills in isolation first (unit), then test pipeline (integration) |
| Q-05 | Edge case inputs not operationalized. "Empty input" for a slash command is ambiguous — empty file? Missing file? Empty argument? | MEDIUM | Define concrete edge case scenarios per input type (file-based, argument-based, multi-file) |
| Q-06 | WO-specific vs stage-generic threshold ambiguity. WO says 8/8 + 5/6, stage says 100% + 80%. They're equivalent here but could diverge. | LOW | Clarify: WO thresholds override stage defaults |
| Q-07 | Remediation loop undefined. Stage says FAIL → back to Pablo, but no guidance on how Pablo addresses issues, re-submits, or what a re-test looks like. | MEDIUM | Add remediation workflow: Pablo fixes → re-submits with delta → Quinn re-tests failed cases only |
Stage 6 (Deploy) — Gap/Lessons List
What Works
- Three deployment options cover realistic scenarios
- Rollback plan concept is right
- Verification checks are mostly right
Gaps Found
| # | Gap | Severity | Fix |
|---|---|---|---|
| D-01 | No packaging manifest. No list of what ships to client vs what's internal. Agent-ea has ~30 top-level items. Someone must decide: do TODO.md, install-agent.ps1, ARCHITECTURE-*.md, notes/, prompts/, emp/, Agent-ae/, .claude/skills/ ship? |
HIGH | Add Phase 6.0: "Create packaging manifest" — list include/exclude patterns. Or add to assembly.md as Phase H output. |
| D-02 | No sanitization step. CLAUDE.md contains factory-internal paths (bmad-v6), dev freeze notices, dev scripts. These must be stripped before client delivery. No stage mentions this. | HIGH | Add Step 6.1b: "Sanitize agent configuration — remove factory references, dev state, internal paths" |
| D-03 | /help verification check is wrong. /help is a Claude Code built-in command, not an agent feature. Testing it proves Claude Code works, not the agent. |
MEDIUM | Replace with: "Agent responds correctly when asked about its capabilities (checks CLAUDE.md role definition is active)" |
| D-04 | No deployment checklist template. Diego has nothing to fill in as evidence. | MEDIUM | Create production-lines/digital-talent/templates/deployment-manifest-template.md |
| D-05 | No guidance on .mcp.json. Agent-ea has one referencing ea-apps MCP server. Does this ship? Does client need it? |
MEDIUM | Add to verification checklist: "Review .mcp.json — verify all referenced servers are available in client environment or remove" |
| D-06 | No guidance on dev-only artifacts. .claude/skills/ contains dev tools (skill-creator, improve-agent, dev-html) that shouldn't ship. |
MEDIUM | Add to packaging manifest: exclude .claude/skills/ dev tools |
| D-07 | "Naming conventions active" check is vague. Naming is enforced by CLAUDE.md instructions, not code. What does "active" mean? | LOW | Rewrite: "Run a skill, verify output file follows naming convention from CLAUDE.md" |
Stage 7 (Deliver) — Gap/Lessons List
What Works
- Handover session structure is well-designed (6 agenda items with timing)
- Client operational capability confirmation is concrete (3+ skills, create request, understand halts)
- Initial feedback collection is a good practice
Gaps Found
| # | Gap | Severity | Fix |
|---|---|---|---|
| V-01 | No demo script per product type. Dana needs to know which 3 skills to demo. For EA: note-revue → intrant → diagramme. But the stage doesn't reference the WO for recommended demos. | MEDIUM | Add: "Consult work order for recommended demo scenarios. If none specified, demo: intake skill, core capability skill, output skill." |
| V-02 | WO-specific feedback questions not referenced. WO Section 12 has detailed EA-specific feedback questions, but Stage 7 only asks generic ones. | MEDIUM | Add: "Supplement generic feedback with product-specific questions from the work order (Section 12 or equivalent)." |
| V-03 | Quick-start cheat sheet expected but never built. Stage 7 lists it as a deliverable; assembly.md Phase G doesn't mention it. | MEDIUM | Add to assembly.md Phase G. Create template. |
| V-04 | Delivery confirmation format undefined. "Signed by client" — email? Markdown? Confluence page? | LOW | Define standard format (suggest: markdown template with client signature line, stored in order records) |
| V-05 | Documentation delivery blocked. User guide, config guide, and skill reference card don't exist for agent-ea. Stage 7 can't deliver what Stage 4 didn't produce. | HIGH (but root cause is B-04) | Fix B-04 in Stage 4; this resolves downstream |
| V-06 | No handover notes template. Dana needs to capture what happened in the session but has no structure for it. | LOW | Create template with: attendees, demo results, questions asked, action items, client confidence score |
Cross-Stage Issues
| # | Issue | Stages Affected | Fix |
|---|---|---|---|
| X-01 | No artifact templates anywhere. The line defines activities and outputs but provides zero templates for any output document (assembly checklist, QA certification, deployment manifest, delivery confirmation, handover notes). Every factory worker improvises. | 4, 5, 6, 7 | Create a production-lines/digital-talent/templates/ directory with all templates |
| X-02 | Handoff protocol between stages is implicit. How does Pablo hand off to Quinn? Git tag? Slack message? Markdown file in the order directory? Each transition is undocumented. | 4→5, 5→6, 6→7 | Define standard handoff: create stage-N-complete.md in the order directory with checklist + links |
| X-03 | Work order is underutilized. Stages 5-7 should reference WO sections for product-specific details (test cases, feedback questions, demo scenarios) but they don't link back to the WO. | 5, 6, 7 | Add "Consult work order Section X for product-specific details" to each stage |
| X-04 | No order directory tracking. There's no convention for where stage artifacts (checklist, QA cert, deployment manifest, delivery confirmation) are stored. | 4-7 | Define: production-lines/orders/{order-id}/stage-artifacts/ |
Priority Summary
Must Fix Before Next Production Run (HIGH)
- B-01: CLAUDE.md template
- B-02: Per-request folder structure in assembly
- B-03: Skill template file
- B-04: Documentation templates (user guide, config guide, skill card, cheat sheet)
- Q-01: Test execution protocol
- Q-02: Test data specification
- Q-03: QA certification template
- D-01: Packaging manifest
- D-02: Sanitization step
- X-01: All artifact templates
Should Fix (MEDIUM)
- B-05, B-06, B-07, Q-04, Q-05, Q-07, D-03, D-04, D-05, D-06, V-01, V-02, V-03, X-02, X-03, X-04
Nice to Have (LOW)
- B-08, B-09, B-10, B-11, B-12, Q-06, D-07, V-04, V-06
Agent-ea Quick Fixes
These fell out naturally from the audit — low-effort fixes to the reference implementation:
| # | Fix | Effort | Priority |
|---|---|---|---|
| A-01 | Remove /ea-note and /ea-validate phantom references from CLAUDE.md |
5 min | Do now |
| A-02 | Document or remove 5 extra skills from .claude/commands/ | 15 min | Do now |
| A-03 | Remove bmad-v6 factory reference from CLAUDE.md | 5 min | Before deploy |
| A-04 | Strip "WORKSPACE GELE" dev freeze notice from CLAUDE.md | 5 min | Before deploy |
| A-05 | Create client documentation (user guide, config guide, skill card) | 2-3 hours | Before deliver |
| A-06 | Move dev-only skills from .claude/skills/ to a non-shipping location | 10 min | Before deploy |
| A-07 | Review .mcp.json — verify ea-apps server is needed for client | 10 min | Before deploy |
| A-08 | Clean up internal dev files (ARCHITECTURE-.md, STRUCTURE-ANALYSIS-.md, TODO.md, MEMORY-*.md, NEW-PROJECT-SETUP.md) | 15 min | Before deploy |
Conclusion
The production line's design is right. The operational tooling is what's missing. The highest-leverage fix is creating the templates directory (X-01) — this single action addresses B-01, B-03, B-04, B-06, Q-03, D-04, and V-06. Second highest leverage is the sanitization/packaging step (D-01 + D-02) which prevents shipping factory internals to clients.
The line is ready for a second dry run after templates are in place. The agent-ea quick fixes (A-01 through A-08) can be done in parallel.