PMFlow/docs/superpowers/plans/2026-04-13-chat-external-fact-qa-guardrails.md

# Chat External Fact QA Guardrails Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Stop chat-mode Feishu Q&A from using internal PMFlow evidence to answer external latest/fact questions, and make factual pushback stay in natural chat instead of falling into AI PMO correction prose.

**Architecture:** Keep requirement-plan and draft-confirm chains unchanged. Implement a narrow chat-only policy inside Adaptive RAG that detects external factual chat questions, isolates retrieval from internal project noise, downgrades safely when no live external evidence exists, and keeps follow-up fact challenges in the same chat scene.

**Tech Stack:** FastAPI backend, Adaptive RAG mixins, PostgreSQL-backed QA records, control-plane routing, pytest.

---

### Task 1: Add Chat External-Fact Policy Flags

**Files:**
- Modify: `backend/app/services/adaptive_rag_service/mixin_answer_mainline.py`
- Modify: `backend/app/services/adaptive_rag_service/mixin_followup_anchor.py`
- Test: `backend/tests/test_adaptive_rag_mainline_contract.py`

- [ ] **Step 1: Write failing tests for chat-only external fact follow-up**

Add tests that prove:
- an external-trend/latest question is marked as chat external fact policy
- a follow-up like “不对吧，已经有 xxx 了” anchored to a prior `external_trend` / `industry_trend` answer stays in chat QA and does not inherit AI PMO correction behavior

- [ ] **Step 2: Resolve anchored prior answer scene from QA record**

Read the anchored `QARecord` via existing follow-up resolution and derive the prior prompt variant from `policy_version` so chat logic can tell whether the previous reply was `external_trend` or `industry_trend`.

- [ ] **Step 3: Add a narrow chat external-fact policy helper**

Implement a helper in `mixin_answer_mainline.py` that derives flags such as:
- `external_fact_chat`
- `requires_freshness`
- `chat_fact_challenge`
- `allow_internal_refine`

These flags must be derived from current scene + question + anchored prior QA record, not from requirement-plan state.

- [ ] **Step 4: Thread the policy flags through answer runtime**

Pass the new flags into retrieval/execution flow without changing requirement-plan ingress or workflow planner behavior.

- [ ] **Step 5: Run focused tests**

Run: `pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "external or correction or followup" -v`


### Task 2: Isolate Retrieval for External Latest Questions

**Files:**
- Modify: `backend/app/services/adaptive_rag_service/mixin_multistep_flow.py`
- Modify: `backend/app/services/adaptive_rag_service/mixin_multistep_core.py`
- Modify: `backend/app/services/adaptive_rag_service/mixin_answer_retrieval.py`
- Test: `backend/tests/test_adaptive_rag_mainline_contract.py`

- [ ] **Step 1: Write failing tests for retrieval contamination**

Add regression tests that prove:
- `external_trend` / latest questions do not expand into `Project#104`, `#codex-plan*`, module paths, or feature catalog hints
- second-pass query refinement does not append internal entities/modules for this chat policy

- [ ] **Step 2: Disable internal second-pass hint expansion for external fact chat**

When `external_fact_chat=true`, skip internal refinement based on manifest, graph entities, project anchors, feature catalog, and symbol hints.

- [ ] **Step 3: Restrict evidentiary domains**

For external factual latest questions, internal `db / graph / pmflow / feishu / feature_catalog` sources must not be used as grounding for the external fact itself.

Allowed behavior:
- explicit external URL/doc evidence may ground the answer
- internal PMFlow context may only support “why this matters to your tenant/project”, never “what is the latest external fact”

- [ ] **Step 4: Preserve existing internal QA behavior**

Make sure non-external chat questions and PMFlow internal feature/status questions still use the current retrieval stack unchanged.

- [ ] **Step 5: Run focused tests**

Run: `pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "external_trend or industry_trend or retrieval" -v`


### Task 3: Add Freshness Guard and Natural Chat Correction Style

**Files:**
- Modify: `backend/app/services/adaptive_rag_service/mixin_deep_guard_rules.py`
- Modify: `backend/app/services/adaptive_rag_service/mixin_answer_retrieval.py`
- Modify: `backend/app/services/adaptive_rag_service/mixin_deep_guard_answer.py`
- Modify: `backend/app/config/ai_prompts_catalog/part_scenes_06.py`
- Test: `backend/tests/test_adaptive_rag_mainline_contract.py`
- Test: `backend/tests/test_draft_confirm_handler.py`

- [ ] **Step 1: Write failing tests for unsafe “grounded latest” replies**

Cover these cases:
- no external evidence + question asks “最新/目前/截至现在” => answer must explicitly say no real-time data
- grounding status must not be `grounded`
- reply must not inject project IDs, internal routes, or AI config guidance unless user explicitly asked for tenant impact

- [ ] **Step 2: Add external freshness guard**

Extend guard logic so external factual latest questions are blocked or downgraded when there is no explicit external evidence. Static internal sources must not satisfy freshness requirements.

- [ ] **Step 3: Rewrite answer fallback shape**

When no live external evidence exists, answer should look like normal chat:
- first sentence admits lack of real-time data / inability to confirm latest
- second sentence may give bounded generic context
- optional final sentence can ask whether to check tenant config separately

Forbidden output in this policy:
- “结论 / 证据链 / 待确认清单” tribunal style
- treating internal tags or knowledge-graph entities as external market evidence

- [ ] **Step 4: Strengthen prompt constraints as secondary guard**

Update `PROMPT_EXTERNAL_TREND_QA` / `PROMPT_INDUSTRY_TREND_QA` so they explicitly say:
- internal PMFlow graph/feature/catalog sources are not external search results
- if freshness is required and no external evidence is provided, do not claim exact latest version/model status

- [ ] **Step 5: Review deep-routing behavior**

Do not let this path silently force low-first deep reasoning for fact challenges. If needed, keep the safe downgrade path on low tier now; if quality still falls short, introduce a dedicated external-fact answer function in a separate patch rather than raising all chat traffic.

- [ ] **Step 6: Run focused tests**

Run:
- `pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "freshness or external or correction" -v`
- `pytest backend/tests/test_draft_confirm_handler.py -k "industry_trend or external_trend" -v`


### Task 4: Verify Runtime Safety and Sync Guardrail Docs

**Files:**
- Modify: `docs/architecture/RUNTIME_GUARDRAILS.md`
- Test: `backend/tests/test_adaptive_rag_mainline_contract.py`

- [ ] **Step 1: Add runtime guardrail note**

Document that chat-mode external factual latest questions cannot treat PMFlow internal graph/catalog/project evidence as external fact grounding, and anchored factual pushback must stay in chat unless explicitly tied to requirement/draft workflow.

- [ ] **Step 2: Run broader regression checks**

Run:
- `python -m py_compile backend/app`
- `pytest backend/tests/test_adaptive_rag_mainline_contract.py -v`

- [ ] **Step 3: Sanity-check that requirement-plan mainline is untouched**

Run a narrow requirement-plan regression slice:
- `pytest backend/tests/test_ai_pmo_ingress_mainline.py -k "requirement_plan or draft" -v`

- [ ] **Step 4: Commit**

```bash
git add backend/app/services/adaptive_rag_service backend/app/config/ai_prompts_catalog/part_scenes_06.py backend/tests/test_adaptive_rag_mainline_contract.py backend/tests/test_draft_confirm_handler.py docs/architecture/RUNTIME_GUARDRAILS.md docs/superpowers/plans/2026-04-13-chat-external-fact-qa-guardrails.md
git commit -m "fix: harden chat external fact QA guardrails"
```