linyuhang
/
PMFlow
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
PMFlow/docs/superpowers/plans/2026-04-13-chat-external-fa...

7.9 KiB
Raw Blame History

Chat External Fact QA Guardrails Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stop chat-mode Feishu Q&A from using internal PMFlow evidence to answer external latest/fact questions, and make factual pushback stay in natural chat instead of falling into AI PMO correction prose.

Architecture: Keep requirement-plan and draft-confirm chains unchanged. Implement a narrow chat-only policy inside Adaptive RAG that detects external factual chat questions, isolates retrieval from internal project noise, downgrades safely when no live external evidence exists, and keeps follow-up fact challenges in the same chat scene.

Tech Stack: FastAPI backend, Adaptive RAG mixins, PostgreSQL-backed QA records, control-plane routing, pytest.

Task 1: Add Chat External-Fact Policy Flags

Files:

  • Modify: backend/app/services/adaptive_rag_service/mixin_answer_mainline.py

  • Modify: backend/app/services/adaptive_rag_service/mixin_followup_anchor.py

  • Test: backend/tests/test_adaptive_rag_mainline_contract.py

  • Step 1: Write failing tests for chat-only external fact follow-up

Add tests that prove:

  • an external-trend/latest question is marked as chat external fact policy

  • a follow-up like “不对吧，已经有 xxx 了” anchored to a prior external_trend / industry_trend answer stays in chat QA and does not inherit AI PMO correction behavior

  • Step 2: Resolve anchored prior answer scene from QA record

Read the anchored QARecord via existing follow-up resolution and derive the prior prompt variant from policy_version so chat logic can tell whether the previous reply was external_trend or industry_trend.

  • Step 3: Add a narrow chat external-fact policy helper

Implement a helper in mixin_answer_mainline.py that derives flags such as:

  • external_fact_chat
  • requires_freshness
  • chat_fact_challenge
  • allow_internal_refine

These flags must be derived from current scene + question + anchored prior QA record, not from requirement-plan state.

  • Step 4: Thread the policy flags through answer runtime

Pass the new flags into retrieval/execution flow without changing requirement-plan ingress or workflow planner behavior.

  • Step 5: Run focused tests

Run: pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "external or correction or followup" -v

Task 2: Isolate Retrieval for External Latest Questions

Files:

  • Modify: backend/app/services/adaptive_rag_service/mixin_multistep_flow.py

  • Modify: backend/app/services/adaptive_rag_service/mixin_multistep_core.py

  • Modify: backend/app/services/adaptive_rag_service/mixin_answer_retrieval.py

  • Test: backend/tests/test_adaptive_rag_mainline_contract.py

  • Step 1: Write failing tests for retrieval contamination

Add regression tests that prove:

  • external_trend / latest questions do not expand into Project#104, #codex-plan*, module paths, or feature catalog hints

  • second-pass query refinement does not append internal entities/modules for this chat policy

  • Step 2: Disable internal second-pass hint expansion for external fact chat

When external_fact_chat=true, skip internal refinement based on manifest, graph entities, project anchors, feature catalog, and symbol hints.

  • Step 3: Restrict evidentiary domains

For external factual latest questions, internal db / graph / pmflow / feishu / feature_catalog sources must not be used as grounding for the external fact itself.

Allowed behavior:

  • explicit external URL/doc evidence may ground the answer

  • internal PMFlow context may only support “why this matters to your tenant/project”, never “what is the latest external fact”

  • Step 4: Preserve existing internal QA behavior

Make sure non-external chat questions and PMFlow internal feature/status questions still use the current retrieval stack unchanged.

  • Step 5: Run focused tests

Run: pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "external_trend or industry_trend or retrieval" -v

Task 3: Add Freshness Guard and Natural Chat Correction Style

Files:

  • Modify: backend/app/services/adaptive_rag_service/mixin_deep_guard_rules.py

  • Modify: backend/app/services/adaptive_rag_service/mixin_answer_retrieval.py

  • Modify: backend/app/services/adaptive_rag_service/mixin_deep_guard_answer.py

  • Modify: backend/app/config/ai_prompts_catalog/part_scenes_06.py

  • Test: backend/tests/test_adaptive_rag_mainline_contract.py

  • Test: backend/tests/test_draft_confirm_handler.py

  • Step 1: Write failing tests for unsafe “grounded latest” replies

Cover these cases:

  • no external evidence + question asks “最新/目前/截至现在” => answer must explicitly say no real-time data

  • grounding status must not be grounded

  • reply must not inject project IDs, internal routes, or AI config guidance unless user explicitly asked for tenant impact

  • Step 2: Add external freshness guard

Extend guard logic so external factual latest questions are blocked or downgraded when there is no explicit external evidence. Static internal sources must not satisfy freshness requirements.

  • Step 3: Rewrite answer fallback shape

When no live external evidence exists, answer should look like normal chat:

  • first sentence admits lack of real-time data / inability to confirm latest
  • second sentence may give bounded generic context
  • optional final sentence can ask whether to check tenant config separately

Forbidden output in this policy:

  • “结论 / 证据链 / 待确认清单” tribunal style

  • treating internal tags or knowledge-graph entities as external market evidence

  • Step 4: Strengthen prompt constraints as secondary guard

Update PROMPT_EXTERNAL_TREND_QA / PROMPT_INDUSTRY_TREND_QA so they explicitly say:

  • internal PMFlow graph/feature/catalog sources are not external search results

  • if freshness is required and no external evidence is provided, do not claim exact latest version/model status

  • Step 5: Review deep-routing behavior

Do not let this path silently force low-first deep reasoning for fact challenges. If needed, keep the safe downgrade path on low tier now; if quality still falls short, introduce a dedicated external-fact answer function in a separate patch rather than raising all chat traffic.

  • Step 6: Run focused tests

Run:

  • pytest backend/tests/test_adaptive_rag_mainline_contract.py -k "freshness or external or correction" -v
  • pytest backend/tests/test_draft_confirm_handler.py -k "industry_trend or external_trend" -v

Task 4: Verify Runtime Safety and Sync Guardrail Docs

Files:

  • Modify: docs/architecture/RUNTIME_GUARDRAILS.md

  • Test: backend/tests/test_adaptive_rag_mainline_contract.py

  • Step 1: Add runtime guardrail note

Document that chat-mode external factual latest questions cannot treat PMFlow internal graph/catalog/project evidence as external fact grounding, and anchored factual pushback must stay in chat unless explicitly tied to requirement/draft workflow.

  • Step 2: Run broader regression checks

Run:

  • python -m py_compile backend/app

  • pytest backend/tests/test_adaptive_rag_mainline_contract.py -v

  • Step 3: Sanity-check that requirement-plan mainline is untouched

Run a narrow requirement-plan regression slice:

  • pytest backend/tests/test_ai_pmo_ingress_mainline.py -k "requirement_plan or draft" -v

  • Step 4: Commit

git add backend/app/services/adaptive_rag_service backend/app/config/ai_prompts_catalog/part_scenes_06.py backend/tests/test_adaptive_rag_mainline_contract.py backend/tests/test_draft_confirm_handler.py docs/architecture/RUNTIME_GUARDRAILS.md docs/superpowers/plans/2026-04-13-chat-external-fact-qa-guardrails.md
git commit -m "fix: harden chat external fact QA guardrails"