linyuhang
/
PMFlow
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: add external query and semantic facts runtime

This commit is contained in:
Yuuuu 2026-04-16 11:17:52 +08:00
parent e42652b619
commit 4720c8117d
131 changed files with 17773 additions and 826 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.superpowers/brainstorm/95325-1776240518/.events Normal file
View File

@ -0,0 +1,2 @@
{"type":"click","text":"C. Tab 分对象类型\n \n 概览 | 项目 | 任务 | 信号 | 文档 | 人员 | 宪章 | 发布 | 复盘\n \n 当前 tab facts 数\n 失败数\n 最后抽取\n \n 当前对象类型的列表和详情\n \n \n \n \n 方案 C\n 对象边界最清楚，但你要横向看整体积压和失败会来回切 tab，运营效率最低。","choice":"c","id":null,"timestamp":1776240598958}
{"type":"click","text":"C. Tab 分对象类型\n \n 概览 | 项目 | 任务 | 信号 | 文档 | 人员 | 宪章 | 发布 | 复盘\n \n 当前 tab facts 数\n 失败数\n 最后抽取\n \n 当前对象类型的列表和详情\n \n \n \n \n 方案 C\n 对象边界最清楚，但你要横向看整体积压和失败会来回切 tab，运营效率最低。","choice":"c","id":null,"timestamp":1776241280184}

1
.superpowers/brainstorm/95325-1776240518/.server-stopped Normal file
View File

@ -0,0 +1 @@
{"reason":"idle timeout","timestamp":1776243098908}

1
.superpowers/brainstorm/95325-1776240518/.server.pid Normal file
View File

@ -0,0 +1 @@
95325

96
.superpowers/brainstorm/95325-1776240518/semantic-facts-layout.html Normal file
View File

@ -0,0 +1,96 @@
<h2>Semantic Facts 状态页设计稿</h2>
<p class="subtitle">先确认页面信息架构和操作区布局。后端统一按“提交后自动入队 + 失活旧 facts + 手动重抽入口”设计。</p>
<div class="cards">
<div class="card" data-choice="a" onclick="toggleSelect(this)">
<div class="card-image">
<div class="mockup">
<div class="mockup-header">A. 总览台 + 队列栏 + 对象表 + 详情抽屉</div>
<div class="mockup-body">
<div style="display:grid;grid-template-columns:repeat(4,1fr);gap:10px;margin-bottom:10px;">
<div class="placeholder" style="height:68px;">对象总数</div>
<div class="placeholder" style="height:68px;">active facts</div>
<div class="placeholder" style="height:68px;">失败队列</div>
<div class="placeholder" style="height:68px;">积压时长</div>
</div>
<div style="display:grid;grid-template-columns:1.2fr 0.8fr;gap:10px;margin-bottom:10px;">
<div class="placeholder" style="height:92px;">按对象类型统计：项目 / 任务 / 信号 / 文档 / 人员 / 宪章 / 发布 / 复盘</div>
<div class="placeholder" style="height:92px;">抽取队列状态：pending / running / retry / dead_letter</div>
</div>
<div style="display:grid;grid-template-columns:1fr 0.55fr;gap:10px;">
<div class="placeholder" style="height:170px;">可搜索对象表：对象类型 / 标题 / facts 条数 / 最后抽取 / 队列状态 / 操作</div>
<div class="placeholder" style="height:170px;">右侧详情抽屉：facts 列表 / 证据 / 最近 run / 重抽按钮</div>
</div>
</div>
</div>
</div>
<div class="card-body">
<h3>方案 A</h3>
<p>一页看全：先看风险和积压，再搜对象，再在右侧展开 facts 详情。我推荐这个，最适合排查“哪个对象没抽上、为什么失败、要不要重跑”。</p>
</div>
</div>
<div class="card" data-choice="b" onclick="toggleSelect(this)">
<div class="card-image">
<div class="mockup">
<div class="mockup-header">B. 双栏工作台</div>
<div class="mockup-body">
<div style="display:grid;grid-template-columns:0.9fr 1.1fr;gap:10px;">
<div>
<div class="placeholder" style="height:78px;margin-bottom:10px;">队列状态总览</div>
<div class="placeholder" style="height:194px;">对象列表 + 类型筛选 + 搜索</div>
</div>
<div>
<div class="placeholder" style="height:88px;margin-bottom:10px;">当前对象摘要：facts 数 / 最后抽取 / 最近失败</div>
<div class="placeholder" style="height:184px;">facts 明细、run 时间线、手动重抽</div>
</div>
</div>
</div>
</div>
</div>
<div class="card-body">
<h3>方案 B</h3>
<p>偏“工作台”操作感，适合长时间盯一个对象排障；但全局统计和横向比较会弱一点。</p>
</div>
</div>
<div class="card" data-choice="c" onclick="toggleSelect(this)">
<div class="card-image">
<div class="mockup">
<div class="mockup-header">C. Tab 分对象类型</div>
<div class="mockup-body">
<div class="mock-nav" style="margin-bottom:10px;">概览 | 项目 | 任务 | 信号 | 文档 | 人员 | 宪章 | 发布 | 复盘</div>
<div style="display:grid;grid-template-columns:repeat(3,1fr);gap:10px;margin-bottom:10px;">
<div class="placeholder" style="height:76px;">当前 tab facts 数</div>
<div class="placeholder" style="height:76px;">失败数</div>
<div class="placeholder" style="height:76px;">最后抽取</div>
</div>
<div class="placeholder" style="height:190px;">当前对象类型的列表和详情</div>
</div>
</div>
</div>
<div class="card-body">
<h3>方案 C</h3>
<p>对象边界最清楚，但你要横向看整体积压和失败会来回切 tab，运营效率最低。</p>
</div>
</div>
</div>
<div class="pros-cons" style="margin-top:18px;">
<div class="pros">
<h4>我推荐 A 的原因</h4>
<ul>
<li>既能看全局积压，也能快速钻到单对象 facts。</li>
<li>和现有治理子页风格更一致，适合放到智能配置中心。</li>
<li>后面要加 bulk backfill、失败重放、对象覆盖率都比较自然。</li>
</ul>
</div>
<div class="cons">
<h4>设计约束</h4>
<ul>
<li>这轮“需求”按宪章（charter）展示，不含运行中 draft。</li>
<li>队列状态会直接暴露 pending / retry / dead_letter，不做粉饰。</li>
<li>对象详情默认展示 active facts，历史失活 facts 放二级查看。</li>
</ul>
</div>
</div>

10
AGENTS.md
View File

@ -95,14 +95,24 @@ frontend/ # React 前端
- 新增接口必须遵循响应包络：`{code, message, data}`，不得裸返回
- 禁止在 API 层写业务逻辑，业务逻辑归属 `services/`
- 禁止在前端组件内直接调用 `fetch`，必须通过 `frontend/src/api/` 封装层
- MCP 模板接入中心只允许保存模板参数、命令预览、runtime probe 与“本机已执行”确认状态；远程 `streamable_http` 模板允许由 PMFlow 后端直连校验/调用，本地 `stdio` 模板只能走 PMFlow Local Bridge 或兼容 MCP host；禁止从服务端直接写宿主机 `~/.codex/config.toml`、禁止替客户执行本机命令，GitHub 类模板只允许保存 env var 名称或占位符，不得落明文 token。
- 前端状态管理：服务端数据用 React Query，本地 UI 状态用 `useState`
- 用户可见核心业务内容零兜底：AI 失败、证据不足、结构化失败时必须显式失败或标记待确认，禁止用默认句、模板句、stub 冒充成功。
- 业务语义、问答/执行归因与前台动作拍板必须优先走“对象/上下文绑定 -> `SemanticContextPacket` -> `SemanticDecision` -> gatekeeper”主链，禁止在 `backend/app` 运行时代码里恢复 `looks_like_*`、关键词仓或散落正则拍板。
- `query.answer` 路由与外部查询规划不得在运行时代码里维护 `_INTERNAL_ROUTE_OBJECTS / _EXTERNAL_ROUTE_OBJECTS / _SEMANTIC_QUERY_OBJECT_TYPES` 一类对象词表，也不得新增 `*_probe_is_strong / *_route_probe / query_kind(weather/news/company/market/model)` 这类二次语义拍板。`query_profile / source_policy_hint / SemanticDecision` 只能作为上游结构化输入；若证据不足，只能显式返回 `uncertain/clarify`，禁止在执行层靠对象集合、`subject_focus`、`needs_freshness`、`referenced_objects` 再次裁决 internal vs external。
- `llm_service` 运行时必须显式传入 `routing_meta`；禁止再通过函数名片段、hint 常量或模板词在底层推断 tier、结构化输出或 fallback 策略。
- 显式 `reply/thread`、卡片动作和 `work_item_id` 属于更强锚点，必须优先于旧 session 和记忆续写。
- 进入 `requirement_plan` / draft confirm 的线程必须留在对应主链，禁止掉回 `query.answer`；其中 draft confirm 只允许显式 `reply/thread`、卡片按钮或 `work_item_id` 进入，禁止无锚点按“同群/同人”回捞旧 pending draft。若同 scope 旧 plan/draft 已被判定污染，运行时必须先终止该 scope 内的 active topic/work item，再回退到 `query.answer`。选项协议回复必须复用单点结构化解析器，不得散落关键词/正则。
- 已绑定 active requirement context 的线程里，显式 `退出需求模式` 必须先挂起当前 collecting/ready draft 对应的 requirement topic，再允许回到普通 `query.answer`；同一 `conversation_scope` 下的 topic 选择必须优先命中仍带 active draft 的 live requirement workspace，禁止被后续 idle query topic 抢走。
- draft confirm 若命中显式 `reply/thread` 锚点，候选 mounted draft / pending draft 必须先通过该锚点精确命中；同群、同 scope、同人但锚点不匹配的历史草稿不得参与打分或回捞，避免 requirement-plan A/B/C 线程被旧 draft 劫持。
- Feishu `requirement_plan` 首轮/追问回复若命中 `reply_template_key=requirement_plan_followup`，必须保留模板键并优先渲染交互卡片；卡片按钮回流必须复用 canonical Feishu `reply/thread` 上下文，禁止因 query renderer 吞模板键或按钮回流丢锚点而退回纯文本旧模板。
- 冷启动 `WorkflowSemanticPlanner` / `pmflow_semantic_context_decision` 失败时，运行时只能降级到带 `semantic_planner_degraded=true` 审计标记的中性 `query.answer` 场景；后续优先重试 semantic backfill，显式 active requirement context / reply anchor 仍留在 requirement-plan 主链，无锚点事实问句禁止硬降级成 requirement consultation。
- Feishu 显式 `reply/thread` 到机器人消息时，必须先通过 `feishu_reply_outbox.sent_message_id -> source_message_id` 归一化 canonical thread context，再进入后续主链。
- 外部 freshness 问答的 follow-up 若锚定上一轮“未接入实时外部数据/无法确认最新结论”的 QA 回复，默认继续留在 `query.answer` 聊天链并先尝试实时外部检索；只有用户显式点明系统/项目/配置等内部对象并提出接入动作时，才允许落入内部接入/入库流程。
- non-plan `query.answer` 必须先经过 `Query Route Gateway` 产出 `route_mode / decision_origin / reply_contract / probe_snapshot`，再进入 AdaptiveRAG；回复正文必须遵守显式来源合同（`内部知识库查询：` / `外部实时查询：`），`uncertain` 只允许 `external-first` 或澄清，禁止默认混合答案。
- `route_mode=external` 后不得再直接把 DuckDuckGo HTML 抓取当主执行层；必须统一进入 `External Query Planner -> External Query Gateway -> External Provider Registry -> MCP adapter 优先 -> bounded fallback live search -> external answer verifier` 主链。external 来源过滤必须 fail-closed，禁止把 `Feishu history / doc / pmflow` 这类内部来源混回 `外部实时查询` 的来源摘要。
- `query.answer` 的客户可见回复必须再经过 `Answer Presentation Policy`：客户视图只允许看到中文化 `display_label/display_summary`，禁止直接暴露 `raw_source_label`、`Project database / Semantic facts / Knowledge Graph / Feishu history` 等机器来源或内部术语。prompt context 也只能喂展示字段，不能再把原始英文来源标签直接喂给模型。
- internal `query.answer` 一旦命中 `route_mode=internal`，必须再走“`Operation Contract -> Target Resolver -> Query DSL Plan -> 白名单受控执行器 -> semantic facts 优先（可用时）-> bounded fallback retrieval -> answer verifier`”链路；禁止让 LLM 自由写 SQL、自由决定 join/表名，或在没有证据时直接给内部结论。
- `MountedDraftWorkspace.requirement_card` 是 requirement-plan 唯一业务真源；assistant prose、topic facts、run 文案只可审计，不可反推业务字段。
- `project_create.create / followup.confirm` 的任务拆解是从属子步骤：运行时必须先 low-first 结构化生成，必要时升级 high；仍失败则显式记为 `pending`，禁止冒充成功。
- 运行时长清单、Feishu / requirement-plan / AdaptiveRAG 细则、已知边界与假设，统一以 `docs/architecture/RUNTIME_GUARDRAILS.md` 为准。

18
CLAUDE.md
View File

@ -32,7 +32,7 @@ AIProjectManger/
```
- 详细架构库存已迁到 [`docs/architecture/ARCHITECTURE_INVENTORY.md`](/Users/yuuuu/Documents/AIProjectManger/docs/architecture/ARCHITECTURE_INVENTORY.md)。
- `docs/architecture/` 当前包含：`MASTER_CONTRACT.md`、`RUNTIME_GUARDRAILS.md`、`CODE_REFERENCE_INDEX.md`、`API_CONTRACTS.md`、`DATAFLOW.md`、`TASK_BRIEF_TEMPLATE.md`、`ARCHITECTURE_INVENTORY.md`
- `docs/superpowers/plans/` 保存实现计划，`docs/task_briefs/` 保存任务包产物与验收截图。
- `docs/superpowers/plans/` 保存实现计划，`docs/superpowers/specs/` 保存需求/方案规格，`docs/task_briefs/` 保存任务包产物与验收截图。
## 4. 后端结构（`backend/`）
```text
@ -56,10 +56,18 @@ backend/
- 当前后端重点入口优先看：
- `backend/app/api/` 下的 `*_api/` 子包与聚合 router
- `backend/app/services/ai_pmo/`（AI PMO 主链、Mounted Runtime、semantic context）
- `backend/app/services/adaptive_rag_service/`（只读问答主链）
- `backend/app/services/adaptive_rag_service/`（只读问答主链；`mixin_query_route_gateway.py` 负责 non-plan `query.answer` 的 internal/external/uncertain 来源合同归一化、contract 审计与显式来源回复合同；`mixin_answer_retrieval.py` 现已把 `route_mode=external` 接到独立外部查询主链；内部查询基线已升级为 `Operation Contract -> Target Resolver -> Query DSL Plan -> 受控执行器 -> semantic facts 优先 -> verifier`）
- `backend/app/services/external_query_service/`（external `query.answer` 主链：`External Query Planner -> External Query Gateway -> External Provider Registry -> MCP 优先 -> bounded fallback live search -> verifier`）
- `backend/app/services/mcp_config/`（MCP 模板接入中心：catalog、bundle 持久化、后端直连 runtime probe / Microsoft Learn 检索桥接、Local Bridge 接入说明与本机确认状态）
- `backend/app/services/semantic_fact_service/`（semantic facts 读模型、八类业务对象统一抽取/失活、ORM 提交钩子入队、治理看板查询）
- `backend/app/api/semantic_facts_api/`（semantic facts 治理后台接口：overview / objects / detail / reextract / backfill）
- `backend/app/api/mcp_api/`（MCP 模板接入中心后台接口：catalog / servers / validate / confirm-client-apply）
- `backend/app/services/llm_service/`（显式 `routing_meta` 协议）
- `backend/app/services/integrations/feishu_integration_service/`（飞书入站与卡片回调）
- `backend/app/services/signal/`（信号接入、编排、planner、repair）
- 本轮新增的问答路由审计存储为 `backend/app/models/query_route_memory.py`，通过 `backend/alembic/versions/20260415_add_query_route_memories.py` 建表；只用于 route audit / feedback 回灌 / 离线评估，不做在线自动调参。
- 本轮新增的语义事实读模型为 `backend/app/models/semantic_fact.py` 与 `backend/app/models/semantic_fact_extraction_run.py`，通过 `backend/alembic/versions/20260415_add_semantic_facts.py` 建表；用于内部查询的 facts-first 命中与异步抽取审计。
- 本轮新增的外部查询 fallback 回归测试位于 `backend/tests/test_external_query_planner.py`；配合 `backend/tests/test_adaptive_rag_mainline_contract.py`、`backend/tests/test_ai_pmo_ingress_mainline.py` 与 `backend/tests/test_semantic_lint.py` 一起锁定“Query Route Gateway 不再跑 runtime probes、External Query Planner fallback 不再按对象重分类、external freshness follow-up 不再靠硬编码词表”。
- 详细目录迁移、兼容壳清理与新增模块清单见库存文档。
## 5. 前端结构（`frontend/`）
@ -83,6 +91,8 @@ frontend/
- `frontend/src/api/`（统一 API 封装）
- `frontend/src/pages/inbox/`（消息台）
- `frontend/src/pages/ai-steward/`（AI 管家）
- `frontend/src/pages/ai-config/mcp-config/`（AI Config -> Governance Tools 下的 MCP 模板接入中心）
- `frontend/src/pages/semantic-facts-board/`（AI Config -> Governance Tools 下的 semantic facts 治理页）
- `frontend/src/pages/scheduler-center/`（Mounted AI Console）
- `frontend/src/components/sstc/`（SSTC 工作流组件）
- 页面目录化、Mounted AI Console、Inbox、AI Steward 与 Recipes/Projects/Users 等模块化拆分的详细库存见库存文档。
@ -143,6 +153,7 @@ frontend/
cd frontend
npm run dev
npm run build
npm test
npm run e2e:feishu
```
@ -165,9 +176,12 @@ uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
- 当前最重要的摘要只保留：
- 写操作必须经 Alembic / 受控执行器 / outbox / 审计日志，禁止业务路径直写或同步直发。
- 用户可见核心业务内容禁止 fallback、stub、模板句冒充成功；失败必须显式失败或标记待确认。
- MCP 模板接入中心只保存模板参数、命令预览、runtime probe 和手动确认状态；远程 `streamable_http` 模板可由 PMFlow 后端直连，`stdio` 模板只能走 Local Bridge / 兼容 MCP host；禁止从服务端直写宿主机 `~/.codex/config.toml` 或替客户执行本机命令。
- 运行时业务动作归因统一走“对象/上下文绑定 -> `SemanticContextPacket` -> `SemanticDecision` -> gatekeeper”。
- `backend/app` 运行时代码禁止恢复散落的中文关键词、正则、`looks_like_*`、`*_HINTS/*_KEYWORDS` 业务拍板逻辑。
- `llm_service` 只认显式 `routing_meta` 协议，不再靠函数名或 hint 推断 tier / contract / fallback。
- 内部只读查询必须走 `Operation Contract -> Target Resolver -> Query DSL Plan -> 白名单受控执行器 -> semantic facts 优先 -> verifier`，禁止自由 SQL 和无证据脑补。
- `semantic_facts` 自动抽取统一走 ORM 提交钩子 -> `ingest_events(channel=semantic_fact_extract)` -> 统一 worker，支持 `project/task/signal/document/user/requirement/release/review` 八类对象；禁止在各路由散落手工 enqueue。
- AI Steward、Requirement Plan、Signal Source、AdaptiveRAG 等核心链路的详细 guardrails 统一以 `RUNTIME_GUARDRAILS.md` 为准。
## 9. 新增代码的参考模式

10
HEARTBEAT.md
View File

@ -3,3 +3,13 @@
# Keep this file empty (or with only comments) to skip heartbeat API calls.
# Add tasks below when you want the agent to check something periodically.
## 每日工单推送
- 触发时间：每天 08:30
- 任务：查询当天工单并推送到飞书
- 执行命令：
```bash
docker exec pmflow-db psql "postgresql://readonly:readonly%40%232025@173.1.1.32:5432/middle_platform" -c "SELECT task_code, task_name, excutor_name, status, start_time FROM plat_task_center.task_instance WHERE start_time >= CURRENT_DATE AND start_time < CURRENT_DATE + INTERVAL '1 day' AND ext3 IN ('90101', '901') ORDER BY start_time DESC LIMIT 50;" -x
```
- 目标用户：ou_9e1fefbcf4cc68a84f4a4ff7d1749048

98
backend/alembic/versions/20260415_add_query_route_memories.py Normal file
View File

@ -0,0 +1,98 @@
"""add query route memories
Revision ID: 20260415_add_query_route_memories
Revises: 20260413_rename_control_plane_status
Create Date: 2026-04-15
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
try:
from sqlalchemy.dialects.postgresql import JSONB
except Exception: # pragma: no cover
JSONB = sa.JSON
revision = "20260415_add_query_route_memories"
down_revision = "20260413_rename_control_plane_status"
branch_labels = None
depends_on = None
def _json_type(bind):
return JSONB() if bind.dialect.name == "postgresql" else sa.JSON()
def _table_exists(bind, table_name: str) -> bool:
return sa.inspect(bind).has_table(table_name)
def upgrade() -> None:
bind = op.get_bind()
if _table_exists(bind, "query_route_memories"):
return
op.create_table(
"query_route_memories",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("tenant_id", sa.BigInteger(), nullable=False),
sa.Column("created_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("qa_record_id", sa.Integer(), sa.ForeignKey("qa_records.id", ondelete="CASCADE"), nullable=False),
sa.Column("project_id", sa.BigInteger(), nullable=True),
sa.Column("question_text", sa.Text(), nullable=False),
sa.Column("route_mode_predicted", sa.String(length=32), nullable=False),
sa.Column("route_mode_effective", sa.String(length=32), nullable=False),
sa.Column("decision_origin", sa.String(length=64), nullable=True),
sa.Column("reply_contract", _json_type(bind), nullable=True),
sa.Column("probe_snapshot", _json_type(bind), nullable=True),
sa.Column("query_profile", _json_type(bind), nullable=True),
sa.Column("feedback_outcome", _json_type(bind), nullable=True),
sa.Column("question_embedding", _json_type(bind), nullable=True),
sa.UniqueConstraint(
"tenant_id",
"qa_record_id",
name="uq_query_route_memories_tenant_qa_record",
),
)
op.create_index("ix_query_route_memories_tenant_id", "query_route_memories", ["tenant_id"], unique=False)
op.create_index("ix_query_route_memories_qa_record_id", "query_route_memories", ["qa_record_id"], unique=False)
op.create_index("ix_query_route_memories_project_id", "query_route_memories", ["project_id"], unique=False)
op.create_index(
"ix_query_route_memories_route_mode_predicted",
"query_route_memories",
["route_mode_predicted"],
unique=False,
)
op.create_index(
"ix_query_route_memories_route_mode_effective",
"query_route_memories",
["route_mode_effective"],
unique=False,
)
op.create_index(
"ix_query_route_memories_decision_origin",
"query_route_memories",
["decision_origin"],
unique=False,
)
def downgrade() -> None:
bind = op.get_bind()
if not _table_exists(bind, "query_route_memories"):
return
for index_name in (
"ix_query_route_memories_decision_origin",
"ix_query_route_memories_route_mode_effective",
"ix_query_route_memories_route_mode_predicted",
"ix_query_route_memories_project_id",
"ix_query_route_memories_qa_record_id",
"ix_query_route_memories_tenant_id",
):
op.drop_index(index_name, table_name="query_route_memories")
op.drop_table("query_route_memories")

138
backend/alembic/versions/20260415_add_semantic_facts.py Normal file
View File

@ -0,0 +1,138 @@
"""add semantic facts
Revision ID: 20260415_add_semantic_facts
Revises: 20260415_add_query_route_memories
Create Date: 2026-04-15
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "20260415_add_semantic_facts"
down_revision = "20260415_add_query_route_memories"
branch_labels = None
depends_on = None
def _table_exists(bind, table_name: str) -> bool:
return sa.inspect(bind).has_table(table_name)
def upgrade() -> None:
bind = op.get_bind()
if not _table_exists(bind, "semantic_facts"):
op.create_table(
"semantic_facts",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("tenant_id", sa.BigInteger(), nullable=False),
sa.Column("created_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("entity_type", sa.String(length=32), nullable=False),
sa.Column("entity_id", sa.BigInteger(), nullable=False),
sa.Column("fact_namespace", sa.String(length=64), nullable=False),
sa.Column("fact_key", sa.String(length=64), nullable=False),
sa.Column("fact_value_text", sa.Text(), nullable=False),
sa.Column("fact_value_norm", sa.String(length=255), nullable=False),
sa.Column("value_type", sa.String(length=32), nullable=False),
sa.Column("confidence", sa.Float(), nullable=False, server_default="0"),
sa.Column("evidence_source_type", sa.String(length=32), nullable=True),
sa.Column("evidence_source_id", sa.BigInteger(), nullable=True),
sa.Column("evidence_excerpt", sa.Text(), nullable=True),
sa.Column("extract_version", sa.String(length=64), nullable=True),
sa.Column("freshness_at", sa.String(length=64), nullable=True),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
sa.UniqueConstraint(
"tenant_id",
"entity_type",
"entity_id",
"fact_key",
"fact_value_norm",
name="uq_semantic_facts_tenant_entity_key_norm",
),
)
op.create_index("ix_semantic_facts_tenant_id", "semantic_facts", ["tenant_id"], unique=False)
op.create_index("ix_semantic_facts_entity_type", "semantic_facts", ["entity_type"], unique=False)
op.create_index("ix_semantic_facts_entity_id", "semantic_facts", ["entity_id"], unique=False)
op.create_index("ix_semantic_facts_fact_namespace", "semantic_facts", ["fact_namespace"], unique=False)
op.create_index("ix_semantic_facts_fact_key", "semantic_facts", ["fact_key"], unique=False)
op.create_index("ix_semantic_facts_fact_value_norm", "semantic_facts", ["fact_value_norm"], unique=False)
op.create_index("ix_semantic_facts_is_active", "semantic_facts", ["is_active"], unique=False)
if not _table_exists(bind, "semantic_fact_extraction_runs"):
op.create_table(
"semantic_fact_extraction_runs",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
sa.Column("tenant_id", sa.BigInteger(), nullable=False),
sa.Column("created_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("updated_at", sa.TIMESTAMP(), server_default=sa.func.now(), nullable=False),
sa.Column("entity_type", sa.String(length=32), nullable=False),
sa.Column("entity_id", sa.BigInteger(), nullable=False),
sa.Column("trigger_type", sa.String(length=32), nullable=False),
sa.Column("status", sa.String(length=32), nullable=False),
sa.Column("input_fingerprint", sa.String(length=64), nullable=True),
sa.Column("facts_written", sa.Integer(), nullable=False, server_default="0"),
sa.Column("failure_reason", sa.Text(), nullable=True),
sa.Column("extract_version", sa.String(length=64), nullable=True),
sa.Column("started_at", sa.String(length=64), nullable=True),
sa.Column("finished_at", sa.String(length=64), nullable=True),
)
op.create_index(
"ix_semantic_fact_extraction_runs_tenant_id",
"semantic_fact_extraction_runs",
["tenant_id"],
unique=False,
)
op.create_index(
"ix_semantic_fact_extraction_runs_entity_type",
"semantic_fact_extraction_runs",
["entity_type"],
unique=False,
)
op.create_index(
"ix_semantic_fact_extraction_runs_entity_id",
"semantic_fact_extraction_runs",
["entity_id"],
unique=False,
)
op.create_index(
"ix_semantic_fact_extraction_runs_status",
"semantic_fact_extraction_runs",
["status"],
unique=False,
)
op.create_index(
"ix_semantic_fact_extraction_runs_input_fingerprint",
"semantic_fact_extraction_runs",
["input_fingerprint"],
unique=False,
)
def downgrade() -> None:
bind = op.get_bind()
if _table_exists(bind, "semantic_fact_extraction_runs"):
for index_name in (
"ix_semantic_fact_extraction_runs_input_fingerprint",
"ix_semantic_fact_extraction_runs_status",
"ix_semantic_fact_extraction_runs_entity_id",
"ix_semantic_fact_extraction_runs_entity_type",
"ix_semantic_fact_extraction_runs_tenant_id",
):
op.drop_index(index_name, table_name="semantic_fact_extraction_runs")
op.drop_table("semantic_fact_extraction_runs")
if _table_exists(bind, "semantic_facts"):
for index_name in (
"ix_semantic_facts_is_active",
"ix_semantic_facts_fact_value_norm",
"ix_semantic_facts_fact_key",
"ix_semantic_facts_fact_namespace",
"ix_semantic_facts_entity_id",
"ix_semantic_facts_entity_type",
"ix_semantic_facts_tenant_id",
):
op.drop_index(index_name, table_name="semantic_facts")
op.drop_table("semantic_facts")

4
backend/app/api/mcp_api/__init__.py Normal file
View File

@ -0,0 +1,4 @@
from . import routes
from .routes import router
__all__ = ["router", "routes"]

136
backend/app/api/mcp_api/routes.py Normal file
View File

@ -0,0 +1,136 @@
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Depends
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from ...core.database import get_db
from ...core.response import error_response, success_response
from ...dependencies import require_manager_user
from ...models import User
from ...services.mcp_config import McpConfigService
router = APIRouter(prefix="/mcp", tags=["mcp_config"])
class McpServerCreateRequest(BaseModel):
name: str
display_name: str | None = None
template_id: str
params: dict[str, Any] = Field(default_factory=dict)
class McpServerUpdateRequest(BaseModel):
name: str | None = None
display_name: str | None = None
template_id: str | None = None
params: dict[str, Any] | None = None
def _service(db: AsyncSession, current_user: User) -> McpConfigService:
return McpConfigService(db, tenant_id=int(current_user.tenant_id))
@router.get("/admin/catalog")
async def get_mcp_catalog(
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
return success_response(await _service(db, current_user).get_catalog())
@router.get("/admin/servers")
async def list_mcp_servers(
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
items = await _service(db, current_user).list_servers()
return success_response({"items": items})
@router.post("/admin/servers")
async def create_mcp_server(
body: McpServerCreateRequest,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = _service(db, current_user)
try:
item = await service.create_server(
name=body.name,
display_name=body.display_name,
template_id=body.template_id,
params=body.params,
)
except ValueError as exc:
await db.rollback()
return JSONResponse(status_code=400, content=error_response(400, str(exc)))
await db.commit()
return success_response(item)
@router.put("/admin/servers/{server_id}")
async def update_mcp_server(
server_id: str,
body: McpServerUpdateRequest,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = _service(db, current_user)
try:
item = await service.update_server(
server_id=server_id,
name=body.name,
display_name=body.display_name,
template_id=body.template_id,
params=body.params,
)
except ValueError as exc:
await db.rollback()
return JSONResponse(status_code=400, content=error_response(400, str(exc)))
except LookupError as exc:
await db.rollback()
return JSONResponse(status_code=404, content=error_response(404, str(exc)))
await db.commit()
return success_response(item)
@router.post("/admin/servers/{server_id}/validate")
async def validate_mcp_server(
server_id: str,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = _service(db, current_user)
try:
payload = await service.validate_server(server_id=server_id)
except ValueError as exc:
await db.rollback()
return JSONResponse(status_code=400, content=error_response(400, str(exc)))
except LookupError as exc:
await db.rollback()
return JSONResponse(status_code=404, content=error_response(404, str(exc)))
await db.commit()
return success_response(payload)
@router.post("/admin/servers/{server_id}/confirm-client-apply")
async def confirm_mcp_server_client_apply(
server_id: str,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = _service(db, current_user)
try:
payload = await service.confirm_client_apply(server_id=server_id)
except ValueError as exc:
await db.rollback()
return JSONResponse(status_code=400, content=error_response(400, str(exc)))
except LookupError as exc:
await db.rollback()
return JSONResponse(status_code=404, content=error_response(404, str(exc)))
await db.commit()
return success_response(payload)

4
backend/app/api/router.py
View File

@ -36,7 +36,9 @@ from .operation_catalog import router as operation_catalog_router
from .ai_pmo_api import router as ai_pmo_router
from .audit import router as audit_router
from .model_routing import router as model_routing_router
from .semantic_facts_api import router as semantic_facts_router
from .task_briefs import router as task_briefs_router
from .mcp_api import router as mcp_router
# 创建主 API 路由器
api_router = APIRouter()
@ -72,4 +74,6 @@ api_router.include_router(operation_catalog_router)
api_router.include_router(ai_pmo_router)
api_router.include_router(audit_router)
api_router.include_router(model_routing_router)
api_router.include_router(semantic_facts_router)
api_router.include_router(task_briefs_router)
api_router.include_router(mcp_router)

4
backend/app/api/semantic_facts_api/__init__.py Normal file
View File

@ -0,0 +1,4 @@
from . import routes
from .routes import router
__all__ = ["router", "routes"]

104
backend/app/api/semantic_facts_api/routes.py Normal file
View File

@ -0,0 +1,104 @@
from __future__ import annotations
from typing import Optional
from fastapi import APIRouter, Depends, Query
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from ...core.database import get_db
from ...core.response import error_response, success_response
from ...dependencies import require_manager_user
from ...models import User
from ...services.semantic_fact_service import SemanticFactService
router = APIRouter(prefix="/semantic-facts", tags=["semantic_facts"])
class SemanticFactsBackfillRequest(BaseModel):
entity_types: Optional[list[str]] = Field(default=None)
only_missing: bool = Field(default=True)
limit_per_type: int = Field(default=200, ge=1, le=1000)
@router.get("/admin/overview")
async def get_semantic_facts_overview(
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = SemanticFactService(db, tenant_id=int(current_user.tenant_id))
return success_response(await service.get_admin_overview())
@router.get("/admin/objects")
async def list_semantic_fact_objects(
entity_type: str | None = Query(default=None),
q: str | None = Query(default=None),
status: str | None = Query(default=None),
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = SemanticFactService(db, tenant_id=int(current_user.tenant_id))
items = await service.list_admin_objects(
entity_types=[entity_type] if entity_type else None,
q=q,
status=status,
)
return success_response({"items": items})
@router.get("/admin/objects/{entity_type}/{entity_id}")
async def get_semantic_fact_object_detail(
entity_type: str,
entity_id: int,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = SemanticFactService(db, tenant_id=int(current_user.tenant_id))
try:
detail = await service.get_admin_detail(entity_type=entity_type, entity_id=int(entity_id))
except ValueError as exc:
return JSONResponse(status_code=404, content=error_response(404, str(exc)))
return success_response(detail)
@router.post("/admin/objects/{entity_type}/{entity_id}/reextract")
async def reextract_semantic_fact_object(
entity_type: str,
entity_id: int,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = SemanticFactService(db, tenant_id=int(current_user.tenant_id))
row = await service.enqueue_entity_event(
entity_type=entity_type,
entity_id=int(entity_id),
operation="extract",
trigger_type="manual_retry",
)
await db.commit()
return success_response(
{
"queued": True,
"event_id": int(row.id),
"entity_type": str(entity_type),
"entity_id": int(entity_id),
}
)
@router.post("/admin/backfill")
async def enqueue_semantic_fact_backfill(
body: SemanticFactsBackfillRequest,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(require_manager_user),
):
service = SemanticFactService(db, tenant_id=int(current_user.tenant_id))
result = await service.enqueue_backfill(
entity_types=body.entity_types,
only_missing=bool(body.only_missing),
limit_per_type=int(body.limit_per_type),
)
await db.commit()
return success_response(result)

258
backend/app/config/ai_prompts.py
View File

@ -165,23 +165,31 @@ PROMPT_RAG_QUERY_SEMANTICS = """
你的职责不是靠关键词仓库猜，而是基于当前消息表达的目的、对象、作用域和 bundle 上下文，输出：
- conversation intent（question / statement / mixed）
- query_type（count / list / status / explain / unknown）
- query_type（existence / count / list / status / explain / diagnose / compare / unknown）
- object_mentions（用户当前想解析的对象提及，保留原始业务名词短语）
- retrieval_domains（db / graph / doc / feishu / pmflow / url）
- subject_focus（people / project / task / signal / charter / workload / feature / unknown）
- scope_hint（none / global / project）
- feature_focus（none / purpose / entry / catalog）
- query_profile（needs_private_context / needs_freshness / referenced_objects）
硬约束：
- 不要复述 workflow bundle 的内部字段名给用户。
- 如果 bundle 已经明确显示这是 query_flow，优先按 query 处理；只有消息本身同时包含明确执行动作时才输出 mixed。
- query_type 只用于 Adaptive RAG 的检索与快路径规划，不代表最终回复场景。
- query_type 取值只能是：count、list、status、explain、unknown。
- query_type 取值只能是：existence、count、list、status、explain、diagnose、compare、unknown。
- retrieval_domains 只能从：db、graph、doc、feishu、pmflow、url 中选择。
- subject_focus 只能是：people、project、task、signal、charter、workload、feature、unknown。
- scope_hint 只能是：none、global、project。
- feature_focus 只能是：none、purpose、entry、catalog。
- query_profile.referenced_objects 只能从：unknown、system_config、model_routing、project_manifest、feature_catalog、project_object、internal_document、person_object、public_model、public_company、weather、market_data、sports_event、news_topic 中选择。
- 如果问题是在问人员、成员、负责人清单或人数，subject_focus 应为 people。
- 如果问题是在问功能做什么、入口在哪、模块如何进入，subject_focus 应为 feature，并结合 feature_focus 标明是 purpose / entry / catalog。
- 如果问题是在问“有没有 / 是否存在 / 系统里有无”这类内部对象存在性，query_type 应为 existence；不要因为对象名里出现“系统 / 平台 / 助手”就改判成 explain。像 `帮我看看系统里面有没有开票系统。`，应输出 `query_type=existence`，并把 `object_mentions=["开票系统"]`。
- 外部公开事实示例：`当前 codex 最新的模型用的是什么？`
query_profile 应体现 `needs_freshness=true`、`referenced_objects=["public_model"]`，retrieval_domains 应优先为 ["url"]，不要因为出现“模型”二字就改判成系统内部配置问题。
- 内部运行时配置示例：`我们系统当前接的是哪个 codex 模型配置？`
query_profile 应体现 `needs_private_context=true`、`referenced_objects=["system_config","model_routing"]`，retrieval_domains 应优先为 ["db","pmflow"]，这是在问系统配置而不是公开外部事实。
- 若证据不足，query_type 输出 unknown，retrieval_domains 选择保守但完整的查询范围。
- 输出必须是一个 JSON 对象，不要输出 Markdown。
@ -190,11 +198,18 @@ JSON schema:
"intent": "question | statement | mixed",
"question_part": "string",
"statement_part": "string",
"query_type": "count | list | status | explain | unknown",
"query_type": "existence | count | list | status | explain | diagnose | compare | unknown",
"object_mentions": ["string"],
"retrieval_domains": ["db | graph | doc | feishu | pmflow | url"],
"subject_focus": "people | project | task | signal | charter | workload | feature | unknown",
"scope_hint": "none | global | project",
"feature_focus": "none | purpose | entry | catalog",
"query_profile": {{
"needs_private_context": false,
"needs_freshness": false,
"referenced_objects": ["unknown"],
"confidence": 0.0
}},
"confidence": 0.0
}}
@ -248,6 +263,241 @@ message:
{message}
""".strip()
PROMPT_INTERNAL_TARGET_RESOLUTION = """
你是 PMFlow 内部查询链路的 Target Resolver rerank 层。
你的输入已经包含：
1. 用户问题
2. 已经确定的 operation_contract
3. 受控候选对象列表
你的职责只有一个：
- 在给定候选里选出最符合当前问题语义的目标对象
- 如果候选之间仍然无法稳定区分，就要求澄清
硬约束：
- 只能在候选列表里选择，禁止生成新的对象、ID、类型或 target_ref。
- 如果是存在性问题，优先选择“被问到的业务对象”，不要把对象名里带“系统/平台/助手”的词面自动改判成 feature explain。
- 若前两名候选分差很小、证据不足或候选语义冲突，输出 `clarify=true`。
- 输出必须是 JSON 对象，不要输出 Markdown。
JSON schema:
{{
"selected_index": 0,
"clarify": false,
"confidence": 0.0,
"reason": "string"
}}
question:
{question}
operation_contract:
{operation_contract}
candidates:
{candidates}
""".strip()
PROMPT_EXTERNAL_QUERY_PLAN = """
你是 PMFlow External Query Planner。
你的职责不是直接回答外部事实，而是把当前问题整理成一个受控的外部查询计划。
硬约束：
- 只输出 JSON 对象，不要输出 Markdown。
- 不要直接给外部事实结论。
- `query_kind` 只能是：weather、news、company、model、market、generic_web、unknown。
- `answer_contract` 只能是：direct_fact、summary_with_sources、compare_with_sources、clarify。
- `search_queries` 必须是简短、可直接执行的查询短语。
- `preferred_domains` 只放域名，不要带协议头。
- 若用户问题明显要求最新、当前、今天、最近等外部信息，`freshness_required=true`。
- 若问题更像对比/比较，`requires_multi_source=true` 且 `answer_contract=compare_with_sources`。
- 结构化失败时优先保守，不要编造实体。
JSON schema:
{{
"plan_version": "v1",
"query_kind": "weather | news | company | model | market | generic_web | unknown",
"answer_contract": "direct_fact | summary_with_sources | compare_with_sources | clarify",
"freshness_required": false,
"topic": "string",
"subject_entities": ["string"],
"location_hint": "string | null",
"time_range_hint": "string | null",
"preferred_domains": ["string"],
"search_queries": ["string"],
"result_limit": 3,
"requires_multi_source": false,
"clarify_on_ambiguity": false
}}
question:
{question}
query_semantics:
{query_semantics}
fallback_queries:
{fallback_queries}
""".strip()
def _build_semantic_fact_extract_prompt(
*,
entity_label: str,
goal_text: str,
allowed_examples: list[str],
payload_key: str,
) -> str:
examples_text = "\n".join(f"- `{item}`" for item in allowed_examples)
payload_marker = f"<<{payload_key}>>"
return f"""
你是 PMFlow 的{entity_label}语义事实抽取器。
目标：
- {goal_text}
- 只输出有证据支撑的事实，不要脑补
- 如果没有明确事实，返回空数组
允许抽取的事实示例：
{examples_text}
输出要求：
- 只输出 JSON 对象
- `facts` 必须是数组
- 每个 fact 必须包含：
- `fact_namespace`
- `fact_key`
- `fact_value_text`
- `fact_value_norm`
- `value_type`
- `confidence`
- `evidence_excerpt`
JSON schema:
{{
"facts": [
{{
"fact_namespace": "domain | alias | purpose | audience | status | product | role",
"fact_key": "string",
"fact_value_text": "string",
"fact_value_norm": "string",
"value_type": "keyword | text",
"confidence": 0.0,
"evidence_excerpt": "string"
}}
]
}}
{payload_key}:
{payload_marker}
""".strip()
PROMPT_SEMANTIC_FACT_EXTRACT_PROJECT = _build_semantic_fact_extract_prompt(
entity_label="项目",
goal_text="从项目名称、标题、描述里抽取稳定、可复用、适合内部查询的语义事实",
allowed_examples=[
"market_theme：A股、港股、美股",
"business_domain：开票、财务、采购、投研",
"aliases：开票系统、A股项目、库存报表工具",
"audience：投顾团队、研究团队、财务运营",
"product_form：助手、平台、报表工具",
],
payload_key="project_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_TASK = _build_semantic_fact_extract_prompt(
entity_label="任务",
goal_text="从任务标题、描述、状态、优先级里抽取任务主题、对象别名、交付目标和责任线索",
allowed_examples=[
"deliverable：开票接口、财务报表页、信号看板",
"business_domain：开票、财务、采购、风控",
"aliases：开票任务、A股需求任务",
"owner_role：后端、前端、AI",
"status_hint：待办、进行中、已完成",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_SIGNAL = _build_semantic_fact_extract_prompt(
entity_label="信号",
goal_text="从信号标题、摘要、正文、状态里抽取业务主题、问题对象、影响域和别名",
allowed_examples=[
"business_domain：开票、结算、投研、采购",
"problem_object：开票系统、报表平台、A股项目",
"aliases：开票需求、发票流程问题",
"signal_type：bug、需求、机会",
"status_hint：待处理、已关闭",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_DOCUMENT = _build_semantic_fact_extract_prompt(
entity_label="文档",
goal_text="从文档标题、来源、内容预览里抽取文档主题、业务对象、别名和适用团队",
allowed_examples=[
"business_domain：开票、财务、采购、投研",
"document_topic：实施方案、接口文档、复盘总结",
"aliases：开票系统文档、A股项目说明",
"audience：财务团队、研发团队、运营团队",
"source_form：文件、网页、知识库条目",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_USER = _build_semantic_fact_extract_prompt(
entity_label="人员",
goal_text="从成员姓名、角色、领域、简介里抽取稳定的职责、领域、别名和角色事实",
allowed_examples=[
"role_scope：后端、前端、AI、管理员",
"business_domain：财务、采购、投研",
"aliases：开票负责人、A股项目 owner",
"audience：财务团队、研发团队",
"status_hint：在岗、离线",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_REQUIREMENT = _build_semantic_fact_extract_prompt(
entity_label="需求宪章",
goal_text="从需求宪章标题、状态和内容里抽取需求主题、业务对象、别名和目标团队",
allowed_examples=[
"business_domain：开票、财务、采购、投研",
"target_object：开票系统、报表平台、A股项目",
"aliases：开票需求、开票平台需求",
"audience：财务团队、研究团队",
"status_hint：草稿、对齐中、已对齐",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_RELEASE = _build_semantic_fact_extract_prompt(
entity_label="发布",
goal_text="从发布标题、状态和内容里抽取发布对象、业务域、别名和阶段状态",
allowed_examples=[
"business_domain：开票、财务、采购、投研",
"release_object：开票系统、结算平台、A股工具",
"aliases：开票版本、财务发布",
"status_hint：pending、verified、online",
"audience：财务团队、运营团队",
],
payload_key="entity_payload",
)
PROMPT_SEMANTIC_FACT_EXTRACT_REVIEW = _build_semantic_fact_extract_prompt(
entity_label="复盘",
goal_text="从复盘标题、状态和内容里抽取复盘对象、业务主题、别名和适用团队",
allowed_examples=[
"business_domain：开票、财务、采购、投研",
"review_object：开票系统、A股项目、财务流程",
"aliases：开票复盘、阶段复盘",
"status_hint：draft、archived",
"audience：财务团队、研发团队",
],
payload_key="entity_payload",
)
PROMPT_RAG_ANSWER_SCENE_SEMANTICS = """
你是 PMFlow Adaptive RAG 的 answer scene 规划层。
@ -584,6 +834,8 @@ __all__ = [
"PMFLOW_SEMANTIC_CONTEXT_DECISION_PROMPT",
"PROMPT_RAG_QUERY_SEMANTICS",
"PROMPT_RAG_FOLLOWUP_ANCHOR_SEMANTICS",
"PROMPT_INTERNAL_TARGET_RESOLUTION",
"PROMPT_EXTERNAL_QUERY_PLAN",
"PROMPT_RAG_ANSWER_SCENE_SEMANTICS",
"PMFLOW_REQUIREMENT_PLAN_ROUND_PROMPT",
"PMFLOW_REQUIREMENT_PLAN_DRAFT_PROMPT",

2
backend/app/main.py
View File

@ -31,10 +31,12 @@ from .services.scheduler_center_core import (
WorkloadRecalcScheduler,
)
from .services.skill import init_system_skill_tags
from .services.semantic_fact_service import register_semantic_fact_hooks
configure_logging(settings.debug)
logger = get_logger(__name__)
register_context_cache_invalidation_hooks()
register_semantic_fact_hooks()
@asynccontextmanager

6
backend/app/models/__init__.py
View File

@ -45,6 +45,9 @@ from .knowledge_base_doc import KnowledgeBaseDoc
from .knowledge_base_chunk import KnowledgeBaseChunk
from .qa_record import QARecord
from .qa_feedback import QAFeedback
from .query_route_memory import QueryRouteMemory
from .semantic_fact import SemanticFact
from .semantic_fact_extraction_run import SemanticFactExtractionRun
from .feedback_center import FeedbackCenterRecord, FeedbackCenterRecordRevision
from .reply_iteration import QualityAsset, QualityAssetPair, JudgeRun, OptimizerCandidate, PromptRelease
from .training_pipeline import (
@ -130,6 +133,9 @@ __all__ = [
"KnowledgeBaseChunk",
"QARecord",
"QAFeedback",
"QueryRouteMemory",
"SemanticFact",
"SemanticFactExtractionRun",
"FeedbackCenterRecord",
"FeedbackCenterRecordRevision",
"QualityAsset",

44
backend/app/models/query_route_memory.py Normal file
View File

@ -0,0 +1,44 @@
"""问答路由记忆样本。"""
from __future__ import annotations
from typing import Any, Optional
from sqlalchemy import BigInteger, ForeignKey, Integer, Text, UniqueConstraint, VARCHAR
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
from .types import JSONB
class QueryRouteMemory(Base):
__tablename__ = "query_route_memories"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
qa_record_id: Mapped[int] = mapped_column(
Integer,
ForeignKey("qa_records.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
project_id: Mapped[Optional[int]] = mapped_column(BigInteger, nullable=True, index=True)
question_text: Mapped[str] = mapped_column(Text, nullable=False)
route_mode_predicted: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, index=True)
route_mode_effective: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, index=True)
decision_origin: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True, index=True)
reply_contract: Mapped[Optional[Any]] = mapped_column(JSONB, nullable=True)
probe_snapshot: Mapped[Optional[Any]] = mapped_column(JSONB, nullable=True)
query_profile: Mapped[Optional[Any]] = mapped_column(JSONB, nullable=True)
feedback_outcome: Mapped[Optional[Any]] = mapped_column(JSONB, nullable=True)
question_embedding: Mapped[Optional[Any]] = mapped_column(JSONB, nullable=True)
__table_args__ = (
UniqueConstraint(
"tenant_id",
"qa_record_id",
name="uq_query_route_memories_tenant_qa_record",
),
)
__all__ = ["QueryRouteMemory"]

44
backend/app/models/semantic_fact.py Normal file
View File

@ -0,0 +1,44 @@
"""语义事实读模型。"""
from __future__ import annotations
from typing import Optional
from sqlalchemy import BigInteger, Boolean, Float, Integer, Text, UniqueConstraint, VARCHAR
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class SemanticFact(Base):
__tablename__ = "semantic_facts"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
entity_type: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, index=True)
entity_id: Mapped[int] = mapped_column(BigInteger, nullable=False, index=True)
fact_namespace: Mapped[str] = mapped_column(VARCHAR(64), nullable=False, index=True)
fact_key: Mapped[str] = mapped_column(VARCHAR(64), nullable=False, index=True)
fact_value_text: Mapped[str] = mapped_column(Text, nullable=False)
fact_value_norm: Mapped[str] = mapped_column(VARCHAR(255), nullable=False, index=True)
value_type: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, default="keyword")
confidence: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
evidence_source_type: Mapped[Optional[str]] = mapped_column(VARCHAR(32), nullable=True)
evidence_source_id: Mapped[Optional[int]] = mapped_column(BigInteger, nullable=True)
evidence_excerpt: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
extract_version: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True)
freshness_at: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True)
is_active: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True, index=True)
__table_args__ = (
UniqueConstraint(
"tenant_id",
"entity_type",
"entity_id",
"fact_key",
"fact_value_norm",
name="uq_semantic_facts_tenant_entity_key_norm",
),
)
__all__ = ["SemanticFact"]

29
backend/app/models/semantic_fact_extraction_run.py Normal file
View File

@ -0,0 +1,29 @@
"""语义事实抽取运行记录。"""
from __future__ import annotations
from typing import Optional
from sqlalchemy import BigInteger, Integer, Text, VARCHAR
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
class SemanticFactExtractionRun(Base):
__tablename__ = "semantic_fact_extraction_runs"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
entity_type: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, index=True)
entity_id: Mapped[int] = mapped_column(BigInteger, nullable=False, index=True)
trigger_type: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, default="manual_retry")
status: Mapped[str] = mapped_column(VARCHAR(32), nullable=False, default="queued", index=True)
input_fingerprint: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True, index=True)
facts_written: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
failure_reason: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
extract_version: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True)
started_at: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True)
finished_at: Mapped[Optional[str]] = mapped_column(VARCHAR(64), nullable=True)
__all__ = ["SemanticFactExtractionRun"]

8
backend/app/services/adaptive_rag_service/mixin_answer_entry.py
View File

@ -68,6 +68,9 @@ class AdaptiveRAGAnswerEntryMixin:
message_intent: Optional[str] = None,
mixed_signal_part: Optional[str] = None,
consultant_mode_preference: Optional[str] = None,
source_policy: Optional[str] = None,
precomputed_query_semantics: Optional[dict[str, Any]] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
started = time.perf_counter()
clean_q = str(question or "").strip()
@ -108,6 +111,9 @@ class AdaptiveRAGAnswerEntryMixin:
action_mode=action_mode,
source_sender_open_id=source_sender_open_id,
thread_context=thread_context,
source_policy=source_policy,
precomputed_query_semantics=precomputed_query_semantics,
route_decision=route_decision,
)
clean_q = str(runtime.get("resolved_question") or clean_q).strip() or clean_q
session_id = runtime["session_id"]
@ -121,6 +127,7 @@ class AdaptiveRAGAnswerEntryMixin:
query_semantics = runtime["query_semantics"] if isinstance(runtime.get("query_semantics"), dict) else {}
retrieval_domains = list(runtime.get("retrieval_domains") or [])
scene_semantics = runtime["scene_semantics"] if isinstance(runtime.get("scene_semantics"), dict) else {}
route_decision = runtime["route_decision"] if isinstance(runtime.get("route_decision"), dict) else {}
runtime_message_intent = str(runtime.get("message_intent") or "").strip().lower() or None
effective_message_intent = str(message_intent or runtime_message_intent or "").strip().lower() or None
enforce_scene_prompt_only = self._should_enforce_scene_prompt_only(
@ -181,4 +188,5 @@ class AdaptiveRAGAnswerEntryMixin:
mixed_signal_part=mixed_signal_part,
anchor_resolution=anchor_resolution,
consultant_mode_preference=consultant_mode_preference,
route_decision=route_decision,
)

566
backend/app/services/adaptive_rag_service/mixin_answer_mainline.py
View File

@ -5,80 +5,150 @@ from .shared import *
class AdaptiveRAGAnswerMainlineMixin:
_EXTERNAL_FACT_FRESHNESS_MARKERS = (
"最新",
"目前",
"当前",
"截至现在",
"截至当前",
"截止当前",
"今天",
"现在",
"最近",
"出到什么",
"发布了吗",
"发布到",
"版本",
"最高模型",
)
_MODEL_ENTITY_MARKERS = (
"codex",
"gpt",
"chatgpt",
"claude",
"deepseek",
"qwen",
"gemini",
"copilot",
"grok",
"kimi",
"llama",
"openai",
"anthropic",
)
_INTERNAL_MODEL_CONFIG_MARKERS = (
"ai-config",
"/ai-config",
"模型配置",
"配置项",
"配置值",
"路由规则",
"初始化参数",
"llm服务",
"llm 服务",
"运行时",
"当前系统",
"我们系统",
"pmflow",
"本系统",
"接入的模型",
"接的是",
"用的是哪个模型配置",
)
_PROJECT_CONTEXT_POSITIVE_MARKERS = (
"这个项目",
"当前项目",
"本项目",
"我们项目",
"项目所在行业",
"对项目的影响",
"对这个项目的影响",
"对当前项目的影响",
"对我们项目的影响",
"结合项目",
"结合当前项目",
"落到项目",
)
_PROJECT_CONTEXT_NEGATIVE_MARKERS = (
"不要提项目",
"别提项目",
"不要结合项目",
"不用结合项目",
"不需要结合项目",
"先别结合项目",
"不要绑定项目",
"别绑定项目",
)
@staticmethod
def _internal_answer_contract_for_query_type(query_type: str) -> str:
normalized = str(query_type or QUERY_TYPE_UNKNOWN).strip().lower()
if normalized == QUERY_TYPE_EXISTENCE:
return "yes_no_with_evidence"
if normalized == QUERY_TYPE_COUNT:
return "count_scalar"
if normalized == QUERY_TYPE_LIST:
return "entity_list"
if normalized == QUERY_TYPE_STATUS:
return "entity_snapshot"
if normalized == QUERY_TYPE_EXPLAIN:
return "explanation_with_citations"
if normalized == QUERY_TYPE_DIAGNOSE:
return "diagnosis_with_evidence"
if normalized == QUERY_TYPE_COMPARE:
return "comparison_with_basis"
return "clarify_or_fail"
@staticmethod
def _internal_entity_type_for_subject_focus(subject_focus: str) -> str:
normalized = str(subject_focus or "").strip().lower()
if normalized == "people":
return "user"
if normalized in {"project", "task", "signal", "feature"}:
return normalized
return "unknown"
def _build_internal_query_plan(
self,
*,
question: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]],
operation_contract: Optional[dict[str, Any]] = None,
target_resolution: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
operation = operation_contract if isinstance(operation_contract, dict) else (
semantics.get("operation_contract") if isinstance(semantics.get("operation_contract"), dict) else {}
)
target = target_resolution if isinstance(target_resolution, dict) else (
semantics.get("target_resolution") if isinstance(semantics.get("target_resolution"), dict) else {}
)
source_policy = str(
semantics.get("effective_source_policy") or semantics.get("source_policy") or ""
).strip().lower()
if source_policy != "internal":
return None
query_type = str(
operation.get("query_type")
or semantics.get("query_type")
or QUERY_TYPE_UNKNOWN
).strip().lower() or QUERY_TYPE_UNKNOWN
subject_focus = self._query_subject_focus(semantics) if hasattr(self, "_query_subject_focus") else "unknown"
entity_type = str(
target.get("target_entity_type")
or self._internal_entity_type_for_subject_focus(subject_focus)
).strip().lower() or "unknown"
scope_hint = self._query_scope_hint(semantics) if hasattr(self, "_query_scope_hint") else "none"
target_ref = str(target.get("target_ref") or "").strip() or None
filters: list[dict[str, Any]] = []
if entity_type == "project" and target_ref and target_ref.startswith("project:"):
try:
resolved_project_id = int(target_ref.split(":", 1)[1])
except Exception:
resolved_project_id = None
if resolved_project_id is not None:
filters.append(
{
"field": "project_id",
"operator": "equals",
"value": resolved_project_id,
"value_source": "target_resolution",
}
)
elif entity_type == "project":
mention_collector = getattr(self, "_collect_lookup_terms", None)
if callable(mention_collector):
operation_mentions = mention_collector(
question=question,
query_semantics=semantics,
operation_contract=operation,
target_resolution=target,
limit=3,
)
else:
operation_mentions = [
str(item or "").strip()
for item in list(operation.get("object_mentions") or [])
if str(item or "").strip()
]
if operation_mentions:
for raw in operation_mentions[:3]:
filters.append(
{
"field": "semantic_label",
"operator": "equals_or_alias",
"value": raw,
"value_source": "operation_contract",
}
)
elif entity_type == "feature" and target_ref:
filters.append(
{
"field": "feature_ref",
"operator": "equals",
"value": target_ref,
"value_source": "target_resolution",
}
)
elif entity_type == "config" and target_ref:
filters.append(
{
"field": "config_ref",
"operator": "equals",
"value": target_ref,
"value_source": "target_resolution",
}
)
return {
"plan_version": "v2" if operation or target else "v1",
"query_type": query_type,
"entity_type": entity_type,
"answer_contract": self._internal_answer_contract_for_query_type(query_type),
"target_ref": target_ref,
"filters": filters,
"scope": {
"tenant_bound": True,
"project_id": int(project_id) if project_id is not None else None,
"scope_hint": scope_hint,
},
"retrieval_policy": {
"facts_first": True,
"allow_fallback_text": True,
},
"verifier_checks": [
"contract_matches_query_type",
"evidence_backed",
],
}
@classmethod
def _extract_prompt_variant_from_policy_version(cls, policy_version: Optional[str]) -> Optional[str]:
@ -93,46 +163,6 @@ class AdaptiveRAGAnswerMainlineMixin:
return value or None
return None
@classmethod
def _question_explicitly_requests_project_context(
cls,
*,
question: str,
project_id: Optional[int],
) -> bool:
if project_id is None:
return False
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
if any(marker in normalized for marker in cls._PROJECT_CONTEXT_NEGATIVE_MARKERS):
return False
return any(marker in normalized for marker in cls._PROJECT_CONTEXT_POSITIVE_MARKERS)
@classmethod
def _looks_like_external_freshness_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
return any(marker in normalized for marker in cls._EXTERNAL_FACT_FRESHNESS_MARKERS)
@classmethod
def _looks_like_internal_model_config_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
return any(marker in normalized for marker in cls._INTERNAL_MODEL_CONFIG_MARKERS)
@classmethod
def _looks_like_model_fact_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
has_entity = any(marker in normalized for marker in cls._MODEL_ENTITY_MARKERS)
if not has_entity:
return False
return "模型" in normalized or cls._looks_like_external_freshness_question(question)
async def _resolve_anchor_prompt_variant(
self,
*,
@ -161,40 +191,18 @@ class AdaptiveRAGAnswerMainlineMixin:
scene_semantics: Optional[dict[str, Any]],
anchor_prompt_variant: Optional[str] = None,
) -> dict[str, Any]:
_ = question
normalized = dict(scene_semantics or {})
scene_key = str(normalized.get("scene_key") or "default").strip().lower() or "default"
anchor_variant = str(anchor_prompt_variant or "").strip().lower()
explicit_project_context = self._question_explicitly_requests_project_context(
question=question,
project_id=project_id,
)
requires_freshness = self._looks_like_external_freshness_question(question)
internal_model_config_question = self._looks_like_internal_model_config_question(question)
model_fact_question = self._looks_like_model_fact_question(question)
anchored_trend_followup = anchor_variant in {"external_trend", "industry_trend"}
if scene_key == "ai_pmo" and model_fact_question and requires_freshness and not internal_model_config_question:
scene_key = "external_trend" if explicit_project_context else "industry_trend"
if scene_key in {"ai_pmo", "ai_pmo_correction"} and anchor_variant in {"external_trend", "industry_trend"}:
scene_key = anchor_variant
normalized["snapshot_focus"] = "none"
normalized["deep_intent"] = "none"
normalized["external_fact_chat"] = True
normalized["requires_freshness"] = True
if scene_key == "ai_pmo_correction" and anchored_trend_followup:
scene_key = "external_trend" if explicit_project_context else "industry_trend"
normalized["snapshot_focus"] = "none"
normalized["deep_intent"] = "none"
normalized["external_fact_chat"] = True
if scene_key == "external_trend" and not explicit_project_context:
if scene_key == "external_trend" and project_id is None:
scene_key = "industry_trend"
if scene_key in {"external_trend", "industry_trend"}:
normalized["external_fact_chat"] = True
normalized["requires_freshness"] = bool(requires_freshness)
if requires_freshness:
normalized["deep_intent"] = "none"
normalized["scene_key"] = scene_key
return normalized
@ -217,7 +225,12 @@ class AdaptiveRAGAnswerMainlineMixin:
normalized_thinking_mode = normalize_thinking_mode(requested_thinking_mode)
normalized_query_type = str(query_type or QUERY_TYPE_UNKNOWN).strip().lower()
normalized_consultant_mode = str(consultant_mode_preference or "").strip().lower()
fast_path_candidate = normalized_query_type in {QUERY_TYPE_COUNT, QUERY_TYPE_LIST, QUERY_TYPE_STATUS}
fast_path_candidate = normalized_query_type in {
QUERY_TYPE_EXISTENCE,
QUERY_TYPE_COUNT,
QUERY_TYPE_LIST,
QUERY_TYPE_STATUS,
}
normalized_question = str(question or "").strip()
normalized_effective = str(effective_question or "").strip()
@ -290,6 +303,9 @@ class AdaptiveRAGAnswerMainlineMixin:
source_message_id: Optional[str] = None,
source_sender_open_id: Optional[str] = None,
thread_context: Optional[dict[str, Any]] = None,
source_policy: Optional[str] = None,
precomputed_query_semantics: Optional[dict[str, Any]] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
external_conversation_key = self._build_external_conversation_key(
asked_via=asked_via,
@ -358,20 +374,57 @@ class AdaptiveRAGAnswerMainlineMixin:
session_state=state_row,
anchor_resolution=anchor_resolution,
)
query_semantics = await self._resolve_query_semantics(
message=effective_q,
source_system=asked_via,
metadata={
"project_id": project_id,
"source_chat_id": source_chat_id,
"source_message_id": source_message_id,
"conversation_scope_key": external_conversation_key,
"reply_to_message_id": str((thread_context or {}).get("reply_to_message_id") or "").strip() or None,
"root_message_id": str((thread_context or {}).get("root_message_id") or "").strip() or None,
},
query_semantics = dict(precomputed_query_semantics or {}) if isinstance(precomputed_query_semantics, dict) else {}
route_preview = await self.preview_query_route_decision(
question=effective_q,
asked_via=asked_via,
project_id=project_id,
source_chat_id=source_chat_id,
source_message_id=source_message_id,
conversation_scope_key=external_conversation_key,
reply_to_message_id=str((thread_context or {}).get("reply_to_message_id") or "").strip() or None,
root_message_id=str((thread_context or {}).get("root_message_id") or "").strip() or None,
source_policy_hint=source_policy,
query_semantics=query_semantics,
) if not isinstance(route_decision, dict) or not query_semantics else {
"query_semantics": self._apply_route_decision_to_query_semantics(
query_semantics=query_semantics,
route_decision=route_decision,
),
"route_decision": self._normalize_query_route_decision(route_decision),
}
query_semantics = (
dict(route_preview.get("query_semantics") or {})
if isinstance(route_preview.get("query_semantics"), dict)
else {}
)
query_semantics = dict(query_semantics) if isinstance(query_semantics, dict) else {}
query_type = str(query_semantics.get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower() or QUERY_TYPE_UNKNOWN
operation_contract = None
operation_contract_builder = getattr(self, "_build_operation_contract", None)
if callable(operation_contract_builder):
operation_contract = operation_contract_builder(
question=effective_q,
query_semantics=query_semantics,
)
if isinstance(operation_contract, dict):
query_semantics["operation_contract"] = operation_contract
internal_query_plan = self._build_internal_query_plan(
question=effective_q,
project_id=project_id,
query_semantics=query_semantics,
operation_contract=operation_contract,
)
if internal_query_plan:
query_semantics["query_plan"] = internal_query_plan
route_decision = (
dict(route_preview.get("route_decision") or {})
if isinstance(route_preview.get("route_decision"), dict)
else {}
)
query_type = str(
(operation_contract or {}).get("query_type")
or query_semantics.get("query_type")
or QUERY_TYPE_UNKNOWN
).strip().lower() or QUERY_TYPE_UNKNOWN
conversation_text = "\n".join(
[
f"Q: {str(row.get('question') or '').strip()}\nA: {str(row.get('answer') or '').strip()}"
@ -386,6 +439,16 @@ class AdaptiveRAGAnswerMainlineMixin:
message_intent=str(query_semantics.get("message_intent") or query_semantics.get("intent") or "").strip().lower() or None,
query_type=query_type,
conversation_text=conversation_text,
source_policy=str(
route_decision.get("effective_source_policy")
or query_semantics.get("effective_source_policy")
or query_semantics.get("source_policy")
or source_policy
or ""
).strip().lower()
or None,
project_id=project_id,
query_semantics=query_semantics,
)
anchor_prompt_variant = await self._resolve_anchor_prompt_variant(
anchor_resolution=anchor_resolution,
@ -400,6 +463,24 @@ class AdaptiveRAGAnswerMainlineMixin:
if bool(scene_semantics.get("requires_freshness")):
query_semantics["external_freshness_only"] = True
query_semantics["retrieval_domains"] = ["url"]
query_semantics["source_policy"] = "external"
query_semantics["effective_source_policy"] = "external"
route_decision = self._normalize_query_route_decision(
{
**route_decision,
"route_mode": str(route_decision.get("route_mode") or "external"),
"effective_source_policy": "external",
"decision_origin": str(route_decision.get("decision_origin") or "freshness_guard"),
"reply_contract": {
"mode": "explicit_source",
"head": "外部实时查询：",
"tail_hint": None,
},
"query_profile": query_semantics.get("query_profile"),
"probe_snapshot_summary": route_decision.get("probe_snapshot_summary"),
"probe_snapshot": route_decision.get("probe_snapshot"),
}
)
requested_action_mode = self._normalize_action_mode(action_mode)
effective_action_mode = await self._resolve_action_mode_async(
requested_mode=requested_action_mode,
@ -423,6 +504,7 @@ class AdaptiveRAGAnswerMainlineMixin:
"scene_semantics": scene_semantics if isinstance(scene_semantics, dict) else {},
"effective_action_mode": effective_action_mode,
"anchor_resolution": anchor_resolution,
"route_decision": route_decision,
}
async def _execute_base_answer_mainline(
@ -455,7 +537,26 @@ class AdaptiveRAGAnswerMainlineMixin:
mixed_signal_part: Optional[str],
anchor_resolution: Optional[FollowupAnchorResolution] = None,
consultant_mode_preference: Optional[str] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
operation_contract = (
dict(query_semantics.get("operation_contract") or {})
if isinstance(query_semantics.get("operation_contract"), dict)
else {}
)
operation_contract_builder = getattr(self, "_build_operation_contract", None)
if not operation_contract and callable(operation_contract_builder):
operation_contract = operation_contract_builder(
question=effective_q,
query_semantics=query_semantics,
)
if isinstance(operation_contract, dict):
query_semantics["operation_contract"] = operation_contract
query_type = str(
operation_contract.get("query_type")
or query_type
or QUERY_TYPE_UNKNOWN
).strip().lower() or QUERY_TYPE_UNKNOWN
planner = self._plan_query_runtime(
question=clean_q,
effective_question=effective_q,
@ -483,11 +584,31 @@ class AdaptiveRAGAnswerMainlineMixin:
anchor_resolution=anchor_resolution,
session_state=state_row,
)
route_clarification_payload = self._build_route_clarification_payload(route_decision=route_decision)
meta_followup_payload = await self._build_meta_followup_payload(
question=clean_q,
anchor_resolution=anchor_resolution,
conversation_scope_key=self._get_active_external_qa_context().get("conversation_scope_key"),
)
clarification_result = await self._maybe_return_clarification_response(
clarification_payload=clarification_payload or route_clarification_payload,
started=started,
clean_q=clean_q,
effective_q=effective_q,
query_type=query_type,
asked_by=asked_by,
asked_via=asked_via,
project_id=project_id,
session_id=session_id,
turn_index=turn_index,
state_row=state_row,
rewrite_meta=rewrite_meta,
context_session_id=context_session_id,
active_thinking_mode=active_thinking_mode,
route_decision=route_decision,
)
if clarification_result is not None:
return clarification_result
meta_followup_result = await self._maybe_return_meta_followup_response(
meta_followup_payload=meta_followup_payload,
started=started,
@ -503,27 +624,10 @@ class AdaptiveRAGAnswerMainlineMixin:
rewrite_meta=rewrite_meta,
context_session_id=context_session_id,
active_thinking_mode=active_thinking_mode,
route_decision=route_decision,
)
if meta_followup_result is not None:
return meta_followup_result
clarification_result = await self._maybe_return_clarification_response(
clarification_payload=clarification_payload,
started=started,
clean_q=clean_q,
effective_q=effective_q,
query_type=query_type,
asked_by=asked_by,
asked_via=asked_via,
project_id=project_id,
session_id=session_id,
turn_index=turn_index,
state_row=state_row,
rewrite_meta=rewrite_meta,
context_session_id=context_session_id,
active_thinking_mode=active_thinking_mode,
)
if clarification_result is not None:
return clarification_result
system_snapshot = await self.context_cache.get_system_snapshot(
db=self.db,
@ -534,8 +638,44 @@ class AdaptiveRAGAnswerMainlineMixin:
tenant_id=self.tenant_id,
user_id=asked_by,
)
prefers_external_retrieval = (
str(query_semantics.get("effective_source_policy") or query_semantics.get("source_policy") or "").strip().lower()
== "external"
)
target_resolution = (
dict(query_semantics.get("target_resolution") or {})
if isinstance(query_semantics.get("target_resolution"), dict)
else {}
)
target_resolver = getattr(self, "_resolve_internal_target", None)
if not prefers_external_retrieval and callable(target_resolver):
target_resolution = await target_resolver(
question=effective_q,
project_id=project_id,
query_semantics=query_semantics,
system_snapshot=system_snapshot,
operation_contract=operation_contract,
)
if isinstance(target_resolution, dict):
query_semantics["target_resolution"] = target_resolution
patch_scene_semantics = getattr(self, "_patch_scene_semantics_with_target_resolution", None)
if callable(patch_scene_semantics):
scene_semantics = patch_scene_semantics(
scene_semantics=scene_semantics,
operation_contract=operation_contract,
target_resolution=target_resolution,
)
internal_query_plan = self._build_internal_query_plan(
question=effective_q,
project_id=project_id,
query_semantics=query_semantics,
operation_contract=operation_contract,
target_resolution=target_resolution,
)
if internal_query_plan:
query_semantics["query_plan"] = internal_query_plan
if planner.allow_fast_path:
if planner.allow_fast_path and not prefers_external_retrieval:
deterministic_result = await self._run_deterministic_query(
question=effective_q,
project_id=project_id,
@ -559,37 +699,78 @@ class AdaptiveRAGAnswerMainlineMixin:
rewrite_meta=rewrite_meta,
thinking_mode=active_thinking_mode,
answer_mode=ANSWER_MODE_FAST_PATH,
route_decision=route_decision,
query_semantics=query_semantics,
)
internal_model_config_fast_answer = None
internal_model_config_builder = getattr(self, "_build_internal_model_config_fast_answer", None)
if not prefers_external_retrieval and callable(internal_model_config_builder):
internal_model_config_fast_answer = await internal_model_config_builder(
question=effective_q,
query_semantics=query_semantics,
)
if internal_model_config_fast_answer is not None:
return await self._persist_fast_answer(
question=clean_q,
asked_by=asked_by,
asked_via=asked_via,
project_id=project_id,
fast_result=internal_model_config_fast_answer,
session_id=session_id,
external_session_id=context_session_id,
turn_index=turn_index,
query_type=QUERY_TYPE_STATUS,
missing_fields=[],
state_row=state_row,
effective_question=effective_q,
rewrite_meta=rewrite_meta,
thinking_mode=active_thinking_mode,
answer_mode=ANSWER_MODE_FAST_PATH,
route_decision=route_decision,
query_semantics=query_semantics,
)
direct_answer = None
if planner.allow_fast_path:
allow_snapshot_direct_answer = planner.allow_fast_path and not prefers_external_retrieval
snapshot_direct_guard = getattr(self, "_should_allow_snapshot_direct_answer", None)
if callable(snapshot_direct_guard):
allow_snapshot_direct_answer = allow_snapshot_direct_answer and snapshot_direct_guard(
scene_semantics=scene_semantics,
operation_contract=operation_contract,
target_resolution=target_resolution,
)
if allow_snapshot_direct_answer:
direct_answer = self._build_snapshot_direct_answer(
effective_q,
system_snapshot=system_snapshot,
user_snapshot=user_snapshot,
snapshot_focus=str((scene_semantics or {}).get("snapshot_focus") or "none"),
)
direct_result = await self._maybe_return_snapshot_direct_response(
direct_answer=direct_answer,
started=started,
clean_q=clean_q,
effective_q=effective_q,
asked_by=asked_by,
asked_via=asked_via,
project_id=project_id,
session_id=session_id,
turn_index=turn_index,
state_row=state_row,
rewrite_meta=rewrite_meta,
context_session_id=context_session_id,
active_thinking_mode=active_thinking_mode,
user_snapshot=user_snapshot,
)
direct_result = None
if allow_snapshot_direct_answer:
direct_result = await self._maybe_return_snapshot_direct_response(
direct_answer=direct_answer,
started=started,
clean_q=clean_q,
effective_q=effective_q,
asked_by=asked_by,
asked_via=asked_via,
project_id=project_id,
session_id=session_id,
turn_index=turn_index,
state_row=state_row,
rewrite_meta=rewrite_meta,
context_session_id=context_session_id,
active_thinking_mode=active_thinking_mode,
user_snapshot=user_snapshot,
route_decision=route_decision,
)
if direct_result is not None:
return direct_result
is_snapshot_direct_question = str((scene_semantics or {}).get("snapshot_focus") or "none").strip().lower() != "none"
if planner.allow_fast_path and not is_snapshot_direct_question:
if planner.allow_fast_path and not prefers_external_retrieval and not is_snapshot_direct_question:
glm_semantic_result = await self._try_glm_project_semantic_answer(
clean_q=effective_q,
project_id=project_id,
@ -612,6 +793,8 @@ class AdaptiveRAGAnswerMainlineMixin:
rewrite_meta=rewrite_meta,
thinking_mode=active_thinking_mode,
answer_mode=ANSWER_MODE_FAST_PATH,
route_decision=route_decision,
query_semantics=query_semantics,
)
return await self._build_retrieval_response(
@ -644,4 +827,5 @@ class AdaptiveRAGAnswerMainlineMixin:
mixed_signal_part=mixed_signal_part,
answer_mode=planner.retrieval_answer_mode,
consultant_mode_preference=consultant_mode_preference,
route_decision=route_decision,
)

196
backend/app/services/adaptive_rag_service/mixin_answer_payload.py
View File

@ -10,10 +10,12 @@ class AdaptiveRAGAnswerPayloadMixin:
"db": 90,
"graph": 80,
"doc": 70,
"mcp_search": 68,
"url": 65,
"feishu": 60,
"system": 10,
}
_INTERNAL_SOURCE_TYPES = {"db", "graph", "doc", "pmflow", "feishu"}
@staticmethod
def _safe_source_metadata(raw: Any) -> dict[str, Any]:
@ -69,7 +71,7 @@ class AdaptiveRAGAnswerPayloadMixin:
if record_type and record_id is not None:
return f"{record_type}:{record_id}"
return record_id
if source_type in {"doc", "url"}:
if source_type in {"doc", "url", "mcp_search"}:
return self._source_ref_value(
metadata.get("doc_id")
or metadata.get("knowledge_doc_id")
@ -366,6 +368,191 @@ class AdaptiveRAGAnswerPayloadMixin:
"tool_trace": tool_trace if isinstance(tool_trace, list) else [],
}
def _is_internal_answer_contract_route(
self,
*,
route_decision: Optional[dict[str, Any]] = None,
sources: Optional[list[dict[str, Any]]] = None,
) -> bool:
normalized_route = (
self._normalize_query_route_decision(route_decision)
if hasattr(self, "_normalize_query_route_decision")
else {}
)
if str(normalized_route.get("route_mode") or "").strip().lower() == "internal":
return True
if str(normalized_route.get("effective_source_policy") or "").strip().lower() == "internal":
return True
for item in sources or []:
if not isinstance(item, dict):
continue
if str(item.get("source") or "").strip().lower() in self._INTERNAL_SOURCE_TYPES:
return True
return False
def _verify_external_answer_contract(
self,
*,
answer: str,
sources: Optional[list[dict[str, Any]]] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
normalized_route = (
self._normalize_query_route_decision(route_decision)
if hasattr(self, "_normalize_query_route_decision")
else {}
)
if str(normalized_route.get("effective_source_policy") or "").strip().lower() != "external":
return {"ok": True, "reason_code": None, "missing_fields": []}
source_rows = sources if isinstance(sources, list) else []
if not source_rows:
return {
"ok": False,
"reason_code": "missing_external_evidence",
"missing_fields": ["external_evidence"],
}
external_source_seen = False
for item in source_rows:
if not isinstance(item, dict):
continue
source = str(item.get("source") or "").strip().lower()
metadata = item.get("metadata") if isinstance(item.get("metadata"), dict) else {}
if source == "system":
continue
if source not in {"url", "mcp_search"}:
return {
"ok": False,
"reason_code": "external_source_contract_violation",
"missing_fields": ["external_source_type"],
}
target_url = str(
metadata.get("target_url")
or metadata.get("file_url")
or metadata.get("url")
or ""
).strip().lower()
if not target_url.startswith("http://") and not target_url.startswith("https://"):
return {
"ok": False,
"reason_code": "external_target_url_missing",
"missing_fields": ["target_url"],
}
external_source_seen = True
if not external_source_seen:
return {
"ok": False,
"reason_code": "missing_external_grounding",
"missing_fields": ["external_evidence"],
}
_ = answer
return {"ok": True, "reason_code": None, "missing_fields": []}
def _build_external_answer_contract_guard_payload(
self,
*,
reason_code: Optional[str],
missing_fields: Optional[list[str]] = None,
) -> dict[str, Any]:
normalized_reason = str(reason_code or "").strip().lower() or "external_answer_contract_violation"
normalized_missing = [str(item).strip() for item in (missing_fields or []) if str(item).strip()]
return {
"answer": "当前没有拿到可核验的外部实时证据，已阻止直接返回外部结论。请稍后重试，或把要核验的公开对象说得更具体一些。",
"sources": [
{
"source": "system",
"label": "External Query Contract Guard",
"score": 0.99,
"metadata": {
"mode": "guard",
"reason": normalized_reason,
},
}
],
"missing_fields": normalized_missing or ["external_answer_contract"],
"reason_code": normalized_reason,
"answer_mode": ANSWER_MODE_CLARIFY,
"grounding_status": GROUNDING_STATUS_INSUFFICIENT_EVIDENCE,
}
def _verify_internal_answer_contract(
self,
*,
query_type: str,
answer: str,
sources: Optional[list[dict[str, Any]]] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
source_rows = sources if isinstance(sources, list) else []
if not self._is_internal_answer_contract_route(route_decision=route_decision, sources=source_rows):
return {"ok": True, "reason_code": None, "missing_fields": []}
normalized_query_type = str(query_type or QUERY_TYPE_UNKNOWN).strip().lower()
normalized_answer = str(answer or "").strip()
compact_answer = re.sub(r"\s+", "", normalized_answer.lower())
if normalized_query_type == QUERY_TYPE_EXISTENCE:
if "项目总数" in normalized_answer or "总数为" in normalized_answer:
return {
"ok": False,
"reason_code": "no_total_count_substitution",
"missing_fields": ["matched_evidence", "existence_contract"],
}
if "总数" in compact_answer and "匹配到" not in normalized_answer and "未检索到" not in normalized_answer:
return {
"ok": False,
"reason_code": "no_total_count_substitution",
"missing_fields": ["matched_evidence", "existence_contract"],
}
if not source_rows:
return {
"ok": False,
"reason_code": "missing_internal_evidence",
"missing_fields": ["internal_evidence"],
}
return {"ok": True, "reason_code": None, "missing_fields": []}
def _build_internal_answer_contract_guard_payload(
self,
*,
query_type: str,
reason_code: Optional[str],
missing_fields: Optional[list[str]] = None,
) -> dict[str, Any]:
normalized_query_type = str(query_type or QUERY_TYPE_UNKNOWN).strip().lower()
normalized_reason = str(reason_code or "").strip().lower() or "internal_answer_contract_violation"
normalized_missing = [str(item).strip() for item in (missing_fields or []) if str(item).strip()]
if normalized_reason == "no_total_count_substitution" and normalized_query_type == QUERY_TYPE_EXISTENCE:
answer = "当前内部查询结果与问题类型不一致，已阻止直接返回。你问的是“有没有”，请给我更明确的对象条件后我按存在性重新查询。"
elif normalized_reason == "missing_internal_evidence":
answer = "当前内部查询证据不足，已阻止直接返回结论。请补充更明确的对象或范围后我再查。"
else:
answer = "当前内部查询结果未通过回答合同校验，已阻止直接返回。请补充更明确的对象或范围后我再查。"
return {
"answer": answer,
"sources": [
{
"source": "system",
"label": "Internal Query Contract Guard",
"score": 0.99,
"metadata": {
"mode": "guard",
"reason": normalized_reason,
"query_type": normalized_query_type,
},
}
],
"missing_fields": normalized_missing or ["internal_answer_contract"],
"reason_code": normalized_reason,
"answer_mode": ANSWER_MODE_BLOCKED,
"grounding_status": GROUNDING_STATUS_BLOCKED,
}
def _build_response_payload(
self,
*,
@ -408,6 +595,7 @@ class AdaptiveRAGAnswerPayloadMixin:
sources_used: Optional[list[dict[str, Any]]] = None,
evidence_pack: Optional[dict[str, Any]] = None,
prompt_variant_id: Optional[str] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
evidence: list[dict[str, Any]] = []
for item in sources[:8]:
@ -452,6 +640,11 @@ class AdaptiveRAGAnswerPayloadMixin:
reason_code=reason_code,
)
)
normalized_route_decision = (
self._normalize_query_route_decision(route_decision)
if hasattr(self, "_normalize_query_route_decision")
else {}
)
return {
"answer": str(answer or "").strip(),
"evidence": evidence,
@ -498,6 +691,7 @@ class AdaptiveRAGAnswerPayloadMixin:
"citations": normalized_citations,
"sources_used": normalized_sources_used,
"evidence_pack": normalized_evidence_pack,
"route_decision": normalized_route_decision,
}
@staticmethod

92
backend/app/services/adaptive_rag_service/mixin_answer_precheck.py
View File

@ -22,6 +22,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
rewrite_meta: dict[str, Any],
context_session_id: Optional[int],
active_thinking_mode: str,
route_decision: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
if not isinstance(meta_followup_payload, dict):
return None
@ -52,8 +53,16 @@ class AdaptiveRAGAnswerPrecheckMixin:
},
}
]
render_with_contract = getattr(self, "_render_answer_with_route_contract", None)
rendered_answer = answer
if callable(render_with_contract):
rendered_answer = render_with_contract(
answer=answer,
sources=direct_sources,
route_decision=route_decision,
)
runtime_fields = self._build_qa_record_runtime_fields(
answer=answer,
answer=rendered_answer,
sources=direct_sources,
missing_fields=[],
answer_mode=ANSWER_MODE_FAST_PATH,
@ -76,7 +85,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
qa = QARecord(
tenant_id=self.tenant_id,
question=clean_q,
answer=answer,
answer=rendered_answer,
sources=direct_sources,
intent="question",
domains_used=["system_context"],
@ -90,6 +99,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
anchor_kind=anchor_kind,
anchor_ref=str(meta_followup_payload.get("anchor_ref") or "").strip() or None,
routing_snapshot=routing_snapshot,
route_decision=route_decision,
),
**runtime_fields,
)
@ -98,22 +108,31 @@ class AdaptiveRAGAnswerPrecheckMixin:
session_state=state_row,
question=clean_q,
effective_question=effective_q,
answer=answer,
answer=rendered_answer,
query_type=query_type,
ambiguity="clear",
rewrite_meta=rewrite_meta,
sources=direct_sources,
)
await self.db.flush()
persist_route_memory = getattr(self, "_persist_query_route_memory", None)
if callable(persist_route_memory):
await persist_route_memory(
qa_record_id=int(qa.id) if getattr(qa, "id", None) is not None else None,
question_text=clean_q,
project_id=project_id,
route_decision=route_decision,
)
await self.db.commit()
await self.db.refresh(qa)
await self.context_cache.append_session_messages(
tenant_id=self.tenant_id,
session_id=context_session_id,
question=clean_q,
answer=answer,
answer=rendered_answer,
)
return self._build_response_payload(
answer=answer,
answer=rendered_answer,
sources=direct_sources,
domains_used=["system_context"],
response_time_ms=elapsed,
@ -141,6 +160,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
evidence_pack=runtime_fields["evidence_pack"],
tool_trace=runtime_fields["tool_trace"],
prompt_variant_id="meta_followup",
route_decision=route_decision,
)
async def _maybe_return_clarification_response(
@ -160,6 +180,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
rewrite_meta: dict[str, Any],
context_session_id: Optional[int],
active_thinking_mode: str,
route_decision: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
if not isinstance(clarification_payload, dict):
return None
@ -191,8 +212,16 @@ class AdaptiveRAGAnswerPrecheckMixin:
"metadata": {"mode": "clarification", "reason": clarification_code},
}
]
render_with_contract = getattr(self, "_render_answer_with_route_contract", None)
rendered_answer = clarification_answer
if callable(render_with_contract):
rendered_answer = render_with_contract(
answer=clarification_answer,
sources=clarify_sources,
route_decision=route_decision,
)
runtime_fields = self._build_qa_record_runtime_fields(
answer=clarification_answer,
answer=rendered_answer,
sources=clarify_sources,
missing_fields=missing_fields,
answer_mode=ANSWER_MODE_CLARIFY,
@ -214,7 +243,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
qa = QARecord(
tenant_id=self.tenant_id,
question=clean_q,
answer=clarification_answer,
answer=rendered_answer,
sources=clarify_sources,
intent="question",
domains_used=["system_context"],
@ -224,7 +253,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
project_id=project_id,
session_id=active_session_id,
turn_index=turn_index,
**self._build_qa_record_external_fields(),
**self._build_qa_record_external_fields(route_decision=route_decision),
**runtime_fields,
)
self.db.add(qa)
@ -244,22 +273,31 @@ class AdaptiveRAGAnswerPrecheckMixin:
session_state=state_row,
question=clean_q,
effective_question=effective_q,
answer=clarification_answer,
answer=rendered_answer,
query_type=query_type,
ambiguity="needs_clarification",
rewrite_meta=rewrite_meta,
sources=clarify_sources,
)
await self.db.flush()
persist_route_memory = getattr(self, "_persist_query_route_memory", None)
if callable(persist_route_memory):
await persist_route_memory(
qa_record_id=int(qa.id) if getattr(qa, "id", None) is not None else None,
question_text=clean_q,
project_id=project_id,
route_decision=route_decision,
)
await self.db.commit()
await self.db.refresh(qa)
await self.context_cache.append_session_messages(
tenant_id=self.tenant_id,
session_id=context_session_id,
question=clean_q,
answer=clarification_answer,
answer=rendered_answer,
)
return self._build_response_payload(
answer=clarification_answer,
answer=rendered_answer,
sources=clarify_sources,
domains_used=["system_context"],
response_time_ms=elapsed,
@ -286,6 +324,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
evidence_pack=runtime_fields["evidence_pack"],
tool_trace=runtime_fields["tool_trace"],
prompt_variant_id="clarification",
route_decision=route_decision,
)
async def _maybe_return_snapshot_direct_response(
@ -305,6 +344,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
context_session_id: Optional[int],
active_thinking_mode: str,
user_snapshot: Optional[dict[str, Any]],
route_decision: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
if not direct_answer:
return None
@ -327,8 +367,16 @@ class AdaptiveRAGAnswerPrecheckMixin:
"metadata": {"mode": "snapshot_direct", "topic": "current_user"},
}
)
render_with_contract = getattr(self, "_render_answer_with_route_contract", None)
rendered_answer = direct_answer
if callable(render_with_contract):
rendered_answer = render_with_contract(
answer=direct_answer,
sources=direct_sources,
route_decision=route_decision,
)
runtime_fields = self._build_qa_record_runtime_fields(
answer=direct_answer,
answer=rendered_answer,
sources=direct_sources,
missing_fields=[],
answer_mode=ANSWER_MODE_FAST_PATH,
@ -347,7 +395,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
qa = QARecord(
tenant_id=self.tenant_id,
question=clean_q,
answer=direct_answer,
answer=rendered_answer,
sources=direct_sources,
intent="question",
domains_used=["system_context"],
@ -357,7 +405,7 @@ class AdaptiveRAGAnswerPrecheckMixin:
project_id=project_id,
session_id=int(session_id) if session_id is not None else (int(context_session_id) if context_session_id is not None else None),
turn_index=turn_index,
**self._build_qa_record_external_fields(),
**self._build_qa_record_external_fields(route_decision=route_decision),
**runtime_fields,
)
self.db.add(qa)
@ -365,22 +413,31 @@ class AdaptiveRAGAnswerPrecheckMixin:
session_state=state_row,
question=clean_q,
effective_question=effective_q,
answer=direct_answer,
answer=rendered_answer,
query_type=QUERY_TYPE_EXPLAIN,
ambiguity="clear",
rewrite_meta=rewrite_meta,
sources=direct_sources,
)
await self.db.flush()
persist_route_memory = getattr(self, "_persist_query_route_memory", None)
if callable(persist_route_memory):
await persist_route_memory(
qa_record_id=int(qa.id) if getattr(qa, "id", None) is not None else None,
question_text=clean_q,
project_id=project_id,
route_decision=route_decision,
)
await self.db.commit()
await self.db.refresh(qa)
await self.context_cache.append_session_messages(
tenant_id=self.tenant_id,
session_id=context_session_id,
question=clean_q,
answer=direct_answer,
answer=rendered_answer,
)
return self._build_response_payload(
answer=direct_answer,
answer=rendered_answer,
sources=direct_sources,
domains_used=["system_context"],
response_time_ms=elapsed,
@ -404,4 +461,5 @@ class AdaptiveRAGAnswerPrecheckMixin:
evidence_pack=runtime_fields["evidence_pack"],
tool_trace=runtime_fields["tool_trace"],
prompt_variant_id="snapshot_direct",
route_decision=route_decision,
)

407
backend/app/services/adaptive_rag_service/mixin_answer_retrieval.py
View File

@ -5,6 +5,237 @@ from .shared import *
class AdaptiveRAGAnswerRetrievalMixin:
async def _execute_external_mcp_search(self, plan) -> dict[str, Any]:
from ..external_query_service.schemas import ExternalAdapterResult, ExternalQueryPlan
from ..mcp_config import McpConfigService
try:
normalized_plan = plan if isinstance(plan, ExternalQueryPlan) else ExternalQueryPlan.model_validate(plan)
except Exception as exc:
return ExternalAdapterResult(
adapter_name="mcp",
status="error",
evidence=[],
meta={"reason": "plan_invalid", "error": str(exc)},
).model_dump(mode="json")
query = str((normalized_plan.search_queries[:1] or [normalized_plan.topic])[0] or "").strip()
if not query:
return ExternalAdapterResult(
adapter_name="mcp",
status="empty",
evidence=[],
meta={"reason": "empty_query"},
).model_dump(mode="json")
db = getattr(self, "db", None)
if db is None:
return ExternalAdapterResult(
adapter_name="mcp",
status="unavailable",
evidence=[],
meta={"reason": "db_unavailable"},
).model_dump(mode="json")
try:
service = McpConfigService(db, tenant_id=int(getattr(self, "tenant_id", 1) or 1))
evidence = await service.search_external(query=query, result_limit=normalized_plan.result_limit)
except Exception as exc:
return ExternalAdapterResult(
adapter_name="mcp",
status="error",
evidence=[],
meta={"reason": "backend_direct_runtime_error", "error": str(exc), "query": query},
).model_dump(mode="json")
if not evidence:
return ExternalAdapterResult(
adapter_name="mcp",
status="empty",
evidence=[],
meta={"reason": "no_backend_direct_result", "query": query},
).model_dump(mode="json")
return ExternalAdapterResult(
adapter_name="mcp",
status="success",
evidence=evidence,
meta={"query": query, "provider": "mcp_backend_direct"},
).model_dump(mode="json")
async def _execute_external_query_gateway(
self,
*,
question: str,
query_semantics: Optional[dict[str, Any]] = None,
result_limit: int = 3,
max_pages: int = 3,
) -> tuple[list[RetrievedItem], dict[str, Any]]:
live_search_ready = all(
callable(item)
for item in (
getattr(self, "_search_live_external_results", None),
getattr(self, "_fetch_url_text", None),
getattr(self, "_chunk_text", None),
getattr(self, "_score_live_external_search_result", None),
getattr(self, "_score_live_external_chunk", None),
)
)
mcp_ready = callable(getattr(self, "_execute_external_mcp_search", None)) and getattr(self, "db", None) is not None
if not live_search_ready and not mcp_ready:
legacy_retrieval = getattr(self, "_retrieve_live_external_context", None)
if callable(legacy_retrieval):
rows, meta = await legacy_retrieval(
question=question,
max_results=result_limit,
max_pages=max_pages,
)
normalized_meta = dict(meta or {})
normalized_meta.setdefault("mode", "live_external_context" if rows else "live_external_context_empty")
normalized_meta.setdefault("domains_used", ["url"])
normalized_meta.setdefault("db_query_ms", 0)
normalized_meta.setdefault("vector_search_ms", 0)
normalized_meta.setdefault("graph_query_ms", 0)
normalized_meta.setdefault("db_timeout", False)
normalized_meta.setdefault("vector_timeout", False)
normalized_meta.setdefault("graph_timeout", False)
normalized_meta.setdefault("tool_trace", [])
return rows, normalized_meta
from ..external_query_service import ExternalQueryGateway, ExternalQueryPlanner, ExternalProviderRegistry
from ..external_query_service.adapters import LiveSearchFallbackAdapter, MCPExternalAdapter
planner = ExternalQueryPlanner(
llm_service=getattr(self, "llm", None),
db=getattr(self, "db", None),
tenant_id=getattr(self, "tenant_id", 1),
query_builder=getattr(self, "_build_live_external_search_queries", None),
)
registry = ExternalProviderRegistry(
adapters=[
MCPExternalAdapter(
search_callable=getattr(self, "_execute_external_mcp_search", None),
fetch_text_callable=getattr(self, "_fetch_url_text", None),
chunk_text_callable=getattr(self, "_chunk_text", None),
score_chunk_callable=getattr(self, "_score_live_external_chunk", None),
max_pages=max_pages,
),
LiveSearchFallbackAdapter(
build_queries_callable=getattr(self, "_build_live_external_search_queries", None),
search_results_callable=getattr(self, "_search_live_external_results", None),
fetch_text_callable=getattr(self, "_fetch_url_text", None),
chunk_text_callable=getattr(self, "_chunk_text", None),
score_search_result_callable=getattr(self, "_score_live_external_search_result", None),
score_chunk_callable=getattr(self, "_score_live_external_chunk", None),
max_pages=max_pages,
),
]
)
execution = await ExternalQueryGateway(
planner=planner,
registry=registry,
).execute(
question=question,
query_semantics=query_semantics,
)
verifier = dict(execution.verifier or {})
if not bool(verifier.get("ok")):
return [], {
"mode": "external_query_gateway_verifier_blocked",
"adapter": execution.adapter_name,
"verifier": verifier,
"planner": dict(execution.planner_meta or {}),
"fallback_used": bool((execution.meta or {}).get("fallback_used")),
"adapter_attempts": list((execution.meta or {}).get("adapter_attempts") or []),
"query": str(((execution.meta or {}).get("query") or (execution.plan.search_queries[:1] or [""])[0])).strip(),
"search_result_count": int((execution.meta or {}).get("search_result_count") or 0),
"fetched_pages": int((execution.meta or {}).get("fetched_pages") or 0),
"tool_trace": [
{
"step": 1,
"tool": "external_query_gateway",
"adapter": execution.adapter_name,
"planner": str((execution.planner_meta or {}).get("planner") or "unknown"),
"verifier_ok": False,
}
],
"domains_used": ["url"],
"db_query_ms": 0,
"vector_search_ms": 0,
"graph_query_ms": 0,
"db_timeout": False,
"vector_timeout": False,
"graph_timeout": False,
}
rows: list[RetrievedItem] = []
for item in list(execution.evidence or [])[: max(1, int(result_limit))]:
source_type = str(getattr(item, "source_type", "url") or "url").strip().lower() or "url"
normalized_source = source_type if source_type in {"url", "mcp_search"} else "url"
title = str(getattr(item, "title", "") or "").strip()
source_label = str(getattr(item, "source_label", "") or "").strip() or title
target_url = str(getattr(item, "target_url", "") or "").strip()
snippet = str(getattr(item, "snippet", "") or "").strip()
freshness_score = float(getattr(item, "freshness_score", 0.0) or 0.0)
authority_score = float(getattr(item, "authority_score", 0.0) or 0.0)
rows.append(
RetrievedItem(
source=normalized_source,
content=(snippet or title or target_url)[:1800],
score=round(max(freshness_score, authority_score, 0.55), 4),
source_label=(
f"MCP[{source_label[:120]}]"
if normalized_source == "mcp_search"
else f"URL[{source_label[:120]}]"
),
metadata={
"title": title[:255],
"url": target_url,
"file_url": target_url,
"target_url": target_url,
"retrieved_via": "external_query_gateway",
"external_source_type": source_type,
"external_adapter": execution.adapter_name,
"freshness_score": round(freshness_score, 4),
"authority_score": round(authority_score, 4),
"grounded_fields": list(getattr(item, "grounded_fields", []) or []),
**(dict(getattr(item, "raw_metadata", {}) or {})),
},
)
)
meta = dict(execution.meta or {})
meta.update(
{
"mode": "external_query_gateway" if rows else "external_query_gateway_empty",
"adapter": execution.adapter_name,
"verifier": verifier,
"planner": dict(execution.planner_meta or {}),
"query": str(meta.get("query") or (execution.plan.search_queries[:1] or [""])[0]).strip(),
"search_result_count": int(meta.get("search_result_count") or 0),
"fetched_pages": int(meta.get("fetched_pages") or 0),
"domains_used": ["url"],
"db_query_ms": 0,
"vector_search_ms": 0,
"graph_query_ms": 0,
"db_timeout": False,
"vector_timeout": False,
"graph_timeout": False,
"tool_trace": [
{
"step": 1,
"tool": "external_query_gateway",
"adapter": execution.adapter_name,
"planner": str((execution.planner_meta or {}).get("planner") or "unknown"),
"verifier_ok": True,
"result_count": len(rows),
"fallback_used": bool(meta.get("fallback_used")),
}
],
}
)
return rows, meta
async def _maybe_fetch_live_external_freshness_context(
self,
*,
@ -19,16 +250,30 @@ class AdaptiveRAGAnswerRetrievalMixin:
search_question = str(effective_question or question or "").strip()
if not search_question:
return [], {"mode": "skip_empty_question"}
execute_gateway = getattr(self, "_execute_external_query_gateway", None)
try:
rows, live_meta = await self._retrieve_live_external_context(
question=search_question,
max_results=3,
max_pages=3,
)
if callable(execute_gateway):
rows, live_meta = await execute_gateway(
question=search_question,
query_semantics=query_semantics,
result_limit=3,
max_pages=3,
)
else:
rows, live_meta = await self._retrieve_live_external_context(
question=search_question,
max_results=3,
max_pages=3,
)
except Exception as exc:
return [], {"mode": "live_external_context_error", "error": str(exc)}
mode = "external_query_gateway_error" if callable(execute_gateway) else "live_external_context_error"
return [], {"mode": mode, "error": str(exc)}
meta = dict(live_meta or {})
meta["mode"] = "live_external_context" if rows else "live_external_context_empty"
if not str(meta.get("mode") or "").strip():
if callable(execute_gateway):
meta["mode"] = "external_query_gateway" if rows else "external_query_gateway_empty"
else:
meta["mode"] = "live_external_context" if rows else "live_external_context_empty"
meta["external_freshness_only"] = bool(
isinstance(query_semantics, dict) and query_semantics.get("external_freshness_only")
)
@ -66,22 +311,43 @@ class AdaptiveRAGAnswerRetrievalMixin:
mixed_signal_part: Optional[str] = None,
answer_mode: str = ANSWER_MODE_RETRIEVAL_NORMAL,
consultant_mode_preference: Optional[str] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
domains = [str(item).strip().lower() for item in retrieval_domains if str(item or "").strip()]
if not domains:
domains = ["db", "graph", "doc", "feishu", "pmflow", "url"]
if str(asked_via or "").strip().lower() == "feishu" and source_chat_id and "feishu" not in domains:
domains.append("feishu")
retrieved, search_meta = await self._multi_step_search(
question=effective_q,
domains=domains,
project_id=project_id,
source_chat_id=source_chat_id,
query_semantics=query_semantics,
session_state=state_row,
max_steps_override=deep_retrieval_steps if deep_thinking else None,
vector_top_k=deep_vector_top_k if deep_thinking else None,
normalize_route_decision = getattr(self, "_normalize_query_route_decision", None)
normalized_route_decision = (
normalize_route_decision(route_decision)
if callable(normalize_route_decision)
else (dict(route_decision or {}) if isinstance(route_decision, dict) else {})
)
external_route_only = str(
normalized_route_decision.get("effective_source_policy")
or query_semantics.get("effective_source_policy")
or query_semantics.get("source_policy")
or ""
).strip().lower() == "external"
if external_route_only:
retrieved, search_meta = await self._execute_external_query_gateway(
question=effective_q,
query_semantics=query_semantics,
result_limit=3,
max_pages=3,
)
else:
retrieved, search_meta = await self._multi_step_search(
question=effective_q,
domains=domains,
project_id=project_id,
source_chat_id=source_chat_id,
query_semantics=query_semantics,
session_state=state_row,
max_steps_override=deep_retrieval_steps if deep_thinking else None,
vector_top_k=deep_vector_top_k if deep_thinking else None,
)
db_query_ms = int(search_meta.get("db_query_ms") or 0)
vector_search_ms = int(search_meta.get("vector_search_ms") or 0)
graph_query_ms = int(search_meta.get("graph_query_ms") or 0)
@ -101,6 +367,16 @@ class AdaptiveRAGAnswerRetrievalMixin:
),
},
)
tool_trace.insert(
0,
{
"step": 0,
"tool": "query_route_gateway",
"route_mode": normalized_route_decision.get("route_mode"),
"effective_source_policy": normalized_route_decision.get("effective_source_policy"),
"decision_origin": normalized_route_decision.get("decision_origin"),
},
)
final_domains = [str(x) for x in (search_meta.get("domains_used") or domains)]
prompt_variant_id = self._peek_answer_prompt_variant()
@ -155,7 +431,7 @@ class AdaptiveRAGAnswerRetrievalMixin:
auto_retry_reason = ""
if not deep_thinking:
auto_retry_reason = str(pre_retry_guard_semantics.get("auto_retry_reason") or "").strip()
if auto_retry_reason:
if auto_retry_reason and not external_route_only:
deep_retrieved, deep_meta = await self._multi_step_search(
question=effective_q,
domains=domains,
@ -426,6 +702,92 @@ class AdaptiveRAGAnswerRetrievalMixin:
"vector_timeout": bool(vector_timeout),
"graph_timeout": bool(graph_timeout),
}
filter_sources_for_route = getattr(self, "_filter_sources_for_route", None)
if callable(filter_sources_for_route):
used_sources = filter_sources_for_route(
sources=used_sources,
route_decision=normalized_route_decision,
)
render_with_contract = getattr(self, "_render_answer_with_route_contract", None)
if callable(render_with_contract):
answer_text = render_with_contract(
answer=answer_text,
sources=used_sources,
route_decision=normalized_route_decision,
)
effective_source_policy = str(normalized_route_decision.get("effective_source_policy") or "").strip().lower()
contract_verdict = {"ok": True, "reason_code": None, "missing_fields": []}
guard_payload: Optional[dict[str, Any]] = None
if effective_source_policy == "external" and answer_mode != ANSWER_MODE_CLARIFY:
contract_verdict = self._verify_external_answer_contract(
answer=answer_text,
sources=used_sources,
route_decision=normalized_route_decision,
)
if not bool(contract_verdict.get("ok")):
guard_payload = self._build_external_answer_contract_guard_payload(
reason_code=str(contract_verdict.get("reason_code") or ""),
missing_fields=list(contract_verdict.get("missing_fields") or []),
)
else:
contract_verdict = self._verify_internal_answer_contract(
query_type=query_type,
answer=answer_text,
sources=used_sources,
route_decision=normalized_route_decision,
)
if not bool(contract_verdict.get("ok")):
guard_payload = self._build_internal_answer_contract_guard_payload(
query_type=query_type,
reason_code=str(contract_verdict.get("reason_code") or ""),
missing_fields=list(contract_verdict.get("missing_fields") or []),
)
if guard_payload:
answer_text = str(guard_payload.get("answer") or answer_text)
used_sources = list(guard_payload.get("sources") or [])
answer_mode = str(guard_payload.get("answer_mode") or answer_mode or ANSWER_MODE_BLOCKED)
max_score = min(float(max_score or 0.0), 0.35)
if "system_context" not in final_domains:
final_domains.append("system_context")
tool_trace.append(
{
"step": 100,
"tool": (
"external_answer_contract_guard"
if effective_source_policy == "external"
else "internal_answer_contract_guard"
),
"reason": str(
guard_payload.get("reason_code")
or (
"external_answer_contract_violation"
if effective_source_policy == "external"
else "internal_answer_contract_violation"
)
),
}
)
grounding_verdict = GroundingVerdict(
status=str(
guard_payload.get("grounding_status")
or (GROUNDING_STATUS_INSUFFICIENT_EVIDENCE if effective_source_policy == "external" else GROUNDING_STATUS_BLOCKED)
),
missing_fields=list(guard_payload.get("missing_fields") or []),
reason_code=str(
guard_payload.get("reason_code")
or (
"external_answer_contract_violation"
if effective_source_policy == "external"
else "internal_answer_contract_violation"
)
),
)
inject_runtime_summaries = getattr(self, "_inject_internal_query_runtime_summaries", None)
if callable(inject_runtime_summaries):
retrieval_metrics = inject_runtime_summaries(
retrieval_metrics=retrieval_metrics,
query_semantics=query_semantics,
)
runtime_fields = self._build_qa_record_runtime_fields(
answer=answer_text,
sources=used_sources,
@ -455,6 +817,7 @@ class AdaptiveRAGAnswerRetrievalMixin:
anchor_kind=None,
anchor_ref=None,
routing_snapshot=routing_snapshot,
route_decision=normalized_route_decision,
),
**runtime_fields,
)
@ -473,6 +836,15 @@ class AdaptiveRAGAnswerRetrievalMixin:
rewrite_meta=rewrite_meta,
sources=used_sources,
)
await self.db.flush()
persist_route_memory = getattr(self, "_persist_query_route_memory", None)
if callable(persist_route_memory):
await persist_route_memory(
qa_record_id=int(qa.id) if getattr(qa, "id", None) is not None else None,
question_text=clean_q,
project_id=project_id,
route_decision=normalized_route_decision,
)
await self.db.commit()
await self.db.refresh(qa)
await self.context_cache.append_session_messages(
@ -510,4 +882,5 @@ class AdaptiveRAGAnswerRetrievalMixin:
sources_used=runtime_fields["sources_used"],
evidence_pack=runtime_fields["evidence_pack"],
prompt_variant_id=prompt_variant_id,
route_decision=normalized_route_decision,
)

204
backend/app/services/adaptive_rag_service/mixin_core_intent.py
View File

@ -8,16 +8,22 @@ from ..ai_pmo.workflow_semantic_planner import WorkflowSemanticPlanner
class AdaptiveRAGCoreIntentMixin:
_SEMANTIC_INTENTS = {"question", "statement", "mixed"}
_SEMANTIC_QUERY_TYPES = {
QUERY_TYPE_EXISTENCE,
QUERY_TYPE_COUNT,
QUERY_TYPE_LIST,
QUERY_TYPE_STATUS,
QUERY_TYPE_EXPLAIN,
QUERY_TYPE_DIAGNOSE,
QUERY_TYPE_COMPARE,
QUERY_TYPE_UNKNOWN,
}
_SEMANTIC_RETRIEVAL_DOMAINS = ("db", "graph", "doc", "feishu", "pmflow", "url")
_INTERNAL_RETRIEVAL_DOMAINS = ("db", "graph", "doc", "feishu", "pmflow")
_EXTERNAL_RETRIEVAL_DOMAINS = ("url",)
_SEMANTIC_SUBJECT_FOCI = {"people", "project", "task", "signal", "charter", "workload", "feature", "unknown"}
_ALLOWED_SCOPE_VALUES = {"none", "global", "project"}
_SEMANTIC_FEATURE_FOCI = {"none", "purpose", "entry", "catalog"}
_SOURCE_POLICIES = {"internal", "external", "uncertain"}
def __init__(self, db: AsyncSession, *, tenant_id: int) -> None:
self.db = db
@ -118,8 +124,43 @@ class AdaptiveRAGCoreIntentMixin:
"routing_decision": routing_decision or None,
"routing_config_version": routing_config_version or None,
}
query_route = payload.get("query_route") if isinstance(payload.get("query_route"), dict) else {}
if query_route:
normalized["query_route"] = {
"route_mode": str(query_route.get("route_mode") or "").strip().lower() or None,
"effective_source_policy": str(query_route.get("effective_source_policy") or "").strip().lower() or None,
"decision_origin": str(query_route.get("decision_origin") or "").strip() or None,
"reply_contract": (
dict(query_route.get("reply_contract") or {})
if isinstance(query_route.get("reply_contract"), dict)
else {}
),
"query_profile": (
dict(query_route.get("query_profile") or {})
if isinstance(query_route.get("query_profile"), dict)
else {}
),
"probe_snapshot_summary": (
dict(query_route.get("probe_snapshot_summary") or {})
if isinstance(query_route.get("probe_snapshot_summary"), dict)
else {}
),
}
return {key: value for key, value in normalized.items() if value is not None}
def _merge_query_route_audit_into_routing_snapshot(
self,
*,
routing_snapshot: Optional[dict[str, Any]],
route_decision: Optional[dict[str, Any]],
) -> dict[str, Any]:
normalized_snapshot = self._normalize_routing_snapshot(routing_snapshot)
if hasattr(self, "_build_query_route_audit_snapshot"):
query_route = self._build_query_route_audit_snapshot(route_decision=route_decision)
if query_route:
normalized_snapshot["query_route"] = query_route
return normalized_snapshot
def _peek_last_answer_routing_snapshot(self) -> dict[str, Any]:
payload = getattr(self.llm, "_last_ai_call_summary", None)
if not isinstance(payload, dict):
@ -137,9 +178,13 @@ class AdaptiveRAGCoreIntentMixin:
anchor_kind: Optional[str] = None,
anchor_ref: Optional[str] = None,
routing_snapshot: Optional[dict[str, Any]] = None,
route_decision: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
context = self._get_active_external_qa_context()
normalized_snapshot = self._normalize_routing_snapshot(routing_snapshot) or None
normalized_snapshot = self._merge_query_route_audit_into_routing_snapshot(
routing_snapshot=routing_snapshot,
route_decision=route_decision,
) or None
return {
"source_message_id": context.get("source_message_id"),
"source_chat_id": context.get("source_chat_id"),
@ -270,6 +315,15 @@ class AdaptiveRAGCoreIntentMixin:
except Exception:
return float(default)
@staticmethod
def _safe_int_value(value: Any) -> Optional[int]:
try:
if value is None:
return None
return int(value)
except Exception:
return None
@staticmethod
def _parse_json_object_safe(raw: Any) -> dict[str, Any]:
if isinstance(raw, dict):
@ -323,12 +377,94 @@ class AdaptiveRAGCoreIntentMixin:
return normalized
return "none"
@classmethod
def _normalize_source_policy(cls, value: Any) -> str:
normalized = str(value or "").strip().lower()
if normalized in cls._SOURCE_POLICIES:
return normalized
return "uncertain"
@classmethod
def _normalize_query_profile_object(cls, value: Any) -> str:
normalized = re.sub(r"[^a-z0-9:_\-]", "", str(value or "").strip().lower())
return normalized[:48]
@classmethod
def _normalize_query_profile(cls, payload: Any) -> dict[str, Any]:
raw = payload if isinstance(payload, dict) else {}
referenced_objects: list[str] = []
for item in list(raw.get("referenced_objects") or []):
normalized = cls._normalize_query_profile_object(item)
if not normalized:
continue
if normalized not in referenced_objects:
referenced_objects.append(normalized)
if not referenced_objects:
referenced_objects = ["unknown"]
try:
confidence = max(0.0, min(1.0, float(raw.get("confidence") or 0.0)))
except Exception:
confidence = 0.0
return {
"needs_private_context": bool(raw.get("needs_private_context")),
"needs_freshness": bool(raw.get("needs_freshness")),
"referenced_objects": referenced_objects,
"confidence": confidence,
}
@staticmethod
def _normalize_semantic_object_mentions(payload: Any) -> list[str]:
rows = payload if isinstance(payload, list) else []
mentions: list[str] = []
for item in rows:
text = str(item or "").strip(" ，。,.!?？：:;；“”\"'()（）")
if len(text) >= 2 and text not in mentions:
mentions.append(text[:32])
return mentions[:4]
@classmethod
def _apply_source_policy_to_domains(
cls,
*,
domains: list[str],
effective_source_policy: str,
) -> list[str]:
normalized_policy = cls._normalize_source_policy(effective_source_policy)
if normalized_policy == "internal":
internal_domains = [item for item in domains if item in cls._INTERNAL_RETRIEVAL_DOMAINS]
if internal_domains:
return internal_domains
return list(cls._INTERNAL_RETRIEVAL_DOMAINS)
external_domains = [item for item in domains if item in cls._EXTERNAL_RETRIEVAL_DOMAINS]
if external_domains:
return external_domains
return list(cls._EXTERNAL_RETRIEVAL_DOMAINS)
@classmethod
def _should_normalize_to_external_contract(
cls,
*,
source_policy: str,
effective_source_policy: str,
parsed_domains: list[str],
query_profile: dict[str, Any],
) -> bool:
if source_policy != "uncertain" or effective_source_policy != "uncertain":
return False
normalized_domains = [str(item or "").strip().lower() for item in list(parsed_domains or []) if str(item or "").strip()]
if not normalized_domains:
return False
if any(item not in cls._EXTERNAL_RETRIEVAL_DOMAINS for item in normalized_domains):
return False
return bool(query_profile.get("needs_freshness")) and not bool(query_profile.get("needs_private_context"))
async def _resolve_query_semantics(
self,
*,
message: str,
source_system: str,
metadata: Optional[dict[str, Any]] = None,
source_policy_hint: Optional[str] = None,
) -> dict[str, Any]:
text = str(message or "").strip()
default_domains = list(self._SEMANTIC_RETRIEVAL_DOMAINS)
@ -339,7 +475,14 @@ class AdaptiveRAGCoreIntentMixin:
"question_part": "",
"statement_part": "",
"query_type": QUERY_TYPE_UNKNOWN,
"object_mentions": [],
"retrieval_domains": default_domains,
"source_policy": "uncertain",
"effective_source_policy": "uncertain",
"source_reason": "empty_message",
"decision_origin": "empty_message",
"fallback_taken": False,
"query_profile": self._normalize_query_profile(None),
"confidence": 1.0,
}
@ -402,6 +545,50 @@ class AdaptiveRAGCoreIntentMixin:
query_type = str(parsed.get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower()
if query_type not in self._SEMANTIC_QUERY_TYPES:
query_type = QUERY_TYPE_UNKNOWN
object_mentions = self._normalize_semantic_object_mentions(parsed.get("object_mentions"))
subject_focus = self._normalize_semantic_subject_focus(parsed.get("subject_focus"))
scope_hint = self._normalize_semantic_scope_hint(parsed.get("scope_hint"))
feature_focus = self._normalize_semantic_feature_focus(parsed.get("feature_focus"))
parsed_domains = self._normalize_semantic_domains(parsed.get("retrieval_domains"))
raw_source_policy = self._normalize_source_policy(parsed.get("source_policy"))
raw_effective_source_policy = self._normalize_source_policy(parsed.get("effective_source_policy"))
query_profile = self._normalize_query_profile(parsed.get("query_profile"))
normalized_hint = self._normalize_source_policy(source_policy_hint)
fallback_taken = False
if normalized_hint in {"internal", "external"}:
source_policy = normalized_hint
effective_source_policy = normalized_hint
source_reason = "upstream_source_policy_hint"
decision_origin = "upstream_hint"
fallback_taken = raw_source_policy not in {"uncertain", normalized_hint}
else:
source_policy = raw_source_policy
if raw_effective_source_policy in {"internal", "external"}:
effective_source_policy = raw_effective_source_policy
elif source_policy in {"internal", "external"}:
effective_source_policy = source_policy
else:
effective_source_policy = "uncertain"
source_reason = "query_semantics_contract"
decision_origin = "query_semantics_contract"
if self._should_normalize_to_external_contract(
source_policy=source_policy,
effective_source_policy=effective_source_policy,
parsed_domains=parsed_domains,
query_profile=query_profile,
):
source_policy = "external"
effective_source_policy = "external"
source_reason = "query_semantics_url_only_external_contract"
decision_origin = "query_semantics_contract_normalized"
if effective_source_policy == "uncertain":
retrieval_domains = list(parsed_domains)
else:
retrieval_domains = self._apply_source_policy_to_domains(
domains=parsed_domains,
effective_source_policy=effective_source_policy,
)
confidence = self._safe_float(parsed.get("confidence"), planner_confidence)
normalized_message_intent = planner_message_intent or intent
@ -411,10 +598,17 @@ class AdaptiveRAGCoreIntentMixin:
"question_part": question_part,
"statement_part": statement_part,
"query_type": query_type,
"retrieval_domains": self._normalize_semantic_domains(parsed.get("retrieval_domains")),
"subject_focus": self._normalize_semantic_subject_focus(parsed.get("subject_focus")),
"scope_hint": self._normalize_semantic_scope_hint(parsed.get("scope_hint")),
"feature_focus": self._normalize_semantic_feature_focus(parsed.get("feature_focus")),
"object_mentions": object_mentions,
"retrieval_domains": retrieval_domains,
"subject_focus": subject_focus,
"scope_hint": scope_hint,
"feature_focus": feature_focus,
"query_profile": query_profile,
"source_policy": source_policy,
"effective_source_policy": effective_source_policy,
"source_reason": source_reason,
"decision_origin": decision_origin,
"fallback_taken": fallback_taken,
"confidence": max(0.0, min(1.0, confidence)),
}

345
backend/app/services/adaptive_rag_service/mixin_deep_guard_answer.py
View File

@ -29,81 +29,7 @@ class AdaptiveRAGDeepGuardAnswerMixin:
"engine_process_query",
"runtime_state_query",
}
_PROJECT_CONTEXT_POSITIVE_MARKERS = (
"这个项目",
"当前项目",
"本项目",
"我们项目",
"项目所在行业",
"对项目的影响",
"对这个项目的影响",
"对当前项目的影响",
"对我们项目的影响",
"结合项目",
"结合当前项目",
"落到项目",
)
_PROJECT_CONTEXT_NEGATIVE_MARKERS = (
"不要提项目",
"别提项目",
"不要结合项目",
"不用结合项目",
"不需要结合项目",
"先别结合项目",
"不要绑定项目",
"别绑定项目",
)
_EXTERNAL_FACT_FRESHNESS_MARKERS = (
"最新",
"目前",
"当前",
"截至现在",
"截至当前",
"截止当前",
"今天",
"现在",
"最近",
"出到什么",
"发布了吗",
"发布到",
"版本",
"最高模型",
)
_MODEL_ENTITY_MARKERS = (
"codex",
"gpt",
"chatgpt",
"claude",
"deepseek",
"qwen",
"gemini",
"copilot",
"grok",
"kimi",
"llama",
"openai",
"anthropic",
)
_INTERNAL_MODEL_CONFIG_MARKERS = (
"ai-config",
"/ai-config",
"模型配置",
"配置项",
"配置值",
"路由规则",
"初始化参数",
"llm服务",
"llm 服务",
"运行时",
"当前系统",
"我们系统",
"pmflow",
"本系统",
"接入的模型",
"接的是",
"用的是哪个模型配置",
)
_EXTERNAL_MODEL_TOKEN_RE = re.compile(
_MODEL_TOKEN_RE = re.compile(
r"\b(?:GPT|gpt)[- ]?\d(?:\.\d)?(?:[- ]?(?:codex|turbo|mini|nano))?\b",
flags=re.IGNORECASE,
)
@ -115,46 +41,6 @@ class AdaptiveRAGDeepGuardAnswerMixin:
return text
return None
@classmethod
def _question_explicitly_requests_project_context(
cls,
*,
question: str,
project_id: Optional[int],
) -> bool:
if project_id is None:
return False
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
if any(marker in normalized for marker in cls._PROJECT_CONTEXT_NEGATIVE_MARKERS):
return False
return any(marker in normalized for marker in cls._PROJECT_CONTEXT_POSITIVE_MARKERS)
@classmethod
def _looks_like_external_freshness_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
return any(marker in normalized for marker in cls._EXTERNAL_FACT_FRESHNESS_MARKERS)
@classmethod
def _looks_like_internal_model_config_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
return any(marker in normalized for marker in cls._INTERNAL_MODEL_CONFIG_MARKERS)
@classmethod
def _looks_like_model_fact_question(cls, question: str) -> bool:
normalized = re.sub(r"\s+", "", str(question or "")).strip().lower()
if not normalized:
return False
has_entity = any(marker in normalized for marker in cls._MODEL_ENTITY_MARKERS)
if not has_entity:
return False
return "模型" in normalized or cls._looks_like_external_freshness_question(question)
@classmethod
def _normalize_external_model_evidence_text(cls, text: str) -> str:
normalized = str(text or "")
@ -175,7 +61,7 @@ class AdaptiveRAGDeepGuardAnswerMixin:
search_windows = [latest_match.group(1)] if latest_match else []
search_windows.append(normalized)
for window in search_windows:
match = cls._EXTERNAL_MODEL_TOKEN_RE.search(str(window or ""))
match = cls._MODEL_TOKEN_RE.search(str(window or ""))
if not match:
continue
token = re.sub(r"\s+", "-", str(match.group(0) or "").strip())
@ -191,8 +77,7 @@ class AdaptiveRAGDeepGuardAnswerMixin:
retrieved: list[RetrievedItem],
scene_semantics: Optional[dict[str, Any]] = None,
) -> Optional[str]:
if not cls._looks_like_model_fact_question(question) or not cls._looks_like_external_freshness_question(question):
return None
_ = question
normalized_scene = scene_semantics if isinstance(scene_semantics, dict) else {}
scene_key = str(normalized_scene.get("scene_key") or "").strip().lower()
if scene_key and scene_key not in {"industry_trend", "external_trend"}:
@ -230,6 +115,141 @@ class AdaptiveRAGDeepGuardAnswerMixin:
)
return None
async def _build_internal_model_config_fast_answer(
self,
*,
question: str,
query_semantics: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
_ = question
semantics = query_semantics if isinstance(query_semantics, dict) else {}
source_policy = str(
semantics.get("effective_source_policy") or semantics.get("source_policy") or ""
).strip().lower()
if source_policy and source_policy != "internal":
return None
operation_contract = semantics.get("operation_contract") if isinstance(semantics.get("operation_contract"), dict) else {}
query_type = str(
operation_contract.get("query_type")
or semantics.get("query_type")
or QUERY_TYPE_UNKNOWN
).strip().lower()
if query_type not in {QUERY_TYPE_STATUS, QUERY_TYPE_EXPLAIN}:
return None
query_profile = semantics.get("query_profile") if isinstance(semantics.get("query_profile"), dict) else {}
referenced_objects = {
str(item).strip().lower()
for item in list(query_profile.get("referenced_objects") or [])
if str(item).strip()
}
if not referenced_objects.intersection({"system_config", "model_routing"}):
return None
target_resolution = (
semantics.get("target_resolution")
if isinstance(semantics.get("target_resolution"), dict)
else {}
)
target_entity_type = str(target_resolution.get("target_entity_type") or "").strip().lower()
if target_entity_type and target_entity_type != "config":
return None
get_bundle = getattr(self.llm, "_get_or_init_bundle", None)
resolve_active_engine = getattr(self.llm, "_resolve_active_engine_from_bundle", None)
resolve_engine_for_call = getattr(self.llm, "_resolve_engine_for_call", None)
if not callable(get_bundle) or not callable(resolve_active_engine) or not callable(resolve_engine_for_call):
return None
try:
_row, bundle = await get_bundle(self.db, tenant_id=self.tenant_id)
bundle = dict(bundle) if isinstance(bundle, dict) else {}
active_engine = dict(resolve_active_engine(bundle) or {})
normal_engine = dict(
await resolve_engine_for_call(
self.db,
tenant_id=self.tenant_id,
function_name="adaptive_rag_answer_normal",
)
or {}
)
deep_engine = dict(
await resolve_engine_for_call(
self.db,
tenant_id=self.tenant_id,
function_name="adaptive_rag_answer_deep",
)
or {}
)
except Exception:
return None
settings = bundle.get("settings") if isinstance(bundle.get("settings"), dict) else {}
local_kb_enabled = bool(settings.get("enable_local_kb_qa"))
def _engine_summary(engine: dict[str, Any]) -> str:
name = str(engine.get("name") or "unknown").strip() or "unknown"
provider = str(engine.get("provider") or "unknown").strip() or "unknown"
model = str(engine.get("model") or "unknown").strip() or "unknown"
routing_meta = engine.get("_routing_meta") if isinstance(engine.get("_routing_meta"), dict) else {}
tier = (
str(routing_meta.get("executed_tier") or routing_meta.get("planned_tier") or "")
.strip()
.lower()
)
if tier:
return f"{name}（provider={provider}, model={model}, tier={tier}）"
return f"{name}（provider={provider}, model={model}）"
active_summary = _engine_summary(active_engine)
normal_summary = _engine_summary(normal_engine)
deep_summary = _engine_summary(deep_engine)
answer_lines = [
f"当前活动引擎是 {active_summary}。",
f"`adaptive_rag_answer_normal` 当前走 {normal_summary}。",
f"`adaptive_rag_answer_deep` 当前走 {deep_summary}。",
f"`enable_local_kb_qa={'true' if local_kb_enabled else 'false'}`。",
]
active_engine_id = str(bundle.get("active_engine_id") or "").strip() or None
return {
"answer": "\n".join(answer_lines),
"sources": [
{
"source": "db",
"label": "Integration[llm_engine]",
"score": 0.99,
"metadata": {
"integration_type": "llm_engine",
"active_engine_id": active_engine_id,
"provider": str(active_engine.get("provider") or ""),
"model": str(active_engine.get("model") or ""),
},
},
{
"source": "db",
"label": "Integration[model_routing:adaptive_rag_answer_normal]",
"score": 0.98,
"metadata": {
"function_name": "adaptive_rag_answer_normal",
"provider": str(normal_engine.get("provider") or ""),
"model": str(normal_engine.get("model") or ""),
},
},
{
"source": "db",
"label": "Integration[model_routing:adaptive_rag_answer_deep]",
"score": 0.98,
"metadata": {
"function_name": "adaptive_rag_answer_deep",
"provider": str(deep_engine.get("provider") or ""),
"model": str(deep_engine.get("model") or ""),
},
},
],
"query_type": QUERY_TYPE_STATUS,
"confidence": 0.99,
"reason_code": "internal_model_config_direct",
"prompt_variant_id": "internal_model_config_direct",
}
@classmethod
def _apply_consultant_mode_preference(
cls,
@ -529,56 +549,54 @@ class AdaptiveRAGDeepGuardAnswerMixin:
message_intent: Optional[str],
query_type: Optional[str],
conversation_text: str,
source_policy: Optional[str] = None,
project_id: Optional[int] = None,
query_semantics: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
_ = (effective_question, conversation_text)
fallback = self._default_answer_scene_semantics(
asked_via=asked_via,
message_intent=message_intent,
)
prompt = PROMPT_RAG_ANSWER_SCENE_SEMANTICS.format(
asked_via=str(asked_via or "").strip().lower() or "pmflow",
message_intent=str(message_intent or "").strip().lower() or "unknown",
query_type=str(query_type or QUERY_TYPE_UNKNOWN).strip().lower() or QUERY_TYPE_UNKNOWN,
conversation_tail=self._truncate_compact_text(conversation_text, limit=1200),
question=str(question or "").strip()[:800],
effective_question=str(effective_question or question or "").strip()[:1000],
).strip()
parsed: dict[str, Any] = {}
try:
raw = await self.llm.generate(
prompt,
db=self.db,
tenant_id=self.tenant_id,
function_name="rag_answer_scene_semantics",
temperature=0.0,
max_tokens=260,
use_prompt_profile=False,
semantics = query_semantics if isinstance(query_semantics, dict) else {}
normalized_source_policy = str(source_policy or "").strip().lower()
if normalized_source_policy == "external":
has_project_scope = (
project_id is not None
and str(semantics.get("scope_hint") or "none").strip().lower() == "project"
)
parser = getattr(self, "_parse_json_object_safe", None)
parsed = parser(raw) if callable(parser) else {}
parsed = dict(parsed) if isinstance(parsed, dict) else {}
except Exception:
parsed = {}
scene_key = "external_trend" if has_project_scope else "industry_trend"
return {
"scene_key": scene_key,
"snapshot_focus": "none",
"deep_intent": "none",
"confidence": 1.0,
"reason": "source_policy:external",
}
scene_key = str(parsed.get("scene_key") or fallback["scene_key"]).strip().lower()
subject_focus = str(semantics.get("subject_focus") or "unknown").strip().lower()
feature_focus = str(semantics.get("feature_focus") or "none").strip().lower()
scope_hint = str(semantics.get("scope_hint") or "none").strip().lower()
normalized_query_type = str(query_type or QUERY_TYPE_UNKNOWN).strip().lower()
scene_key = str(fallback["scene_key"])
if subject_focus == "workload":
scene_key = "member_workload"
elif subject_focus == "project" and normalized_query_type == QUERY_TYPE_STATUS:
scene_key = "project_status"
if scene_key not in self._ANSWER_SCENE_KEYS:
scene_key = str(fallback["scene_key"])
snapshot_focus = str(parsed.get("snapshot_focus") or fallback["snapshot_focus"]).strip().lower()
snapshot_focus = "feature" if (subject_focus == "feature" or feature_focus != "none") else str(fallback["snapshot_focus"])
if snapshot_focus not in self._SNAPSHOT_FOCI:
snapshot_focus = "none"
deep_intent = str(parsed.get("deep_intent") or fallback["deep_intent"]).strip().lower()
if deep_intent not in self._DEEP_INTENTS:
deep_intent = "none"
safe_float = getattr(self, "_safe_float", None)
confidence = safe_float(parsed.get("confidence"), 0.0) if callable(safe_float) else 0.0
snapshot_focus = str(fallback["snapshot_focus"])
confidence = 0.82 if scene_key != str(fallback["scene_key"]) or snapshot_focus != str(fallback["snapshot_focus"]) or scope_hint == "project" else 0.6
return {
"scene_key": scene_key,
"snapshot_focus": snapshot_focus,
"deep_intent": deep_intent,
"deep_intent": str(fallback["deep_intent"]),
"confidence": confidence,
"reason": str(parsed.get("reason") or "").strip(),
"reason": "query_semantics_mapping",
}
@staticmethod
@ -655,18 +673,7 @@ class AdaptiveRAGDeepGuardAnswerMixin:
)
normalized_scene_key = str(scene_semantics.get("scene_key") or "default").strip().lower()
normalized_message_intent = str(message_intent or "").strip().lower()
explicit_project_context = self._question_explicitly_requests_project_context(
question=question,
project_id=project_id,
)
internal_model_config_question = self._looks_like_internal_model_config_question(question)
model_fact_question = self._looks_like_model_fact_question(question)
freshness_question = self._looks_like_external_freshness_question(question)
if normalized_scene_key == "ai_pmo" and model_fact_question and freshness_question and not internal_model_config_question:
normalized_scene_key = "external_trend" if explicit_project_context else "industry_trend"
if normalized_scene_key == "external_trend" and not explicit_project_context:
if normalized_scene_key == "external_trend" and project_id is None:
normalized_scene_key = "industry_trend"
if normalized_scene_key == "member_workload":
@ -1170,6 +1177,14 @@ class AdaptiveRAGDeepGuardAnswerMixin:
message_intent=message_intent,
query_type=query_type,
conversation_text=conversation_text,
source_policy=str(
(query_semantics or {}).get("effective_source_policy")
or (query_semantics or {}).get("source_policy")
or ""
).strip().lower()
or None,
project_id=project_id,
query_semantics=query_semantics,
)
member_workload_direct_answer = await self._build_member_workload_direct_answer(
question,

2
backend/app/services/adaptive_rag_service/mixin_deep_guard_rules.py
View File

@ -167,7 +167,7 @@ class AdaptiveRAGDeepGuardRulesMixin:
continue
source = str(row.get("source") or "").strip().lower()
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
if source == "url":
if source in {"url", "mcp_search"}:
return True
for key in ("target_url", "file_url", "doc_url", "url"):
value = str(metadata.get(key) or "").strip().lower()

73
backend/app/services/adaptive_rag_service/mixin_deterministic_query_core.py
View File

@ -26,14 +26,33 @@ class AdaptiveRAGDeterministicQueryCoreMixin:
project_id=project_id,
query_semantics=query_semantics,
)
if q_type == QUERY_TYPE_EXISTENCE:
existence = await self._try_fast_project_existence_answer(
clean_q=question,
project_id=project_id,
query_semantics=query_semantics,
)
if existence:
existence["query_type"] = QUERY_TYPE_EXISTENCE
existence.setdefault("confidence", 0.96)
existence.setdefault("missing_fields", [])
return existence
if q_type == QUERY_TYPE_STATUS:
created_at = await self._try_fast_project_created_at_answer(clean_q=question, project_id=project_id)
created_at = await self._try_fast_project_created_at_answer(
clean_q=question,
project_id=project_id,
query_semantics=query_semantics,
)
if created_at:
created_at["query_type"] = QUERY_TYPE_STATUS
created_at.setdefault("confidence", 0.97)
created_at.setdefault("missing_fields", [])
return created_at
progress = await self._try_fast_project_progress_answer(clean_q=question, project_id=project_id)
progress = await self._try_fast_project_progress_answer(
clean_q=question,
project_id=project_id,
query_semantics=query_semantics,
)
if progress:
progress["query_type"] = QUERY_TYPE_STATUS
progress.setdefault("confidence", 0.98)
@ -56,6 +75,7 @@ class AdaptiveRAGDeterministicQueryCoreMixin:
*,
question: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]] = None,
) -> tuple[Optional[int], Optional[str], str]:
if project_id is not None:
row = (
@ -72,18 +92,50 @@ class AdaptiveRAGDeterministicQueryCoreMixin:
return (None, None, "")
return (int(row.id), str(row.title or row.name or f"项目#{int(row.id)}"), "")
semantics = query_semantics if isinstance(query_semantics, dict) else {}
target_resolution = semantics.get("target_resolution") if isinstance(semantics.get("target_resolution"), dict) else {}
target_ref = str(target_resolution.get("target_ref") or "").strip()
if target_ref.startswith("project:"):
try:
resolved_project_id = int(target_ref.split(":", 1)[1])
except Exception:
resolved_project_id = None
if resolved_project_id is not None:
row = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id == resolved_project_id,
)
)
)
).scalar_one_or_none()
if row is not None:
return (
int(row.id),
str(row.title or row.name or f"项目#{int(row.id)}"),
str(target_resolution.get("resolved_label") or row.title or row.name or "")[:32],
)
direct_project_id, direct_project_name = await self._match_project_name_in_question(question=question)
if direct_project_id is not None and direct_project_name:
return (int(direct_project_id), str(direct_project_name), str(direct_project_name)[:24])
keyword = self._extract_project_lookup_keyword(question)
terms = self._extract_query_terms(question, preferred=keyword)
for term in terms:
matched = await self._search_projects_by_keyword(keyword=term, project_id=None, limit=5)
term_collector = getattr(self, "_collect_lookup_terms", None)
lookup_terms = term_collector(
question=question,
query_semantics=semantics,
target_resolution=target_resolution,
limit=4,
) if callable(term_collector) else []
searcher = getattr(self, "_search_projects_by_lookup_terms", None)
if callable(searcher) and lookup_terms:
matched, used_term = await searcher(query_terms=lookup_terms, project_id=None, limit=5)
if matched:
target = matched[0]
return (int(target.id), str(target.title or target.name or f"项目#{int(target.id)}"), term)
return (None, None, keyword[:24] if keyword else "")
return (int(target.id), str(target.title or target.name or f"项目#{int(target.id)}"), used_term[:24])
return (None, None, str(lookup_terms[0] if lookup_terms else "")[:24])
@staticmethod
def _normalize_project_subject_text(text: str) -> str:
@ -92,11 +144,6 @@ class AdaptiveRAGDeterministicQueryCoreMixin:
return ""
clean = clean.replace("—", "-").replace("_", " ")
clean = re.sub(r"[^\w\u4e00-\u9fff\- ]+", " ", clean)
clean = re.sub(
r"(我说的是|我说的|请问|请|帮我|一下|这个里面|这里面|里面|这个|那个|一下子|目前|当前|现在|有多少个|多少个|信号源|信号|项目|系统)",
" ",
clean,
)
clean = re.sub(r"[-]+", " ", clean)
clean = re.sub(r"\s+", " ", clean).strip()
return clean

273
backend/app/services/adaptive_rag_service/mixin_fast_answer_project_main.py
View File

@ -16,6 +16,7 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
*,
clean_q: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
question = str(clean_q or "").strip()
if not question:
@ -26,24 +27,16 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
if "项目" not in question and "系统" not in question:
return None
primary_keyword = self._extract_project_lookup_keyword(question)
query_terms = self._extract_query_terms(question, preferred=primary_keyword)
if not query_terms:
return None
started = time.perf_counter()
matched: list[Project] = []
used_keyword = query_terms[0]
for term in query_terms:
rows = await self._search_projects_by_keyword(keyword=term, project_id=project_id, limit=3)
if rows:
matched = rows
used_keyword = term
break
if not matched:
target_project_id, target_name, used_keyword = await self._resolve_project_scope_for_question(
question=question,
project_id=project_id,
query_semantics=query_semantics,
)
if target_project_id is None:
db_query_ms = int((time.perf_counter() - started) * 1000)
return {
"answer": f"未检索到名称或描述包含“{used_keyword}”的项目，无法判断进度。",
"answer": f"未检索到可确认进度的项目，无法判断“{question}”对应对象。",
"sources": [
{
"source": "db",
@ -56,7 +49,36 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
"response_time_ms": db_query_ms,
}
target = matched[0]
target = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id == int(target_project_id),
)
)
)
).scalar_one_or_none()
if target is None:
db_query_ms = int((time.perf_counter() - started) * 1000)
return {
"answer": f"未检索到项目#{int(target_project_id)}，无法判断进度。",
"sources": [
{
"source": "db",
"label": "Project database",
"score": 0.99,
"metadata": {
"table": "projects",
"project_id": int(target_project_id),
"query_keyword": used_keyword,
"match_count": 0,
},
}
],
"db_query_ms": db_query_ms,
"response_time_ms": db_query_ms,
}
task_rows = (
await self.db.execute(
select(Task).where(
@ -124,10 +146,6 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
if asks_risk_or_blockers and latest_update is not None:
answer += f" 最近任务更新时间：{latest_update.strftime('%Y-%m-%d %H:%M')}。"
if len(matched) > 1:
others = "、".join([str(x.title or x.name or f'#{x.id}') for x in matched[1:]])
answer += f" 另有 {len(matched) - 1} 个同名候选项目：{others}。"
return {
"answer": answer,
"sources": [
@ -165,6 +183,7 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
*,
clean_q: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]] = None,
) -> Optional[dict[str, Any]]:
question = str(clean_q or "").strip()
if not question or not self.PROJECT_CREATED_AT_RE.search(question):
@ -172,22 +191,13 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
if "项目" not in question and "系统" not in question:
return None
primary_keyword = self._extract_project_lookup_keyword(question)
query_terms = self._extract_query_terms(question, preferred=primary_keyword)
started = time.perf_counter()
target_project_id, target_name, used_keyword = await self._resolve_project_scope_for_question(
question=question,
project_id=project_id,
query_semantics=query_semantics,
)
if target_project_id is None and query_terms:
for term in query_terms:
rows = await self._search_projects_by_keyword(keyword=term, project_id=project_id, limit=3)
if rows:
target_project_id = int(rows[0].id)
target_name = str(rows[0].title or rows[0].name or f"项目#{int(rows[0].id)}")
used_keyword = term
break
if target_project_id is None:
db_query_ms = int((time.perf_counter() - started) * 1000)
@ -284,6 +294,46 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
subject_focus = self._query_subject_focus(semantics)
scope_hint = self._query_scope_hint(semantics)
message_intent = str(semantics.get("message_intent") or semantics.get("intent") or "").strip().lower()
target_resolution = semantics.get("target_resolution") if isinstance(semantics.get("target_resolution"), dict) else {}
target_entity_type = str(target_resolution.get("target_entity_type") or "").strip().lower()
target_ref = str(target_resolution.get("target_ref") or "").strip()
if target_entity_type == "project" and target_ref.startswith("project:"):
try:
resolved_project_id = int(target_ref.split(":", 1)[1])
except Exception:
resolved_project_id = None
if resolved_project_id is not None:
matched_project = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id == resolved_project_id,
)
)
)
).scalar_one_or_none()
if matched_project is not None:
title = str(matched_project.title or matched_project.name or f"项目#{int(matched_project.id)}")
status = str(matched_project.status or "unknown")
answer = f"有，当前匹配到 1 个相关项目：\n#{int(matched_project.id)} {title}（状态:{status}）"
return {
"answer": answer,
"sources": [
{
"source": "db",
"label": "Project database",
"score": 0.99,
"metadata": {
"table": "projects",
"query_keyword": str(target_resolution.get("resolved_label") or title),
"match_count": 1,
"project_id": int(matched_project.id),
},
}
],
"db_query_ms": 0,
}
if (
project_id is None
and message_intent != "project_lookup_question"
@ -292,20 +342,38 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
):
return None
primary_keyword = self._extract_project_lookup_keyword(question)
query_terms = self._extract_query_terms(question, preferred=primary_keyword)
query_terms = self._collect_lookup_terms(
question=question,
query_semantics=semantics,
operation_contract=semantics.get("operation_contract") if isinstance(semantics.get("operation_contract"), dict) else None,
target_resolution=target_resolution,
limit=4,
)
if not query_terms:
matched_project_id, matched_name = await self._match_project_name_in_question(question=question)
if matched_project_id is not None and matched_name:
query_terms = [str(matched_name)]
if not query_terms:
return None
started = time.perf_counter()
semantic_match = await self._try_fast_project_existence_answer_from_semantic_facts(
query_terms=query_terms,
project_id=project_id,
)
if semantic_match is not None:
semantic_match["db_query_ms"] = int((time.perf_counter() - started) * 1000)
semantic_match.setdefault("response_time_ms", semantic_match["db_query_ms"])
return semantic_match
matched: list[Project] = []
used_keyword = query_terms[0]
for term in query_terms:
rows = await self._search_projects_by_keyword(keyword=term, project_id=project_id, limit=5)
if rows:
matched = rows
used_keyword = term
break
ranked_rows, used_keyword = await self._search_projects_by_lookup_terms(
query_terms=query_terms,
project_id=project_id,
limit=5,
)
matched = list(ranked_rows or [])
db_query_ms = int((time.perf_counter() - started) * 1000)
if matched:
@ -334,6 +402,71 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
"db_query_ms": db_query_ms,
}
async def _try_fast_project_existence_answer_from_semantic_facts(
self,
*,
query_terms: list[str],
project_id: Optional[int],
) -> Optional[dict[str, Any]]:
fact_query = getattr(self, "_query_project_semantic_facts_by_lookup_terms", None)
if not callable(fact_query):
return None
matched_facts, used_keyword = await fact_query(
query_terms=query_terms,
project_id=project_id,
limit=10,
)
if not matched_facts:
return None
matched_project_ids = []
for row in matched_facts:
try:
entity_id = int(row.entity_id)
except Exception:
continue
if entity_id not in matched_project_ids:
matched_project_ids.append(entity_id)
if not matched_project_ids:
return None
project_rows = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id.in_(matched_project_ids),
)
)
)
).scalars().all()
if not project_rows:
return None
lines = [
f"#{int(row.id)} {str(row.title or row.name or '未命名项目')}（状态:{str(row.status or 'unknown')}）"
for row in project_rows
]
answer = f"有，当前匹配到 {len(project_rows)} 个相关项目：\n" + "\n".join(lines)
return {
"answer": answer,
"sources": [
{
"source": "db",
"label": "Semantic facts",
"score": 0.99,
"metadata": {
"table": "semantic_facts",
"query_keyword": used_keyword,
"match_count": len(project_rows),
"matched_project_ids": matched_project_ids,
"fact_keys": sorted({str(row.fact_key or "") for row in matched_facts if str(row.fact_key or "").strip()}),
},
}
],
}
async def _search_projects_by_keyword(
self,
*,
@ -356,3 +489,67 @@ class AdaptiveRAGFastAnswerProjectMainMixin:
)
)
return (await self.db.execute(stmt.limit(max(1, int(limit))))).scalars().all()
async def _search_projects_by_lookup_terms(
self,
*,
query_terms: list[str],
project_id: Optional[int],
limit: int,
) -> tuple[list[Project], str]:
terms = self._dedupe_lookup_terms([str(item or "").strip() for item in query_terms], limit=6)
if not terms:
return ([], "")
for term in terms:
rows = await self._search_projects_by_keyword(keyword=term, project_id=project_id, limit=limit)
if rows:
return (rows, term)
stmt = select(Project).where(Project.tenant_id == self.tenant_id)
if project_id is not None:
stmt = stmt.where(Project.id == int(project_id))
candidates = (
await self.db.execute(
stmt.order_by(Project.updated_at.desc(), Project.id.desc()).limit(max(40, int(limit) * 20))
)
).scalars().all()
if not candidates:
return ([], terms[0])
ranked: list[tuple[float, Project, str]] = []
for row in candidates:
title = str(row.title or "").strip()
name = str(row.name or "").strip()
description = str(row.description or "").strip()
best_term = ""
best_score = 0.0
for term in terms:
score = max(
self._lookup_similarity_score(term, title),
self._lookup_similarity_score(term, name),
self._lookup_similarity_score(term, description) * 0.82,
)
if score > best_score:
best_score = score
best_term = term
if best_score >= 0.62 and best_term:
ranked.append((best_score, row, best_term))
if not ranked:
return ([], terms[0])
ranked.sort(key=lambda item: item[0], reverse=True)
deduped: list[Project] = []
used_keyword = ranked[0][2]
seen_ids: set[int] = set()
for _score, row, term in ranked:
row_id = int(row.id)
if row_id in seen_ids:
continue
seen_ids.add(row_id)
deduped.append(row)
if not used_keyword:
used_keyword = term
if len(deduped) >= max(1, int(limit)):
break
return (deduped, used_keyword)

237
backend/app/services/adaptive_rag_service/mixin_fast_answer_project_utils.py
View File

@ -8,83 +8,188 @@ class AdaptiveRAGFastAnswerProjectUtilsMixin:
_SEMANTIC_SUBJECT_FOCI = {"people", "project", "task", "signal", "charter", "workload", "feature", "unknown"}
_ALLOWED_SCOPE_VALUES = {"none", "global", "project"}
_SEMANTIC_FEATURE_FOCI = {"none", "purpose", "entry", "catalog"}
_LOOKUP_TEXT_TRANSLATION = str.maketrans(
{
"，": " ",
"。": " ",
"！": " ",
"？": " ",
"：": " ",
"；": " ",
"（": " ",
"）": " ",
"(": " ",
")": " ",
"\"": " ",
"'": " ",
}
)
@staticmethod
def _extract_project_lookup_keyword(question: str) -> str:
text = str(question or "").strip()
if not text:
def _normalize_lookup_text(text: str) -> str:
normalized = str(text or "").strip().lower()
if not normalized:
return ""
text = text.replace("？", "?")
normalized = normalized.translate(AdaptiveRAGFastAnswerProjectUtilsMixin._LOOKUP_TEXT_TRANSLATION)
normalized = normalized.replace("—", "-").replace("_", " ")
normalized = re.sub(r"[^\w\u4e00-\u9fff\- ]+", " ", normalized)
normalized = re.sub(r"\s+", " ", normalized).strip()
return normalized[:96]
direct_patterns = [
r"(?:是否有|有没有|有无)([\u4e00-\u9fffA-Za-z0-9_-]{1,24})(?:项目|系统)?",
r"(?:是否存在|存在)([\u4e00-\u9fffA-Za-z0-9_-]{1,24})(?:项目|系统)?",
]
for pattern in direct_patterns:
m = re.search(pattern, text)
if m:
value = str(m.group(1) or "").strip(" ，。,.!?？：:;；“”\"'()（）")
value = QUESTION_NOISE_RE.sub("", value).strip()
if len(value) >= 2:
return value[:24]
anchored_matches = list(re.finditer(r"([\u4e00-\u9fffA-Za-z0-9_-]{1,24})(?:项目|系统)", text))
for m in reversed(anchored_matches):
value = str(m.group(1) or "").strip(" ，。,.!?？：:;；“”\"'()（）")
value = QUESTION_NOISE_RE.sub("", value).strip()
if len(value) >= 2:
return value[:24]
patterns = [
r"有(?:没有)?(.{1,24}?)(?:项目|系统).*\?",
r"项目.*?有(?:没有)?(.{1,24}?)(?:吗|\?)",
r"是否有(.{1,24}?)(?:项目|系统)",
r"(.{1,24}?)(?:项目|系统)(?:有吗|有没有|是否有)",
]
for pattern in patterns:
m = re.search(pattern, text)
if not m:
continue
kw = str(m.group(1) or "").strip(" ，。,.!?？：:;；“”\"'()（）")
kw = QUESTION_NOISE_RE.sub("", kw).strip()
if kw:
return kw[:24]
fallback = re.sub(r"[?？]", " ", text)
fallback = re.sub(
r"(项目|系统|目前|当前|现在|我们|有没有|是否有|有吗|有|吗|什么|进度|层面|阶段|是在|在|是|请问|一下|多少)",
" ",
fallback,
@classmethod
def _collect_structured_object_mentions(
cls,
*,
query_semantics: Optional[dict[str, Any]] = None,
operation_contract: Optional[dict[str, Any]] = None,
target_resolution: Optional[dict[str, Any]] = None,
limit: int = 6,
) -> list[str]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
operation = operation_contract if isinstance(operation_contract, dict) else (
semantics.get("operation_contract") if isinstance(semantics.get("operation_contract"), dict) else {}
)
fallback = re.sub(r"\s+", " ", fallback).strip(" ，。,.!?？：:;；")
return fallback[:24]
target = target_resolution if isinstance(target_resolution, dict) else (
semantics.get("target_resolution") if isinstance(semantics.get("target_resolution"), dict) else {}
)
raw_rows: list[Any] = []
raw_rows.extend(list(operation.get("object_mentions") or []))
raw_rows.extend(list(semantics.get("object_mentions") or []))
resolved_label = str(target.get("resolved_label") or "").strip()
if resolved_label:
raw_rows.append(resolved_label)
@staticmethod
def _extract_query_terms(question: str, *, preferred: str = "") -> list[str]:
base = str(question or "").strip()
mentions: list[str] = []
for item in raw_rows:
text = str(item or "").strip(" ，。,.!?？：:;；“”\"'()（）")
if len(text) < 2 or text in mentions:
continue
mentions.append(text[:32])
if len(mentions) >= max(1, int(limit)):
break
return mentions
@classmethod
def _lookup_similarity_score(cls, left: str, right: str) -> float:
lhs = cls._normalize_lookup_text(left)
rhs = cls._normalize_lookup_text(right)
if not lhs or not rhs:
return 0.0
if lhs == rhs:
return 1.0
if lhs in rhs:
return min(0.98, 0.82 + min(0.16, len(lhs) / max(len(rhs), 1) * 0.16))
if rhs in lhs:
return min(0.94, 0.78 + min(0.16, len(rhs) / max(len(lhs), 1) * 0.16))
return float(SequenceMatcher(None, lhs, rhs).ratio())
@classmethod
def _dedupe_lookup_terms(cls, rows: list[str], *, limit: int = 6) -> list[str]:
terms: list[str] = []
pref = str(preferred or "").strip()
if len(pref) >= 2:
terms.append(pref[:24])
for raw in rows:
text = str(raw or "").strip()
if len(text) < 2 or text in terms:
continue
terms.append(text[:32])
if len(terms) >= max(1, int(limit)):
break
return terms
clean = re.sub(r"[?？,，。!！:：;；()（）\"“”']", " ", base)
clean = QUESTION_NOISE_RE.sub(" ", clean)
clean = re.sub(r"(项目|系统|进度|状态|阶段|完成率|层面)", " ", clean)
clean = re.sub(r"\s+", " ", clean).strip()
@classmethod
def _collect_lookup_terms(
cls,
*,
question: str = "",
query_semantics: Optional[dict[str, Any]] = None,
operation_contract: Optional[dict[str, Any]] = None,
target_resolution: Optional[dict[str, Any]] = None,
limit: int = 6,
) -> list[str]:
mentions = cls._collect_structured_object_mentions(
query_semantics=query_semantics,
operation_contract=operation_contract,
target_resolution=target_resolution,
limit=limit,
)
return mentions
token_candidates: list[str] = []
for token in clean.split(" "):
t = token.strip()
if len(t) >= 2:
token_candidates.append(t[:24])
async def _query_project_semantic_facts_by_lookup_terms(
self,
*,
query_terms: list[str],
project_id: Optional[int],
limit: int = 20,
) -> tuple[list[SemanticFact], str]:
return await self._query_semantic_facts_by_lookup_terms(
entity_type="project",
query_terms=query_terms,
entity_id=project_id,
limit=limit,
)
for m in re.finditer(r"[\u4e00-\u9fff]{2,10}", clean):
token_candidates.append(m.group(0)[:24])
async def _query_semantic_facts_by_lookup_terms(
self,
*,
entity_type: str,
query_terms: list[str],
entity_id: Optional[int],
limit: int = 20,
) -> tuple[list[SemanticFact], str]:
terms = self._dedupe_lookup_terms([str(item or "").strip() for item in query_terms], limit=6)
if not terms:
return ([], "")
for t in sorted(token_candidates, key=len, reverse=True):
if t and t not in terms:
terms.append(t)
return terms[:6]
stmt = select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type == str(entity_type or "").strip(),
SemanticFact.is_active.is_(True),
)
)
if entity_id is not None:
stmt = stmt.where(SemanticFact.entity_id == int(entity_id))
fact_rows = (
await self.db.execute(
stmt.order_by(SemanticFact.confidence.desc(), SemanticFact.updated_at.desc(), SemanticFact.id.desc()).limit(
max(40, int(limit) * 8)
)
)
).scalars().all()
if not fact_rows:
return ([], terms[0])
ranked: list[tuple[float, SemanticFact, str]] = []
for row in fact_rows:
fact_text = str(row.fact_value_text or "").strip()
if not fact_text:
continue
best_term = ""
best_score = 0.0
for term in terms:
score = self._lookup_similarity_score(term, fact_text)
if score > best_score:
best_score = score
best_term = term
if best_score >= 0.74 and best_term:
ranked.append((best_score, row, best_term))
if not ranked:
return ([], terms[0])
ranked.sort(key=lambda item: item[0], reverse=True)
deduped: list[SemanticFact] = []
used_keyword = ranked[0][2]
seen_row_ids: set[int] = set()
for _score, row, term in ranked:
row_id = int(row.id)
if row_id in seen_row_ids:
continue
seen_row_ids.add(row_id)
deduped.append(row)
if not used_keyword:
used_keyword = term
if len(deduped) >= max(1, int(limit)):
break
return (deduped, used_keyword)
@classmethod
def _query_subject_focus(cls, query_semantics: Optional[dict[str, Any]]) -> str:

78
backend/app/services/adaptive_rag_service/mixin_fast_answer_semantic_p2.py
View File

@ -23,10 +23,19 @@ class AdaptiveRAGFastAnswerSemanticMixinP2:
rewrite_meta: Optional[dict[str, Any]] = None,
thinking_mode: str = THINKING_MODE_NORMAL,
answer_mode: str = ANSWER_MODE_FAST_PATH,
route_decision: Optional[dict[str, Any]] = None,
query_semantics: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
started = time.perf_counter()
answer_text = str(fast_result.get("answer") or "未命中结果")
sources = list(fast_result.get("sources") or [])
render_with_contract = getattr(self, "_render_answer_with_route_contract", None)
answer_text = str(fast_result.get("answer") or "未命中结果")
if callable(render_with_contract):
answer_text = render_with_contract(
answer=answer_text,
sources=sources,
route_decision=route_decision,
)
active_session_id = session_id
active_turn_index = turn_index
if active_session_id is None or active_turn_index is None:
@ -44,22 +53,50 @@ class AdaptiveRAGFastAnswerSemanticMixinP2:
missing_fields = list(fast_result.get("missing_fields") or [])
final_query_type = str(fast_result.get("query_type") or query_type or QUERY_TYPE_EXPLAIN)
prompt_variant_id = str(fast_result.get("prompt_variant_id") or "deterministic").strip() or "deterministic"
contract_verdict = self._verify_internal_answer_contract(
query_type=final_query_type,
answer=answer_text,
sources=sources,
route_decision=route_decision,
)
effective_answer_mode = answer_mode
effective_grounding_status = GROUNDING_STATUS_GROUNDED
effective_reason_code = str(fast_result.get("reason_code") or "").strip() or None
if not bool(contract_verdict.get("ok")):
guard_payload = self._build_internal_answer_contract_guard_payload(
query_type=final_query_type,
reason_code=str(contract_verdict.get("reason_code") or ""),
missing_fields=list(contract_verdict.get("missing_fields") or []),
)
answer_text = str(guard_payload.get("answer") or answer_text)
sources = list(guard_payload.get("sources") or [])
missing_fields = list(guard_payload.get("missing_fields") or [])
effective_answer_mode = str(guard_payload.get("answer_mode") or ANSWER_MODE_BLOCKED)
effective_grounding_status = str(guard_payload.get("grounding_status") or GROUNDING_STATUS_BLOCKED)
effective_reason_code = str(guard_payload.get("reason_code") or "").strip() or "internal_answer_contract_violation"
retrieval_metrics = {
"db_query_ms": int(fast_result.get("db_query_ms") or 0),
"vector_search_ms": 0,
"graph_query_ms": 0,
"db_timeout": False,
"vector_timeout": False,
"graph_timeout": False,
}
inject_runtime_summaries = getattr(self, "_inject_internal_query_runtime_summaries", None)
if callable(inject_runtime_summaries):
retrieval_metrics = inject_runtime_summaries(
retrieval_metrics=retrieval_metrics,
query_semantics=query_semantics,
)
runtime_fields = self._build_qa_record_runtime_fields(
answer=answer_text,
sources=sources,
missing_fields=missing_fields,
answer_mode=answer_mode,
grounding_status=GROUNDING_STATUS_GROUNDED,
answer_mode=effective_answer_mode,
grounding_status=effective_grounding_status,
prompt_variant_id=prompt_variant_id,
reason_code=str(fast_result.get("reason_code") or "").strip() or None,
retrieval_metrics={
"db_query_ms": int(fast_result.get("db_query_ms") or 0),
"vector_search_ms": 0,
"graph_query_ms": 0,
"db_timeout": False,
"vector_timeout": False,
"graph_timeout": False,
},
reason_code=effective_reason_code,
retrieval_metrics=retrieval_metrics,
)
qa = QARecord(
@ -75,7 +112,7 @@ class AdaptiveRAGFastAnswerSemanticMixinP2:
project_id=project_id,
session_id=active_session_id_int,
turn_index=active_turn_index,
**self._build_qa_record_external_fields(),
**self._build_qa_record_external_fields(route_decision=route_decision),
**runtime_fields,
)
self.db.add(qa)
@ -91,6 +128,15 @@ class AdaptiveRAGFastAnswerSemanticMixinP2:
rewrite_meta=rewrite_meta or {},
sources=sources,
)
await self.db.flush()
persist_route_memory = getattr(self, "_persist_query_route_memory", None)
if callable(persist_route_memory):
await persist_route_memory(
qa_record_id=int(qa.id) if getattr(qa, "id", None) is not None else None,
question_text=question,
project_id=project_id,
route_decision=route_decision,
)
await self.db.commit()
await self.db.refresh(qa)
cache_session_id = active_session_id_int if active_session_id_int is not None else (
@ -120,12 +166,14 @@ class AdaptiveRAGFastAnswerSemanticMixinP2:
confidence=confidence,
missing_fields=list(missing_fields or []),
thinking_mode=thinking_mode,
answer_mode=answer_mode,
grounding_status=GROUNDING_STATUS_GROUNDED,
answer_mode=effective_answer_mode,
grounding_status=effective_grounding_status,
policy_version=runtime_fields["policy_version"],
citations=runtime_fields["citations"],
sources_used=runtime_fields["sources_used"],
evidence_pack=runtime_fields["evidence_pack"],
tool_trace=runtime_fields["tool_trace"],
prompt_variant_id=prompt_variant_id,
reason_code=effective_reason_code,
route_decision=route_decision,
)

15
backend/app/services/adaptive_rag_service/mixin_followup_anchor.py
View File

@ -579,6 +579,9 @@ class AdaptiveRAGFollowupAnchorMixin:
anchor_reply=chosen,
)
if is_meta_followup and explicit_anchor is None and chosen is None and not candidates:
return FollowupAnchorResolution()
if not candidates:
return FollowupAnchorResolution(
followup_mode=followup_mode,
@ -617,6 +620,9 @@ class AdaptiveRAGFollowupAnchorMixin:
anchor_reply=chosen,
)
if is_meta_followup:
return FollowupAnchorResolution()
return FollowupAnchorResolution(
followup_mode=followup_mode,
anchor_mode="ambiguous_candidates",
@ -764,14 +770,7 @@ class AdaptiveRAGFollowupAnchorMixin:
)
anchor_kind = str(resolution.anchor_kind or "").strip() or None
if anchor_row is None:
return {
"answer": "当前无法追溯到本次回复的路由记录。",
"meta_followup_kind": meta_kind,
"reason_code": "meta_followup_no_route_record",
"anchor_kind": anchor_kind,
"anchor_mode": resolution.anchor_mode,
"routing_snapshot": {},
}
return None
routing_snapshot = (
anchor_row.routing_snapshot

6
backend/app/services/adaptive_rag_service/mixin_manifest_core.py
View File

@ -101,7 +101,10 @@ class AdaptiveRAGManifestCoreMixin:
) -> dict[str, Any]:
normalized = self._normalize_entity_text(question)
alias_terms = self._collect_alias_terms(normalized)
query_terms = self._extract_query_terms(question, preferred=self._extract_project_lookup_keyword(question))
term_collector = getattr(self, "_collect_lookup_terms", None)
query_terms = term_collector(question=question, limit=6) if callable(term_collector) else []
if normalized:
query_terms.append(normalized[:32])
for term in alias_terms:
if term not in query_terms:
query_terms.append(term)
@ -192,4 +195,3 @@ class AdaptiveRAGManifestCoreMixin:
"count": min(len(catalog), max_rows),
"items": catalog[:max_rows],
}

16
backend/app/services/adaptive_rag_service/mixin_manifest_feature_p1.py
View File

@ -101,8 +101,9 @@ class AdaptiveRAGManifestFeatureMixinP1:
catalog = self._feature_catalog_from_snapshot(system_snapshot)
if not catalog:
return []
query_terms = [x.lower() for x in self._extract_query_terms(question)[:8] if str(x).strip()]
matched: list[dict[str, Any]] = []
normalizer = getattr(self, "_normalize_lookup_text", None)
normalized_text = normalizer(question) if callable(normalizer) else self._normalize_entity_text(question)
for item in catalog:
name = str(item.get("name") or "").strip()
route = str(item.get("route") or "").strip()
@ -116,11 +117,13 @@ class AdaptiveRAGManifestFeatureMixinP1:
score = 0.0
for alias in alias_pool:
if alias and alias in text:
score += 0.45
for term in query_terms:
if term and any(term in alias or alias in term for alias in alias_pool):
score += 0.18
if not alias:
continue
if alias == normalized_text:
score = max(score, 0.96)
continue
if alias in text or normalized_text in alias:
score = max(score, 0.82)
if score <= 0:
continue
matched.append(
@ -223,4 +226,3 @@ class AdaptiveRAGManifestFeatureMixinP1:
}
)
return rows[:3]

12
backend/app/services/adaptive_rag_service/mixin_multistep_core.py
View File

@ -5,6 +5,13 @@ from .shared import *
class AdaptiveRAGMultiStepCoreMixin:
@staticmethod
def _skip_internal_hint_expansion(query_semantics: Optional[dict[str, Any]]) -> bool:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
if bool(semantics.get("external_freshness_only")):
return True
return str(semantics.get("effective_source_policy") or semantics.get("source_policy") or "").strip().lower() == "external"
async def _run_single_retrieval_pass(
self,
*,
@ -98,8 +105,9 @@ class AdaptiveRAGMultiStepCoreMixin:
query_semantics: Optional[dict[str, Any]] = None,
) -> tuple[str, dict[str, Any]]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
if bool(semantics.get("external_freshness_only")):
return str(question or "").strip()[:1200], {"mode": "freshness_guard_skip_internal_hint"}
if self._skip_internal_hint_expansion(semantics):
mode = "freshness_guard_skip_internal_hint" if bool(semantics.get("external_freshness_only")) else "external_source_policy_skip_internal_hint"
return str(question or "").strip()[:1200], {"mode": mode}
resolved = await self.entity_resolution(question=question, project_id=project_id, limit=5)
manifest = await self.get_project_manifest(project_id=project_id)
hints: list[str] = []

18
backend/app/services/adaptive_rag_service/mixin_multistep_flow.py
View File

@ -19,12 +19,16 @@ class AdaptiveRAGMultiStepFlowMixin:
) -> tuple[list[RetrievedItem], dict[str, Any]]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
external_freshness_only = bool(semantics.get("external_freshness_only"))
external_source_policy = str(
semantics.get("effective_source_policy") or semantics.get("source_policy") or ""
).strip().lower() == "external"
skip_internal_enrichment = external_freshness_only or external_source_policy
resolved: dict[str, Any] = {"matched_entities": [], "canonical_terms": []}
manifest: dict[str, Any] = {}
system_snapshot: dict[str, Any] = {}
symbol_hits: dict[str, Any] = {"count": 0, "items": []}
feature_hits: list[dict[str, Any]] = []
if not external_freshness_only:
if not skip_internal_enrichment:
resolved = await self.entity_resolution(question=question, project_id=project_id, limit=6)
manifest = await self.get_project_manifest(project_id=project_id)
system_snapshot = await self.context_cache.get_system_snapshot(
@ -87,6 +91,14 @@ class AdaptiveRAGMultiStepFlowMixin:
"mode": "external_freshness_only",
}
)
elif external_source_policy:
tool_trace.append(
{
"step": 0,
"tool": "source_policy_guard",
"mode": "external_only",
}
)
if session_state is not None and isinstance(session_state.last_entity_ids, list):
anchor_ids = [int(x) for x in session_state.last_entity_ids if str(x).isdigit()][:6]
else:
@ -124,7 +136,7 @@ class AdaptiveRAGMultiStepFlowMixin:
vector_top_k=vector_top_k,
)
feature_rows = []
if not external_freshness_only:
if not skip_internal_enrichment:
feature_rows = self._retrieve_feature_catalog_context(
question=current_question,
system_snapshot=system_snapshot,
@ -190,7 +202,7 @@ class AdaptiveRAGMultiStepFlowMixin:
break
next_domains: list[str] = []
if external_freshness_only:
if skip_internal_enrichment:
next_domains = list(current_domains)
else:
for item in [*current_domains, "graph", "db", "doc", "pmflow", "feishu"]:

13
backend/app/services/adaptive_rag_service/mixin_qa_feedback_records.py
View File

@ -35,6 +35,18 @@ class AdaptiveRAGQAFeedbackRecordsMixin:
submitted_by=submitted_by,
)
self.db.add(feedback)
await self.db.flush()
await self._update_query_route_feedback_outcome(
qa_record_id=int(qa_id),
feedback_outcome={
"qa_feedback_id": int(feedback.id) if getattr(feedback, "id", None) is not None else None,
"rating": str(feedback.rating or "").strip().lower(),
"helpfulness": str(feedback.helpfulness or "").strip() or None,
"error_types": [str(x).strip() for x in (feedback.error_types or []) if str(x).strip()],
"correct_answer": str(feedback.correct_answer or "").strip() or None,
"submitted_by": int(submitted_by) if submitted_by is not None else None,
},
)
await self.db.commit()
await self.db.refresh(feedback)
if str(feedback.rating or "").strip().lower() == "negative" and (
@ -152,4 +164,3 @@ class AdaptiveRAGQAFeedbackRecordsMixin:
"page_size": page_size,
"count": len(items),
}

467
backend/app/services/adaptive_rag_service/mixin_query_route_gateway.py Normal file
View File

@ -0,0 +1,467 @@
"""Adaptive RAG query route gateway helpers."""
from __future__ import annotations
from .shared import *
class AdaptiveRAGQueryRouteGatewayMixin:
_QUERY_ROUTE_MODES = {"internal", "external", "uncertain"}
_INTERNAL_ALLOWED_SOURCES = {"db", "graph", "doc", "pmflow", "feishu"}
_EXTERNAL_ALLOWED_SOURCES = {"url", "mcp_search", "system"}
def _normalize_query_route_decision(self, payload: Optional[dict[str, Any]]) -> dict[str, Any]:
raw = payload if isinstance(payload, dict) else {}
reply_contract = raw.get("reply_contract") if isinstance(raw.get("reply_contract"), dict) else {}
query_profile = raw.get("query_profile") if isinstance(raw.get("query_profile"), dict) else {}
probe_snapshot_summary = (
raw.get("probe_snapshot_summary")
if isinstance(raw.get("probe_snapshot_summary"), dict)
else {}
)
return {
"route_mode": self._normalize_query_route_mode(raw.get("route_mode")),
"effective_source_policy": self._normalize_query_route_mode(
raw.get("effective_source_policy") or raw.get("route_mode")
),
"decision_origin": str(raw.get("decision_origin") or "").strip() or "unknown",
"reply_contract": dict(reply_contract),
"query_profile": self._normalize_query_profile(query_profile),
"probe_snapshot_summary": dict(probe_snapshot_summary),
"probe_snapshot": dict(raw.get("probe_snapshot") or {}) if isinstance(raw.get("probe_snapshot"), dict) else {},
"clarification_text": str(raw.get("clarification_text") or "").strip() or None,
}
@classmethod
def _normalize_query_route_mode(cls, value: Any) -> str:
normalized = str(value or "").strip().lower()
if normalized in cls._QUERY_ROUTE_MODES:
return normalized
return "uncertain"
@classmethod
def _normalize_query_profile(cls, payload: Optional[dict[str, Any]]) -> dict[str, Any]:
rows = payload if isinstance(payload, dict) else {}
referenced_objects: list[str] = []
for item in list(rows.get("referenced_objects") or []):
normalized = re.sub(r"[^a-z0-9:_\-]", "", str(item or "").strip().lower())
if normalized and normalized not in referenced_objects:
referenced_objects.append(normalized[:48])
if not referenced_objects:
referenced_objects = ["unknown"]
return {
"needs_private_context": bool(rows.get("needs_private_context")),
"needs_freshness": bool(rows.get("needs_freshness")),
"referenced_objects": referenced_objects,
"confidence": max(0.0, min(1.0, float(rows.get("confidence") or 0.0))),
}
@classmethod
def _build_query_route_reply_contract(
cls,
*,
route_mode: str,
effective_source_policy: str,
clarification_text: Optional[str] = None,
) -> dict[str, Any]:
normalized_route = cls._normalize_query_route_mode(route_mode)
normalized_effective = cls._normalize_query_route_mode(effective_source_policy)
if normalized_route == "internal" or normalized_effective == "internal":
return {
"mode": "explicit_source",
"head": "内部知识库查询：",
"tail_hint": None,
}
if normalized_route == "uncertain" and normalized_effective != "external":
return {
"mode": "clarify",
"head": None,
"tail_hint": None,
"clarification_text": str(clarification_text or "").strip() or None,
}
return {
"mode": "explicit_source",
"head": "外部实时查询：",
"tail_hint": (
"如果你指的是系统内部配置/项目对象，请明确指出。"
if normalized_route == "uncertain"
else None
),
}
@classmethod
def _build_query_route_clarification_text(cls) -> str:
return "你是想问外部公开事实，还是系统内部配置/项目对象？"
@classmethod
def _build_contract_snapshot_summary(
cls,
*,
source_policy_hint: Optional[str],
query_semantics: Optional[dict[str, Any]],
route_mode: str,
effective_source_policy: str,
decision_origin: str,
) -> dict[str, Any]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
return {
"contract": {
"source_policy_hint": cls._normalize_query_route_mode(source_policy_hint),
"semantic_source_policy": cls._normalize_query_route_mode(semantics.get("source_policy")),
"semantic_effective_source_policy": cls._normalize_query_route_mode(
semantics.get("effective_source_policy")
),
"route_mode": cls._normalize_query_route_mode(route_mode),
"effective_source_policy": cls._normalize_query_route_mode(effective_source_policy),
"decision_origin": str(decision_origin or "").strip() or "unknown",
},
}
def _build_query_route_audit_snapshot(
self,
*,
route_decision: Optional[dict[str, Any]],
) -> dict[str, Any]:
normalized = self._normalize_query_route_decision(route_decision)
return {
"route_mode": normalized["route_mode"],
"effective_source_policy": normalized["effective_source_policy"],
"decision_origin": normalized["decision_origin"],
"reply_contract": normalized["reply_contract"],
"query_profile": normalized["query_profile"],
"probe_snapshot_summary": normalized["probe_snapshot_summary"],
}
def _resolve_route_contract(
self,
*,
query_semantics: Optional[dict[str, Any]],
source_policy_hint: Optional[str],
) -> dict[str, Any]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
normalized_hint = self._normalize_query_route_mode(source_policy_hint)
if normalized_hint in {"internal", "external"}:
return {
"source_policy": normalized_hint,
"effective_source_policy": normalized_hint,
"decision_origin": "upstream_hint",
"source_reason": "upstream_source_policy_hint",
}
source_policy = self._normalize_query_route_mode(semantics.get("source_policy"))
semantic_effective = self._normalize_query_route_mode(
semantics.get("effective_source_policy") or semantics.get("source_policy")
)
effective_source_policy = (
semantic_effective
if semantic_effective in {"internal", "external"}
else ("uncertain" if source_policy == "uncertain" else source_policy)
)
return {
"source_policy": source_policy,
"effective_source_policy": effective_source_policy,
"decision_origin": str(semantics.get("decision_origin") or "query_semantics_contract").strip()
or "query_semantics_contract",
"source_reason": str(semantics.get("source_reason") or "query_semantics_contract").strip()
or "query_semantics_contract",
}
def _apply_route_decision_to_query_semantics(
self,
*,
query_semantics: Optional[dict[str, Any]],
route_decision: Optional[dict[str, Any]],
) -> dict[str, Any]:
semantics = dict(query_semantics or {})
normalized = self._normalize_query_route_decision(route_decision)
effective_source_policy = normalized["effective_source_policy"]
if effective_source_policy == "uncertain":
retrieval_domains: list[str] = []
else:
retrieval_domains = self._apply_source_policy_to_domains(
domains=[
str(item).strip().lower()
for item in list(semantics.get("retrieval_domains") or [])
if str(item or "").strip()
],
effective_source_policy=effective_source_policy,
)
semantics["route_decision"] = normalized
semantics["query_profile"] = normalized["query_profile"]
semantics["source_policy"] = normalized["route_mode"]
semantics["effective_source_policy"] = effective_source_policy
semantics["decision_origin"] = normalized["decision_origin"]
semantics["source_reason"] = normalized["decision_origin"]
semantics["retrieval_domains"] = retrieval_domains
return semantics
async def _run_query_route_gateway(
self,
*,
question: str,
asked_via: str,
project_id: Optional[int],
source_chat_id: Optional[str],
query_semantics: Optional[dict[str, Any]],
source_policy_hint: Optional[str],
) -> dict[str, Any]:
_ = (question, asked_via, project_id, source_chat_id)
semantics = dict(query_semantics or {})
query_profile = self._normalize_query_profile(semantics.get("query_profile"))
contract = self._resolve_route_contract(
query_semantics=semantics,
source_policy_hint=source_policy_hint,
)
route_mode = self._normalize_query_route_mode(contract.get("source_policy"))
effective_source_policy = self._normalize_query_route_mode(contract.get("effective_source_policy"))
decision_origin = str(contract.get("decision_origin") or "query_semantics_contract").strip() or "query_semantics_contract"
clarification_text = None
if route_mode == "uncertain" and effective_source_policy != "external":
clarification_text = self._build_query_route_clarification_text()
probe_snapshot = {
"contract": {
"source_policy": route_mode,
"effective_source_policy": effective_source_policy,
"source_reason": str(contract.get("source_reason") or "query_semantics_contract").strip()
or "query_semantics_contract",
"decision_origin": decision_origin,
}
}
return {
"route_mode": route_mode,
"effective_source_policy": effective_source_policy,
"decision_origin": decision_origin,
"reply_contract": self._build_query_route_reply_contract(
route_mode=route_mode,
effective_source_policy=effective_source_policy,
clarification_text=clarification_text,
),
"clarification_text": clarification_text,
"query_profile": query_profile,
"probe_snapshot": probe_snapshot,
"probe_snapshot_summary": self._build_contract_snapshot_summary(
source_policy_hint=source_policy_hint,
query_semantics=semantics,
route_mode=route_mode,
effective_source_policy=effective_source_policy,
decision_origin=decision_origin,
),
}
async def preview_query_route_decision(
self,
*,
question: str,
asked_via: str,
project_id: Optional[int],
source_chat_id: Optional[str],
source_message_id: Optional[str] = None,
conversation_scope_key: Optional[str] = None,
reply_to_message_id: Optional[str] = None,
root_message_id: Optional[str] = None,
source_policy_hint: Optional[str] = None,
query_semantics: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:
semantics = dict(query_semantics or {})
if not semantics:
semantics = await self._resolve_query_semantics(
message=question,
source_system=asked_via,
metadata={
"project_id": project_id,
"source_chat_id": source_chat_id,
"source_message_id": source_message_id,
"conversation_scope_key": conversation_scope_key,
"reply_to_message_id": reply_to_message_id,
"root_message_id": root_message_id,
},
source_policy_hint=source_policy_hint,
)
semantics = dict(semantics or {})
decision = await self._run_query_route_gateway(
question=question,
asked_via=asked_via,
project_id=project_id,
source_chat_id=source_chat_id,
query_semantics=semantics,
source_policy_hint=source_policy_hint or semantics.get("source_policy"),
)
normalized_decision = self._normalize_query_route_decision(decision)
normalized_decision["probe_snapshot"] = dict(decision.get("probe_snapshot") or {})
return {
"query_semantics": self._apply_route_decision_to_query_semantics(
query_semantics=semantics,
route_decision=normalized_decision,
),
"route_decision": normalized_decision,
}
async def _persist_query_route_memory(
self,
*,
qa_record_id: Optional[int],
question_text: str,
project_id: Optional[int],
route_decision: Optional[dict[str, Any]],
) -> None:
normalized = self._normalize_query_route_decision(route_decision)
if qa_record_id is None or not str(question_text or "").strip():
return
if normalized["route_mode"] == "uncertain" and normalized["effective_source_policy"] == "uncertain":
# 澄清样本仍然保留，但不强制要求 embedding 成功。
pass
try:
row = (
await self.db.execute(
select(QueryRouteMemory).where(
and_(
QueryRouteMemory.tenant_id == self.tenant_id,
QueryRouteMemory.qa_record_id == int(qa_record_id),
)
)
)
).scalar_one_or_none()
except Exception:
row = None
question_embedding: list[float] = []
embedding_getter = getattr(getattr(self, "llm", None), "get_embedding_async", None)
if callable(embedding_getter):
try:
question_embedding = await embedding_getter(
str(question_text or "").strip(),
db=self.db,
tenant_id=self.tenant_id,
)
except Exception:
question_embedding = []
if row is None:
row = QueryRouteMemory(
tenant_id=self.tenant_id,
qa_record_id=int(qa_record_id),
project_id=project_id,
question_text=str(question_text or "").strip(),
route_mode_predicted=normalized["route_mode"],
route_mode_effective=normalized["effective_source_policy"],
decision_origin=normalized["decision_origin"],
reply_contract=normalized["reply_contract"],
probe_snapshot=normalized.get("probe_snapshot") or {},
query_profile=normalized["query_profile"],
feedback_outcome=None,
question_embedding=question_embedding or None,
)
self.db.add(row)
return
row.project_id = project_id
row.question_text = str(question_text or "").strip()
row.route_mode_predicted = normalized["route_mode"]
row.route_mode_effective = normalized["effective_source_policy"]
row.decision_origin = normalized["decision_origin"]
row.reply_contract = normalized["reply_contract"]
row.probe_snapshot = normalized.get("probe_snapshot") or {}
row.query_profile = normalized["query_profile"]
if question_embedding:
row.question_embedding = question_embedding
async def _update_query_route_feedback_outcome(
self,
*,
qa_record_id: Optional[int],
feedback_outcome: Optional[dict[str, Any]],
) -> None:
if qa_record_id is None or not isinstance(feedback_outcome, dict):
return
row = (
await self.db.execute(
select(QueryRouteMemory).where(
and_(
QueryRouteMemory.tenant_id == self.tenant_id,
QueryRouteMemory.qa_record_id == int(qa_record_id),
)
)
)
).scalar_one_or_none()
if row is None:
return
row.feedback_outcome = dict(feedback_outcome)
def _filter_sources_for_route(
self,
*,
sources: list[dict[str, Any]],
route_decision: Optional[dict[str, Any]],
) -> list[dict[str, Any]]:
decision = route_decision if isinstance(route_decision, dict) else {}
effective_source_policy = self._normalize_query_route_mode(decision.get("effective_source_policy"))
if effective_source_policy == "internal":
allowed = self._INTERNAL_ALLOWED_SOURCES
elif effective_source_policy == "external":
allowed = self._EXTERNAL_ALLOWED_SOURCES
else:
return list(sources or [])
filtered = [
item
for item in list(sources or [])
if isinstance(item, dict) and str(item.get("source") or "").strip().lower() in allowed
]
return filtered
def _render_answer_with_route_contract(
self,
*,
answer: str,
sources: list[dict[str, Any]],
route_decision: Optional[dict[str, Any]],
) -> str:
text = str(answer or "").strip()
if not text:
return text
decision = route_decision if isinstance(route_decision, dict) else {}
reply_contract = decision.get("reply_contract") if isinstance(decision.get("reply_contract"), dict) else {}
contract_mode = str(reply_contract.get("mode") or "").strip().lower()
if contract_mode == "clarify":
return str(reply_contract.get("clarification_text") or text).strip()
effective_source_policy = self._normalize_query_route_mode(decision.get("effective_source_policy"))
head = str(reply_contract.get("head") or "").strip()
tail_hint = str(reply_contract.get("tail_hint") or "").strip()
formatted = text if not head or text.startswith(head) else f"{head}\n{text}"
labels: list[str] = []
for item in self._filter_sources_for_route(sources=sources, route_decision=decision)[:3]:
if not isinstance(item, dict):
continue
source = str(item.get("source") or "").strip().lower()
if effective_source_policy == "external" and source == "system":
continue
label = str(item.get("label") or item.get("source") or "").strip()
if label and label not in labels:
labels.append(label)
if labels and "来源摘要：" not in formatted:
formatted = f"{formatted}\n\n来源摘要：{'；'.join(labels[:3])}"
if tail_hint and tail_hint not in formatted:
formatted = f"{formatted}\n\n{tail_hint}"
return formatted.strip()
def _build_route_clarification_payload(
self,
*,
route_decision: Optional[dict[str, Any]],
) -> Optional[dict[str, Any]]:
normalized = self._normalize_query_route_decision(route_decision)
reply_contract = (
normalized.get("reply_contract")
if isinstance(normalized.get("reply_contract"), dict)
else {}
)
if str(reply_contract.get("mode") or "").strip().lower() != "clarify":
return None
clarification_text = str(reply_contract.get("clarification_text") or "").strip()
if not clarification_text:
return None
return {
"answer": clarification_text,
"code": "query_route_clarify",
"candidates": [],
"missing_fields": ["query_route"],
}

8
backend/app/services/adaptive_rag_service/mixin_retrieval_graph.py
View File

@ -50,7 +50,13 @@ class AdaptiveRAGRetrievalGraphMixin:
).scalars().all()
if not seed_entities:
terms = self._extract_query_terms(question, preferred=self._extract_project_lookup_keyword(question))
term_collector = getattr(self, "_collect_lookup_terms", None)
terms = term_collector(question=question, limit=6) if callable(term_collector) else []
if not terms:
normalizer = getattr(self, "_normalize_lookup_text", None)
normalized = normalizer(question) if callable(normalizer) else str(question or "").strip()
if normalized:
terms = [normalized[:32]]
if terms:
conds = []
for term in terms[:6]:

203
backend/app/services/adaptive_rag_service/mixin_retrieval_router_db.py
View File

@ -5,6 +5,167 @@ from .shared import *
class AdaptiveRAGRetrievalRouterDBMixin:
async def _retrieve_generic_semantic_fact_rows(
self,
*,
entity_type: str,
question: str,
query_semantics: Optional[dict[str, Any]] = None,
) -> list[RetrievedItem]:
term_collector = getattr(self, "_collect_lookup_terms", None)
query_terms = term_collector(
question=question,
query_semantics=query_semantics,
operation_contract=(
query_semantics.get("operation_contract")
if isinstance(query_semantics, dict) and isinstance(query_semantics.get("operation_contract"), dict)
else None
),
target_resolution=(
query_semantics.get("target_resolution")
if isinstance(query_semantics, dict) and isinstance(query_semantics.get("target_resolution"), dict)
else None
),
limit=4,
) if callable(term_collector) else []
if not query_terms:
return []
fact_query = getattr(self, "_query_semantic_facts_by_lookup_terms", None)
if not callable(fact_query):
return []
matched_facts, used_keyword = await fact_query(
entity_type=entity_type,
query_terms=query_terms,
entity_id=None,
limit=10,
)
if not matched_facts:
return []
grouped: dict[int, list[SemanticFact]] = {}
for row in matched_facts:
try:
entity_id = int(row.entity_id)
except Exception:
continue
grouped.setdefault(entity_id, []).append(row)
if not grouped:
return []
preview = "；".join(
f"#{entity_id} " + " / ".join(
str(item.fact_value_text or "").strip()
for item in rows[:3]
if str(item.fact_value_text or "").strip()
)
for entity_id, rows in list(grouped.items())[:8]
)
metadata = self._normalize_source_registry_metadata(
source_type="db",
metadata={
"table": "semantic_facts",
"entity_type": entity_type,
"query_keyword": used_keyword,
"match_count": len(grouped),
"fact_keys": sorted({str(row.fact_key or "") for row in matched_facts if str(row.fact_key or "").strip()}),
},
)
return [
RetrievedItem(
source="db",
content=f"语义事实命中{entity_type}：" + preview,
score=0.96,
source_label="Semantic facts",
metadata=metadata,
)
]
async def _retrieve_project_semantic_fact_rows(
self,
*,
question: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]] = None,
) -> list[RetrievedItem]:
term_collector = getattr(self, "_collect_lookup_terms", None)
query_terms = term_collector(
question=question,
query_semantics=query_semantics,
operation_contract=(
query_semantics.get("operation_contract")
if isinstance(query_semantics, dict) and isinstance(query_semantics.get("operation_contract"), dict)
else None
),
target_resolution=(
query_semantics.get("target_resolution")
if isinstance(query_semantics, dict) and isinstance(query_semantics.get("target_resolution"), dict)
else None
),
limit=4,
) if callable(term_collector) else []
if not query_terms:
return []
fact_query = getattr(self, "_query_project_semantic_facts_by_lookup_terms", None)
if not callable(fact_query):
return []
matched_facts, used_keyword = await fact_query(
query_terms=query_terms,
project_id=project_id,
limit=10,
)
if not matched_facts:
return []
matched_project_ids: list[int] = []
for fact in matched_facts:
try:
entity_id = int(fact.entity_id)
except Exception:
continue
if entity_id not in matched_project_ids:
matched_project_ids.append(entity_id)
if not matched_project_ids:
return []
projects = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id.in_(matched_project_ids),
)
)
)
).scalars().all()
if not projects:
return []
metadata = self._normalize_source_registry_metadata(
source_type="db",
metadata={
"table": "semantic_facts",
"query_keyword": used_keyword,
"match_count": len(projects),
"matched_project_ids": matched_project_ids,
"fact_keys": sorted({str(row.fact_key or "") for row in matched_facts if str(row.fact_key or "").strip()}),
},
explicit_project_id=matched_project_ids[0] if len(matched_project_ids) == 1 else None,
)
return [
RetrievedItem(
source="db",
content="语义事实命中项目：" + "；".join(
f"#{int(row.id)} {str(row.title or row.name or '未命名项目')}（状态:{str(row.status or 'unknown')}）"
for row in projects[:10]
),
score=0.97,
source_label="Semantic facts",
metadata=metadata,
)
]
async def _retrieve_structured_db(
self,
question: str,
@ -16,7 +177,33 @@ class AdaptiveRAGRetrievalRouterDBMixin:
rows: list[RetrievedItem] = []
subject_focus = self._query_subject_focus(query_semantics)
scope_hint = self._query_scope_hint(query_semantics)
target_resolution = query_semantics.get("target_resolution") if isinstance(query_semantics, dict) and isinstance(query_semantics.get("target_resolution"), dict) else {}
target_entity_type = str(target_resolution.get("target_entity_type") or "").strip().lower()
query_plan = query_semantics.get("query_plan") if isinstance(query_semantics, dict) and isinstance(query_semantics.get("query_plan"), dict) else {}
planned_entity_type = str(query_plan.get("entity_type") or "").strip().lower()
project_subject_requested = subject_focus == "project" or target_entity_type == "project" or planned_entity_type == "project"
project_scope_required = scope_hint == "project" or project_id is not None
semantic_fact_entity_type = ""
if target_entity_type in {"task", "signal", "release", "review", "project", "document", "user", "requirement"}:
semantic_fact_entity_type = target_entity_type
elif planned_entity_type in {"task", "signal", "release", "review", "project", "document", "user", "requirement"}:
semantic_fact_entity_type = planned_entity_type
elif subject_focus == "task":
semantic_fact_entity_type = "task"
elif subject_focus == "signal":
semantic_fact_entity_type = "signal"
elif subject_focus == "people":
semantic_fact_entity_type = "user"
elif subject_focus == "charter":
semantic_fact_entity_type = "requirement"
if semantic_fact_entity_type and semantic_fact_entity_type != "project":
rows.extend(
await self._retrieve_generic_semantic_fact_rows(
entity_type=semantic_fact_entity_type,
question=question,
query_semantics=query_semantics,
)
)
if subject_focus == "people":
scoped_project_id: Optional[int] = None
@ -74,9 +261,21 @@ class AdaptiveRAGRetrievalRouterDBMixin:
)
)
if subject_focus == "project":
if project_subject_requested:
semantic_fact_rows = await self._retrieve_project_semantic_fact_rows(
question=question,
project_id=project_id,
query_semantics=query_semantics,
)
rows.extend(semantic_fact_rows)
stmt = select(Project).where(Project.tenant_id == self.tenant_id)
if project_id:
resolved_project_ref = str(target_resolution.get("target_ref") or "").strip()
if resolved_project_ref.startswith("project:"):
try:
stmt = stmt.where(Project.id == int(resolved_project_ref.split(":", 1)[1]))
except Exception:
pass
elif project_id:
stmt = stmt.where(Project.id == project_id)
projects = (await self.db.execute(stmt.limit(8))).scalars().all()
if projects:

7
backend/app/services/adaptive_rag_service/mixin_session_state.py
View File

@ -176,9 +176,10 @@ class AdaptiveRAGSessionStateMixin:
return f"{subject_name}->{object_name}"[:255]
text = str(question or "").strip()
keyword = self._extract_project_lookup_keyword(text)
if len(keyword) >= 2:
return keyword[:255]
normalizer = getattr(self, "_normalize_lookup_text", None)
normalized_question = normalizer(text) if callable(normalizer) else text
if 2 <= len(normalized_question) <= 64:
return normalized_question[:255]
answer_text = str(answer or "").strip()
m = re.search(r"项目[「\"]?([^」\"\n]{2,40})", answer_text)

766
backend/app/services/adaptive_rag_service/mixin_target_resolution.py Normal file
View File

@ -0,0 +1,766 @@
"""Adaptive RAG internal target resolution helpers."""
from __future__ import annotations
from .shared import *
class AdaptiveRAGTargetResolutionMixin:
_TARGET_ENTITY_TYPES = {"project", "feature", "config", "doc", "person", "signal", "unknown"}
_TARGET_RESOLUTION_MODES = {"deterministic", "llm_rerank", "semantic_filter", "clarify", "none"}
@classmethod
def _normalize_object_mentions(
cls,
raw: Any,
*,
question: str = "",
) -> list[str]:
_ = (cls, question)
mentions: list[str] = []
rows = raw if isinstance(raw, list) else []
for item in rows:
text = str(item or "").strip(" ，。,.!?？：:;；“”\"'()（）")
text = QUESTION_NOISE_RE.sub("", text).strip()
if len(text) >= 2 and text not in mentions:
mentions.append(text[:32])
return mentions[:4]
def _build_operation_contract(
self,
*,
question: str,
query_semantics: Optional[dict[str, Any]],
) -> dict[str, Any]:
_ = question
semantics = query_semantics if isinstance(query_semantics, dict) else {}
query_type = str(semantics.get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower() or QUERY_TYPE_UNKNOWN
query_profile = dict(semantics.get("query_profile") or {}) if isinstance(semantics.get("query_profile"), dict) else {}
referenced_objects = {
str(item or "").strip().lower()
for item in list(query_profile.get("referenced_objects") or [])
if str(item or "").strip()
}
if query_type in {QUERY_TYPE_EXISTENCE, QUERY_TYPE_UNKNOWN} and referenced_objects.intersection(
{"model_routing", "system_config"}
):
query_type = QUERY_TYPE_STATUS
object_mentions = self._normalize_object_mentions(
semantics.get("object_mentions"),
question=question,
)
return {
"query_type": query_type,
"answer_contract": (
self._internal_answer_contract_for_query_type(query_type)
if hasattr(self, "_internal_answer_contract_for_query_type")
else "clarify_or_fail"
),
"query_profile": query_profile,
"object_mentions": object_mentions,
"clarify_policy": "ask_when_target_ambiguous",
}
@classmethod
def _normalize_target_candidate(cls, payload: Optional[dict[str, Any]]) -> Optional[dict[str, Any]]:
row = payload if isinstance(payload, dict) else {}
entity_type = str(row.get("target_entity_type") or row.get("entity_type") or "").strip().lower()
if entity_type not in cls._TARGET_ENTITY_TYPES:
return None
score = max(0.0, min(1.0, float(row.get("score") or 0.0)))
return {
"target_entity_type": entity_type,
"target_ref": str(row.get("target_ref") or "").strip() or None,
"resolved_label": str(row.get("resolved_label") or row.get("display_name") or "").strip() or None,
"score": round(score, 4),
"source_kind": str(row.get("source_kind") or "unknown").strip().lower() or "unknown",
"match_label": str(row.get("match_label") or "").strip() or None,
"source_confidence": round(max(0.0, min(1.0, float(row.get("source_confidence") or score or 0.0))), 4),
}
@staticmethod
def _target_candidate_identity(row: dict[str, Any]) -> str:
return "|".join(
[
str(row.get("target_entity_type") or ""),
str(row.get("target_ref") or ""),
str(row.get("resolved_label") or ""),
]
)
async def _collect_semantic_fact_target_candidates(
self,
*,
mentions: list[str],
project_id: Optional[int],
operation_contract: dict[str, Any],
) -> list[dict[str, Any]]:
if not mentions:
return []
candidates: list[dict[str, Any]] = []
fact_query = getattr(self, "_query_project_semantic_facts_by_lookup_terms", None)
if not callable(fact_query):
return []
for mention in mentions[:3]:
fact_rows, _used_keyword = await fact_query(
query_terms=[mention],
project_id=project_id,
limit=10,
)
project_ids: list[int] = []
fact_confidence_by_project: dict[int, float] = {}
for row in fact_rows:
try:
entity_id = int(row.entity_id)
except Exception:
continue
if entity_id not in project_ids:
project_ids.append(entity_id)
fact_confidence_by_project[entity_id] = max(
float(fact_confidence_by_project.get(entity_id) or 0.0),
float(getattr(row, "confidence", 0.0) or 0.0),
)
if not project_ids:
continue
project_rows = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id.in_(project_ids),
)
)
)
).scalars().all()
for project in project_rows:
confidence = max(0.88, min(0.99, float(fact_confidence_by_project.get(int(project.id)) or 0.0)))
candidates.append(
{
"target_entity_type": "project",
"target_ref": f"project:{int(project.id)}",
"resolved_label": str(project.title or project.name or f"项目#{int(project.id)}"),
"score": round(min(0.99, confidence + 0.03), 4),
"source_kind": "semantic_facts",
"match_label": mention,
"source_confidence": round(confidence, 4),
}
)
return candidates
async def _collect_project_target_candidates(
self,
*,
mentions: list[str],
project_id: Optional[int],
) -> list[dict[str, Any]]:
if not mentions:
return []
candidates: list[dict[str, Any]] = []
similarity_scorer = getattr(self, "_lookup_similarity_score", None)
for mention in mentions[:3]:
matched_rows: list[Project] = []
searcher = getattr(self, "_search_projects_by_keyword", None)
if callable(searcher):
matched_rows = await searcher(keyword=mention, project_id=project_id, limit=6)
else:
stmt = select(Project).where(Project.tenant_id == self.tenant_id)
if project_id is not None:
stmt = stmt.where(Project.id == int(project_id))
like = f"%{mention}%"
stmt = stmt.where(
or_(
Project.title.ilike(like),
Project.name.ilike(like),
Project.description.ilike(like),
)
)
matched_rows = (await self.db.execute(stmt.order_by(Project.updated_at.desc()).limit(6))).scalars().all()
if not matched_rows:
stmt = select(Project).where(Project.tenant_id == self.tenant_id)
if project_id is not None:
stmt = stmt.where(Project.id == int(project_id))
project_rows = (
await self.db.execute(
stmt.order_by(Project.updated_at.desc(), Project.id.desc()).limit(120)
)
).scalars().all()
scored_rows: list[tuple[float, Project]] = []
for project in project_rows:
title = str(project.title or "").strip()
name = str(project.name or "").strip()
description = str(project.description or "").strip()
if callable(similarity_scorer):
score = max(
similarity_scorer(mention, title),
similarity_scorer(mention, name),
similarity_scorer(mention, description) * 0.8,
)
else:
lowered = mention.lower()
title_l = title.lower()
name_l = name.lower()
description_l = description.lower()
score = 0.0
if lowered and (lowered in title_l or lowered in name_l):
score = 0.9
elif lowered and lowered in description_l:
score = 0.76
if score >= 0.62:
scored_rows.append((score, project))
scored_rows.sort(key=lambda item: item[0], reverse=True)
matched_rows = [row for _, row in scored_rows[:6]]
for project in matched_rows:
title = str(project.title or "").strip().lower()
name = str(project.name or "").strip().lower()
description = str(project.description or "").strip().lower()
lowered = mention.lower()
score = 0.82
if callable(similarity_scorer):
score = max(
0.74,
similarity_scorer(mention, title) if title else 0.0,
similarity_scorer(mention, name) if name else 0.0,
(similarity_scorer(mention, description) * 0.8) if description else 0.0,
)
elif lowered and (lowered in title or lowered in name):
score = 0.91
elif lowered and lowered in description:
score = 0.86
candidates.append(
{
"target_entity_type": "project",
"target_ref": f"project:{int(project.id)}",
"resolved_label": str(project.title or project.name or f"项目#{int(project.id)}"),
"score": round(score, 4),
"source_kind": "project_db",
"match_label": mention,
"source_confidence": round(score, 4),
}
)
return candidates
def _collect_feature_target_candidates(
self,
*,
question: str,
mentions: list[str],
system_snapshot: Optional[dict[str, Any]],
) -> list[dict[str, Any]]:
matcher = getattr(self, "_match_feature_catalog_entries", None)
if not callable(matcher):
return []
search_text = " ".join(mentions[:2]) if mentions else question
rows = matcher(
question=search_text,
system_snapshot=system_snapshot,
limit=4,
)
candidates: list[dict[str, Any]] = []
for row in rows:
name = str(row.get("name") or "").strip()
route = str(row.get("route") or "").strip()
match_score = max(0.0, min(1.0, float(row.get("match_score") or 0.0)))
if not name and not route:
continue
score = min(0.92, max(0.72, 0.62 + match_score * 0.25))
target_key = route or name
candidates.append(
{
"target_entity_type": "feature",
"target_ref": f"feature:{target_key}",
"resolved_label": name or target_key,
"score": round(score, 4),
"source_kind": "feature_catalog",
"match_label": name or target_key,
"source_confidence": round(match_score or score, 4),
}
)
return candidates
def _collect_config_target_candidates(
self,
*,
query_semantics: Optional[dict[str, Any]],
operation_contract: dict[str, Any],
) -> list[dict[str, Any]]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
query_profile = semantics.get("query_profile") if isinstance(semantics.get("query_profile"), dict) else {}
referenced_objects = {
str(item or "").strip().lower()
for item in list(query_profile.get("referenced_objects") or [])
if str(item or "").strip()
}
if not {"model_routing", "system_config"}.intersection(referenced_objects):
return []
mentions = list(operation_contract.get("object_mentions") or [])
rows: list[dict[str, Any]] = []
if "model_routing" in referenced_objects:
rows.append(
{
"target_entity_type": "config",
"target_ref": "config:model_routing",
"resolved_label": mentions[0] if mentions else "模型配置",
"score": 0.9,
"source_kind": "system_config",
"match_label": mentions[0] if mentions else "模型配置",
"source_confidence": 0.9,
}
)
elif "system_config" in referenced_objects:
rows.append(
{
"target_entity_type": "config",
"target_ref": "config:system",
"resolved_label": mentions[0] if mentions else "系统配置",
"score": 0.86,
"source_kind": "system_config",
"match_label": mentions[0] if mentions else "系统配置",
"source_confidence": 0.86,
}
)
return rows
async def _collect_kg_target_candidates(
self,
*,
question: str,
mentions: list[str],
project_id: Optional[int],
) -> list[dict[str, Any]]:
resolver = getattr(self, "entity_resolution", None)
if not callable(resolver):
return []
resolution = await resolver(
question=" ".join(mentions[:2]) if mentions else question,
project_id=project_id,
limit=6,
)
matched_entities = resolution.get("matched_entities") if isinstance(resolution, dict) else []
if not isinstance(matched_entities, list) or not matched_entities:
return []
ids: list[int] = []
matched_score: dict[int, float] = {}
for row in matched_entities:
if not isinstance(row, dict):
continue
try:
entity_id = int(row.get("entity_id"))
except Exception:
continue
ids.append(entity_id)
matched_score[entity_id] = max(
float(matched_score.get(entity_id) or 0.0),
float(row.get("score") or 0.0),
)
if not ids:
return []
rows = (
await self.db.execute(
select(KGEntity).where(
and_(
KGEntity.tenant_id == self.tenant_id,
KGEntity.id.in_(ids),
)
)
)
).scalars().all()
candidates: list[dict[str, Any]] = []
for row in rows:
payload = self._safe_dict(row.payload) if hasattr(self, "_safe_dict") else (row.payload if isinstance(row.payload, dict) else {})
entity_score = max(0.0, min(1.0, float(matched_score.get(int(row.id)) or 0.0)))
project_ref = payload.get("project_id")
if project_ref is not None:
try:
project_ref = int(project_ref)
except Exception:
project_ref = None
if project_ref is not None:
candidates.append(
{
"target_entity_type": "project",
"target_ref": f"project:{int(project_ref)}",
"resolved_label": str(row.display_name or row.entity_key or f"项目#{int(project_ref)}"),
"score": round(min(0.89, max(0.68, 0.58 + entity_score * 0.25)), 4),
"source_kind": "knowledge_graph",
"match_label": str(row.display_name or row.entity_key or "").strip() or None,
"source_confidence": round(entity_score or 0.68, 4),
}
)
return candidates
async def _collect_internal_target_candidates(
self,
*,
question: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]],
system_snapshot: Optional[dict[str, Any]],
operation_contract: dict[str, Any],
) -> list[dict[str, Any]]:
_ = query_semantics
mentions = list(operation_contract.get("object_mentions") or [])
providers = await asyncio.gather(
self._collect_semantic_fact_target_candidates(
mentions=mentions,
project_id=project_id,
operation_contract=operation_contract,
),
self._collect_project_target_candidates(
mentions=mentions,
project_id=project_id,
),
self._collect_kg_target_candidates(
question=question,
mentions=mentions,
project_id=project_id,
),
)
rows: list[dict[str, Any]] = []
for bucket in providers:
for item in bucket:
normalized = self._normalize_target_candidate(item)
if normalized is not None:
rows.append(normalized)
for item in self._collect_feature_target_candidates(
question=question,
mentions=mentions,
system_snapshot=system_snapshot,
):
normalized = self._normalize_target_candidate(item)
if normalized is not None:
rows.append(normalized)
for item in self._collect_config_target_candidates(
query_semantics=query_semantics,
operation_contract=operation_contract,
):
normalized = self._normalize_target_candidate(item)
if normalized is not None:
rows.append(normalized)
deduped: dict[str, dict[str, Any]] = {}
for row in rows:
key = self._target_candidate_identity(row)
existing = deduped.get(key)
if existing is None or float(row.get("score") or 0.0) > float(existing.get("score") or 0.0):
deduped[key] = row
return sorted(deduped.values(), key=lambda item: float(item.get("score") or 0.0), reverse=True)[:8]
async def _rerank_internal_target_candidates(
self,
*,
question: str,
operation_contract: dict[str, Any],
candidates: list[dict[str, Any]],
) -> Optional[dict[str, Any]]:
if len(candidates) < 2 or not hasattr(self, "llm"):
return None
prompt = PROMPT_INTERNAL_TARGET_RESOLUTION.format(
question=str(question or "")[:600],
operation_contract=json.dumps(operation_contract, ensure_ascii=False)[:1000],
candidates=json.dumps(
[
{
"index": idx,
"target_entity_type": str(item.get("target_entity_type") or ""),
"target_ref": str(item.get("target_ref") or ""),
"resolved_label": str(item.get("resolved_label") or ""),
"score": float(item.get("score") or 0.0),
"source_kind": str(item.get("source_kind") or ""),
}
for idx, item in enumerate(candidates[:4])
],
ensure_ascii=False,
),
).strip()
try:
raw = await self.llm.generate(
prompt,
db=self.db,
tenant_id=self.tenant_id,
function_name="internal_target_resolution",
temperature=0.0,
max_tokens=220,
use_prompt_profile=False,
)
except Exception:
return None
parser = getattr(self, "_parse_json_object_safe", None)
parsed = parser(raw) if callable(parser) else {}
if not isinstance(parsed, dict):
return None
if bool(parsed.get("clarify")):
return {
"target_entity_type": "unknown",
"target_ref": None,
"resolved_label": None,
"resolution_mode": "clarify",
"confidence": round(max(0.0, min(1.0, float(parsed.get("confidence") or 0.0))), 4),
"candidate_summary": candidates[:4],
}
try:
selected_index = int(parsed.get("selected_index"))
except Exception:
return None
if selected_index < 0 or selected_index >= len(candidates[:4]):
return None
selected = dict(candidates[selected_index])
return {
"target_entity_type": str(selected.get("target_entity_type") or "unknown"),
"target_ref": selected.get("target_ref"),
"resolved_label": selected.get("resolved_label"),
"resolution_mode": "llm_rerank",
"confidence": round(max(0.0, min(1.0, float(parsed.get("confidence") or selected.get("score") or 0.0))), 4),
"candidate_summary": candidates[:4],
}
def _build_semantic_filter_target(
self,
*,
query_semantics: Optional[dict[str, Any]],
operation_contract: dict[str, Any],
) -> Optional[dict[str, Any]]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
subject_focus = str(semantics.get("subject_focus") or "unknown").strip().lower()
query_profile = semantics.get("query_profile") if isinstance(semantics.get("query_profile"), dict) else {}
mentions = list(operation_contract.get("object_mentions") or [])
resolved_label = mentions[0] if mentions else None
referenced_objects = {
str(item or "").strip().lower()
for item in list(query_profile.get("referenced_objects") or [])
if str(item or "").strip()
}
if {"model_routing", "system_config"}.intersection(referenced_objects):
target_ref = "config:model_routing" if "model_routing" in referenced_objects else "config:system"
return {
"target_entity_type": "config",
"target_ref": target_ref,
"resolved_label": resolved_label or ("模型配置" if target_ref.endswith("model_routing") else "系统配置"),
"resolution_mode": "deterministic",
"confidence": 0.84,
"candidate_summary": [],
}
if subject_focus == "project":
return {
"target_entity_type": "project",
"target_ref": None,
"resolved_label": resolved_label,
"resolution_mode": "semantic_filter",
"confidence": 0.78 if resolved_label else 0.64,
"candidate_summary": [],
}
if subject_focus == "feature" or "feature_catalog" in referenced_objects:
return {
"target_entity_type": "feature",
"target_ref": None,
"resolved_label": resolved_label,
"resolution_mode": "semantic_filter",
"confidence": 0.74 if resolved_label else 0.62,
"candidate_summary": [],
}
return None
async def _resolve_internal_target(
self,
*,
question: str,
project_id: Optional[int],
query_semantics: Optional[dict[str, Any]],
system_snapshot: Optional[dict[str, Any]],
operation_contract: Optional[dict[str, Any]],
) -> dict[str, Any]:
semantics = query_semantics if isinstance(query_semantics, dict) else {}
operation = operation_contract if isinstance(operation_contract, dict) else {}
source_policy = str(
semantics.get("effective_source_policy")
or semantics.get("source_policy")
or ""
).strip().lower()
if source_policy != "internal":
return {
"target_entity_type": "unknown",
"target_ref": None,
"resolved_label": None,
"resolution_mode": "none",
"confidence": 0.0,
"candidate_summary": [],
}
candidates = await self._collect_internal_target_candidates(
question=question,
project_id=project_id,
query_semantics=semantics,
system_snapshot=system_snapshot,
operation_contract=operation,
)
if not candidates:
fallback = self._build_semantic_filter_target(
query_semantics=semantics,
operation_contract=operation,
)
if fallback is not None:
return fallback
return {
"target_entity_type": "unknown",
"target_ref": None,
"resolved_label": None,
"resolution_mode": "clarify",
"confidence": 0.0,
"candidate_summary": [],
}
top = candidates[0]
second = candidates[1] if len(candidates) > 1 else None
margin = float(top.get("score") or 0.0) - float(second.get("score") or 0.0) if second else 1.0
if second and str(top.get("target_entity_type") or "") != str(second.get("target_entity_type") or "") and margin < 0.1:
reranked = await self._rerank_internal_target_candidates(
question=question,
operation_contract=operation,
candidates=candidates,
)
if reranked is not None:
return reranked
if float(top.get("score") or 0.0) >= 0.88 or margin >= 0.12 or second is None:
return {
"target_entity_type": str(top.get("target_entity_type") or "unknown"),
"target_ref": top.get("target_ref"),
"resolved_label": top.get("resolved_label"),
"resolution_mode": "deterministic",
"confidence": round(max(0.0, min(1.0, float(top.get("score") or 0.0))), 4),
"candidate_summary": candidates[:4],
}
if str(top.get("target_entity_type") or "") == str(second.get("target_entity_type") or ""):
return {
"target_entity_type": str(top.get("target_entity_type") or "unknown"),
"target_ref": top.get("target_ref"),
"resolved_label": top.get("resolved_label"),
"resolution_mode": "deterministic",
"confidence": round(max(0.0, min(1.0, float(top.get("score") or 0.0))), 4),
"candidate_summary": candidates[:4],
}
return {
"target_entity_type": "unknown",
"target_ref": None,
"resolved_label": None,
"resolution_mode": "clarify",
"confidence": round(max(0.0, min(1.0, float(top.get("score") or 0.0))), 4),
"candidate_summary": candidates[:4],
}
def _patch_scene_semantics_with_target_resolution(
self,
*,
scene_semantics: Optional[dict[str, Any]],
operation_contract: Optional[dict[str, Any]],
target_resolution: Optional[dict[str, Any]],
) -> dict[str, Any]:
normalized = dict(scene_semantics or {})
query_type = str((operation_contract or {}).get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower()
target_entity_type = str((target_resolution or {}).get("target_entity_type") or "unknown").strip().lower()
snapshot_focus = str(normalized.get("snapshot_focus") or "none").strip().lower()
if query_type == QUERY_TYPE_EXISTENCE and target_entity_type == "project":
normalized["snapshot_focus"] = "none"
return normalized
if target_entity_type not in {"feature", "config"} and snapshot_focus == "feature":
normalized["snapshot_focus"] = "none"
return normalized
return normalized
def _should_allow_snapshot_direct_answer(
self,
*,
scene_semantics: Optional[dict[str, Any]],
operation_contract: Optional[dict[str, Any]],
target_resolution: Optional[dict[str, Any]],
) -> bool:
snapshot_focus = str((scene_semantics or {}).get("snapshot_focus") or "none").strip().lower()
query_type = str((operation_contract or {}).get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower()
target_entity_type = str((target_resolution or {}).get("target_entity_type") or "unknown").strip().lower()
if snapshot_focus in {"identity", "architecture"}:
return query_type in {QUERY_TYPE_EXPLAIN, QUERY_TYPE_STATUS}
if snapshot_focus == "feature":
return query_type in {QUERY_TYPE_EXPLAIN, QUERY_TYPE_STATUS, QUERY_TYPE_LIST} and target_entity_type in {
"feature",
"config",
"unknown",
}
return False
@staticmethod
def _build_operation_contract_summary(operation_contract: Optional[dict[str, Any]]) -> dict[str, Any]:
payload = operation_contract if isinstance(operation_contract, dict) else {}
mentions = [
str(item or "").strip()
for item in list(payload.get("object_mentions") or [])
if str(item or "").strip()
]
return {
"query_type": str(payload.get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower() or QUERY_TYPE_UNKNOWN,
"answer_contract": str(payload.get("answer_contract") or "clarify_or_fail").strip() or "clarify_or_fail",
"object_mentions": mentions[:4],
"clarify_policy": str(payload.get("clarify_policy") or "").strip() or None,
}
@staticmethod
def _build_target_resolution_summary(target_resolution: Optional[dict[str, Any]]) -> dict[str, Any]:
payload = target_resolution if isinstance(target_resolution, dict) else {}
candidates: list[dict[str, Any]] = []
for item in list(payload.get("candidate_summary") or [])[:4]:
if not isinstance(item, dict):
continue
candidates.append(
{
"target_entity_type": str(item.get("target_entity_type") or "unknown").strip().lower() or "unknown",
"target_ref": str(item.get("target_ref") or "").strip() or None,
"resolved_label": str(item.get("resolved_label") or "").strip() or None,
"score": round(max(0.0, min(1.0, float(item.get("score") or 0.0))), 4),
"source_kind": str(item.get("source_kind") or "unknown").strip().lower() or "unknown",
}
)
return {
"target_entity_type": str(payload.get("target_entity_type") or "unknown").strip().lower() or "unknown",
"target_ref": str(payload.get("target_ref") or "").strip() or None,
"resolved_label": str(payload.get("resolved_label") or "").strip() or None,
"resolution_mode": str(payload.get("resolution_mode") or "none").strip().lower() or "none",
"confidence": round(max(0.0, min(1.0, float(payload.get("confidence") or 0.0))), 4),
"candidate_summary": candidates,
}
@staticmethod
def _build_query_plan_summary(query_plan: Optional[dict[str, Any]]) -> dict[str, Any]:
payload = query_plan if isinstance(query_plan, dict) else {}
filters: list[dict[str, Any]] = []
for item in list(payload.get("filters") or [])[:4]:
if not isinstance(item, dict):
continue
filters.append(
{
"field": str(item.get("field") or "").strip() or None,
"operator": str(item.get("operator") or "").strip() or None,
"value": item.get("value"),
"value_source": str(item.get("value_source") or "").strip() or None,
}
)
return {
"plan_version": str(payload.get("plan_version") or "unknown").strip() or "unknown",
"query_type": str(payload.get("query_type") or QUERY_TYPE_UNKNOWN).strip().lower() or QUERY_TYPE_UNKNOWN,
"entity_type": str(payload.get("entity_type") or "unknown").strip().lower() or "unknown",
"target_ref": str(payload.get("target_ref") or "").strip() or None,
"answer_contract": str(payload.get("answer_contract") or "").strip() or None,
"filters": filters,
}
def _inject_internal_query_runtime_summaries(
self,
*,
retrieval_metrics: Optional[dict[str, Any]],
query_semantics: Optional[dict[str, Any]],
) -> dict[str, Any]:
metrics = dict(retrieval_metrics or {})
semantics = query_semantics if isinstance(query_semantics, dict) else {}
metrics["operation_contract_summary"] = self._build_operation_contract_summary(semantics.get("operation_contract"))
metrics["target_resolution_summary"] = self._build_target_resolution_summary(semantics.get("target_resolution"))
metrics["query_plan_summary"] = self._build_query_plan_summary(semantics.get("query_plan"))
return metrics
__all__ = ["AdaptiveRAGTargetResolutionMixin"]

4
backend/app/services/adaptive_rag_service/service.py
View File

@ -40,12 +40,15 @@ from .mixin_conversation_utils_a_p1 import AdaptiveRAGConversationUtilsAMixinP1
from .mixin_conversation_utils_a_p2 import AdaptiveRAGConversationUtilsAMixinP2
from .mixin_conversation_utils_b import AdaptiveRAGConversationUtilsBMixin
from .mixin_followup_anchor import AdaptiveRAGFollowupAnchorMixin
from .mixin_query_route_gateway import AdaptiveRAGQueryRouteGatewayMixin
from .mixin_target_resolution import AdaptiveRAGTargetResolutionMixin
class AdaptiveRAGService(
AdaptiveRAGCoreIntentMixin,
AdaptiveRAGAnswerEntryMixin,
AdaptiveRAGAnswerMainlineMixin,
AdaptiveRAGTargetResolutionMixin,
AdaptiveRAGAnswerPrecheckMixin,
AdaptiveRAGAnswerRetrievalMixin,
AdaptiveRAGAnswerPayloadMixin,
@ -82,6 +85,7 @@ class AdaptiveRAGService(
AdaptiveRAGConversationUtilsAMixinP2,
AdaptiveRAGConversationUtilsBMixin,
AdaptiveRAGFollowupAnchorMixin,
AdaptiveRAGQueryRouteGatewayMixin,
):
"""项目内统一 RAG 服务。"""
_symbol_snapshot_cache: dict[str, tuple[float, dict[str, Any]]] = {}

5
backend/app/services/adaptive_rag_service/shared_imports.py
View File

@ -12,6 +12,7 @@ from __future__ import annotations
import ast
import asyncio
from difflib import SequenceMatcher
import hashlib
import json
import math
@ -19,6 +20,7 @@ import os
from pathlib import Path
import re
import time
from urllib.parse import urlparse
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Optional
@ -41,6 +43,7 @@ from ...config.ai_prompts import (
PROMPT_RAG_FOLLOWUP_ANCHOR_SEMANTICS,
PROMPT_RAG_QUERY_SEMANTICS,
PROMPT_RAG_ANSWER_SCENE_SEMANTICS,
PROMPT_INTERNAL_TARGET_RESOLUTION,
)
from ...core.database import AsyncSessionLocal
from ...models import (
@ -56,8 +59,10 @@ from ...models import (
ProjectSignalLink,
QAFeedback,
QARecord,
QueryRouteMemory,
QASession,
QASessionState,
SemanticFact,
Signal,
Task,
TruthNode,

3
backend/app/services/adaptive_rag_service/shared_part_02.py
View File

@ -24,8 +24,11 @@ DEEP_CONTEXT_BLOCKS = _env_int("RAG_DEEP_CONTEXT_BLOCKS", 16, minimum=8, maximum
DEEP_ANSWER_MAX_TOKENS = _env_int("RAG_DEEP_ANSWER_MAX_TOKENS", 1400, minimum=900, maximum=3000)
QUERY_TYPE_COUNT = "count"
QUERY_TYPE_LIST = "list"
QUERY_TYPE_EXISTENCE = "existence"
QUERY_TYPE_STATUS = "status"
QUERY_TYPE_EXPLAIN = "explain"
QUERY_TYPE_DIAGNOSE = "diagnose"
QUERY_TYPE_COMPARE = "compare"
QUERY_TYPE_UNKNOWN = "unknown"
ANSWER_MODE_FAST_PATH = "fast_path"
ANSWER_MODE_RETRIEVAL_NORMAL = "retrieval_normal"

9
backend/app/services/ai_pmo/ai_pmo_kernel/ai_pmo_reply_renderer.py
View File

@ -477,10 +477,15 @@ class ReplyRenderer:
}
if intent.name == INTENT_QUERY:
answer = str(execution_result.get("answer") or "当前暂无可用回答。").strip()
template_key = str(execution_result.get("reply_template_key") or "").strip() or "action_succeeded"
return {
"receipt_phase": self.receipt_phase_action,
"template_key": "action_succeeded",
"payload": {"intent": intent.name, "answer": answer},
"template_key": template_key,
"payload": {
"intent": intent.name,
"answer": answer,
"reply_template_key": template_key,
},
"text": answer,
}
if intent.name == INTENT_PROJECT_CREATE:

6
backend/app/services/ai_pmo/ai_pmo_kernel/mixins/intent/ai_pmo_kernel_mixins_intent_resolve.py
View File

@ -179,12 +179,6 @@ class KernelIntentResolveMixin:
"requires_confirmation": False,
"source_system": str(source_system or "").strip().lower() or None,
"source_message_context": "reply_anchor" if reply_anchor_present else "cold_start",
# Planner outage should not hard-fail the thread. Degrade to the
# controlled requirement consultation chain so downstream logic can
# keep the user inside the requirement workflow instead of emitting
# a generic error template.
"message_intent": "consult_requirement",
"consultant_mode_preference": "clarify",
"semantic_planner_degraded": True,
"semantic_planner_error": str(planner_error)[:240],
}

264
backend/app/services/ai_pmo/ai_pmo_kernel_ingress_flow/mixin_entry.py
View File

@ -32,97 +32,20 @@ class KernelIngressFlowEntryMixin:
"requirement_gathering",
"requirement_convergence",
})
_EXTERNAL_FRESHNESS_FOLLOWUP_TOKENS = frozenset({
"继续核实",
"继续查",
"继续验证",
"继续确认",
"接入外部来源",
"外部来源继续核实",
"联网查",
"联网核实",
"外网查",
"实时来源",
"查最新",
"再核实",
})
_INTERNAL_EXTERNAL_SOURCE_TARGET_MARKERS = frozenset({
"我们系统",
"系统里",
"系统内",
"pmflow",
"/ai-config",
"ai-config",
"aiconfig",
"配置页",
"模型供应商",
"给项目",
"项目id",
"项目#",
"project",
"模块",
"功能",
"接口",
"通道",
"租户",
"环境",
"服务",
})
_INTERNAL_EXTERNAL_SOURCE_ACTION_MARKERS = frozenset({
"出需求",
"需求草稿",
"起草",
"立项",
"工单",
"排期",
"任务",
"入库",
"接一个",
"新增",
"加一个",
"配置",
"接入通道",
"集成",
"实现",
"改造",
})
_EXTERNAL_FRESHNESS_GUARD_MARKERS = frozenset({
"externalfreshnessguard",
"未接入实时外部数据",
"实时外部来源",
"无法确认最新结论",
})
_EXTERNAL_FRESHNESS_GUARD_SOURCE_LABEL = "externalfreshnessguard"
@staticmethod
def _normalize_compact_text(value: Any) -> str:
return re.sub(r"\s+", "", str(value or "")).strip().lower()
@classmethod
def _looks_like_external_freshness_followup(cls, message_text: str) -> bool:
normalized = cls._normalize_compact_text(message_text)
if not normalized:
return False
return any(token in normalized for token in cls._EXTERNAL_FRESHNESS_FOLLOWUP_TOKENS)
@classmethod
def _looks_like_explicit_internal_external_source_request(cls, message_text: str) -> bool:
normalized = cls._normalize_compact_text(message_text)
if not normalized:
return False
return (
any(token in normalized for token in cls._INTERNAL_EXTERNAL_SOURCE_TARGET_MARKERS)
and any(token in normalized for token in cls._INTERNAL_EXTERNAL_SOURCE_ACTION_MARKERS)
)
@classmethod
def _payload_mentions_external_freshness_guard(cls, value: Any) -> bool:
def _payload_has_external_freshness_guard_source(cls, value: Any) -> bool:
if isinstance(value, str):
normalized = cls._normalize_compact_text(value)
return any(marker in normalized for marker in cls._EXTERNAL_FRESHNESS_GUARD_MARKERS)
return cls._normalize_compact_text(value) == cls._EXTERNAL_FRESHNESS_GUARD_SOURCE_LABEL
if isinstance(value, dict):
return any(cls._payload_mentions_external_freshness_guard(item) for item in value.values())
return any(cls._payload_has_external_freshness_guard_source(item) for item in value.values())
if isinstance(value, list):
return any(cls._payload_mentions_external_freshness_guard(item) for item in value)
return any(cls._payload_has_external_freshness_guard_source(item) for item in value)
return False
@classmethod
@ -135,12 +58,13 @@ class KernelIngressFlowEntryMixin:
return (
(
grounding_status in {"insufficient_evidence", "partially_grounded"}
and cls._payload_mentions_external_freshness_guard(getattr(qa_record, "sources_used", None))
and cls._payload_has_external_freshness_guard_source(getattr(qa_record, "sources_used", None))
)
or (
"prompt:industry_trend" in policy_version or "prompt:external_trend" in policy_version
)
and cls._payload_mentions_external_freshness_guard(answer_text)
and cls._payload_has_external_freshness_guard_source(getattr(qa_record, "sources_used", None))
and "无法确认最新结论" in answer_text
)
async def _load_recent_external_freshness_guard_record(
@ -183,34 +107,47 @@ class KernelIngressFlowEntryMixin:
).scalars().first()
if self._is_external_freshness_guard_record(anchored_row):
return anchored_row
if normalized_source != "feishu":
return None
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id=channel_ref,
source_sender_open_id=source_sender_open_id,
sender=None,
)
if not conversation_scope_key:
return None
rows = (
await self.db.execute(
select(QARecord)
.where(
*criteria,
QARecord.conversation_scope_key == conversation_scope_key,
)
.order_by(QARecord.updated_at.desc(), QARecord.id.desc())
.limit(3)
)
).scalars().all()
for row in rows:
if str(getattr(row, "source_message_id", "") or "").strip() == str(source_message_id or "").strip():
continue
if self._is_external_freshness_guard_record(row):
return row
return None
@classmethod
def _is_query_answer_only_bundle(cls, bundle: Any) -> bool:
workflows = list(getattr(bundle, "workflows", []) or [])
primary_workflow = str(getattr(bundle, "primary_workflow", "") or "").strip().lower()
if primary_workflow and primary_workflow != "query_flow":
return False
if not workflows:
return primary_workflow in {"", "query_flow"}
for workflow in workflows:
workflow_type = str(getattr(workflow, "workflow_type", "") or "").strip().lower()
if workflow_type and workflow_type != "query_flow":
return False
operation_codes = [str(code or "").strip().lower() for code in list(getattr(workflow, "operation_codes", []) or []) if str(code or "").strip()]
if operation_codes and any(code != "query.answer" for code in operation_codes):
return False
return True
async def _should_keep_external_freshness_followup_in_query_answer(
self,
*,
message_text: str,
source_system: str,
prior_question: Optional[str],
thread_context: Optional[dict[str, Any]],
) -> bool:
try:
bundle = await WorkflowSemanticPlanner.aplan_message(
message_text=str(message_text or "").strip(),
source_system=str(source_system or "").strip().lower() or "adaptive_rag",
metadata={
"external_freshness_followup": True,
"prior_question": str(prior_question or "").strip()[:240] or None,
"thread_context": dict(thread_context or {}),
},
)
except Exception:
return True
return self._is_query_answer_only_bundle(bundle)
@classmethod
def _build_external_freshness_followup_reply(
cls,
@ -237,7 +174,7 @@ class KernelIngressFlowEntryMixin:
if not base_question:
return followup_text
normalized_followup = cls._normalize_compact_text(followup_text)
if not normalized_followup or cls._looks_like_external_freshness_followup(followup_text):
if not normalized_followup or normalized_followup == cls._normalize_compact_text(base_question):
return base_question
return f"{base_question}。补充要求：{followup_text}"[:1200]
@ -251,10 +188,6 @@ class KernelIngressFlowEntryMixin:
message_text: str,
thread_context: Optional[dict[str, Any]],
) -> Optional[dict[str, Any]]:
if not self._looks_like_external_freshness_followup(message_text):
return None
if self._looks_like_explicit_internal_external_source_request(message_text):
return None
try:
guard_record = await self._load_recent_external_freshness_guard_record(
source_system=source_system,
@ -276,6 +209,13 @@ class KernelIngressFlowEntryMixin:
return None
prior_question = str(getattr(guard_record, "question", "") or "").strip() or None
if not await self._should_keep_external_freshness_followup_in_query_answer(
message_text=message_text,
source_system=source_system,
prior_question=prior_question,
thread_context=thread_context,
):
return None
followup_question = self._build_external_freshness_followup_question(
prior_question=prior_question,
message_text=message_text,
@ -975,37 +915,6 @@ class KernelIngressFlowEntryMixin:
ingest_event=ingest_event,
ingress_options=ingress_options,
)
terminated_requirement_scope = await mounted_runtime.terminate_requirement_plan_scope(
tenant_id=int(self.tenant_id),
source_system=source_system,
channel_ref=channel_ref,
sender=sender,
source_sender_open_id=source_sender_open_id,
source_message_id=synthetic_message_id,
)
if int(terminated_requirement_scope.get("terminated_count") or 0) > 0:
try:
await self._update_session(
tenant_id=int(self.tenant_id),
sender=sender,
session=session,
source_chat_id=channel_ref,
source_sender_open_id=source_sender_open_id,
source_message_id=synthetic_message_id,
thread_context=thread_context,
current_project_id=None,
clear_current_project_id=True,
last_intent=ROUTE_QUERY_ANSWER,
last_message_text=clean_text,
)
except Exception as session_exc:
logger.warning(
"mounted_ai_runtime_terminate_requirement_scope_session_update_failed",
tenant_id=self.tenant_id,
source_system=source_system,
source_message_id=synthetic_message_id,
error=str(session_exc),
)
active_context_followup_result = await self._maybe_handle_active_context_followup(
source_system=source_system,
source_message_id=synthetic_message_id,
@ -1847,6 +1756,9 @@ class KernelIngressFlowEntryMixin:
return topic, draft, self._derive_active_context_kind(topic=topic, draft=draft)
finder = getattr(self, "_find_active_requirement_plan_workspace", None)
if not callable(finder):
executor = getattr(self, "executor", None)
finder = getattr(executor, "_find_active_requirement_plan_workspace", None)
if not callable(finder):
return None
topic, draft = await finder(
@ -2022,9 +1934,49 @@ class KernelIngressFlowEntryMixin:
except Exception:
return False
@staticmethod
def _looks_like_requirement_plan_exit_request(message_text: str) -> bool:
try:
from ..ai_pmo_typed_executor_execute.mixin_execute_query_project import ExecuteQueryProjectMixin
return ExecuteQueryProjectMixin._is_requirement_plan_exit_request(message_text)
except Exception:
return False
async def _build_active_requirement_mode_exit_result(
self,
*,
topic: Any,
source_message_id: str,
) -> dict[str, Any]:
target_topic_id = _safe_int(getattr(topic, "id", None))
topic.status = "suspended"
topic.plan_status = "suspended"
topic.pending_actions = ["resume_requirement_plan"]
topic.metadata_json = {
**(topic.metadata_json if isinstance(getattr(topic, "metadata_json", None), dict) else {}),
"plan_mode": True,
"suspended_by_message_id": str(source_message_id or "").strip() or None,
}
flusher = getattr(self.db, "flush", None)
if callable(flusher):
await flusher()
return self._build_active_context_direct_reply_result(
answer="好，这条需求先暂停。你后面引用原消息或说“继续需求”我再接上；现在可以聊别的。",
reason_code="requirement_mode_suspended",
payload={
"workflow_route": ROUTE_REQUIREMENT_PLAN_COLLECT,
"work_item_id": target_topic_id,
"topic_workspace_id": target_topic_id,
"group_reply_allowed_via_bound_context": True,
"query_type": "plan",
},
)
async def _build_active_requirement_followup_result(
self,
*,
topic_workspace_id: Optional[int],
message_text: str,
source_system: str,
source_message_id: str,
@ -2054,6 +2006,16 @@ class KernelIngressFlowEntryMixin:
if not answer:
return None
direct_payload = dict(direct_result) if isinstance(direct_result, dict) else {}
target_topic_id = _safe_int(topic_workspace_id)
if (
target_topic_id is not None
and _safe_int(direct_payload.get("work_item_id")) is None
and str(direct_payload.get("reply_template_key") or "").strip().lower() == "requirement_plan_followup"
):
# Requirement-plan collect cards need a stable positive target id for button round-trips.
# At collect stage there is often no dedicated work item yet, so we reuse the mounted topic id.
direct_payload["work_item_id"] = int(target_topic_id)
direct_payload.setdefault("topic_workspace_id", int(target_topic_id))
direct_payload["group_reply_allowed_via_bound_context"] = True
return self._build_active_context_direct_reply_result(
answer=answer,
@ -2098,11 +2060,20 @@ class KernelIngressFlowEntryMixin:
if bound_context is None:
return None
topic, draft, active_context_kind = bound_context
if (
str(active_context_kind or "").strip().lower() == "requirement_plan"
and self._looks_like_requirement_plan_exit_request(message_text)
):
return await self._build_active_requirement_mode_exit_result(
topic=topic,
source_message_id=source_message_id,
)
if (
str(active_context_kind or "").strip().lower() == "requirement_plan"
and self._looks_like_requirement_plan_choice_protocol_followup(message_text)
):
direct_protocol_result = await self._build_active_requirement_followup_result(
topic_workspace_id=_safe_int(getattr(topic, "id", None)),
message_text=message_text,
source_system=source_system,
source_message_id=source_message_id,
@ -2168,6 +2139,7 @@ class KernelIngressFlowEntryMixin:
if "context.followup.current_requirement" not in allowed_actions:
return None
direct_result = await self._build_active_requirement_followup_result(
topic_workspace_id=_safe_int(getattr(topic, "id", None)),
message_text=message_text,
source_system=source_system,
source_message_id=source_message_id,

198
backend/app/services/ai_pmo/ai_pmo_typed_executor_execute/mixin_execute_query_project.py
View File

@ -103,7 +103,10 @@ class ExecuteQueryProjectMixin:
message_text: str,
project_id: Optional[int],
message_intent: Optional[str] = None,
source_policy: Optional[str] = None,
) -> Optional[dict[str, Any]]:
if str(source_policy or "").strip().lower() == "external":
return None
normalized_intent = str(message_intent or "").strip().lower()
keyword_extractor = getattr(rag_service, "_extract_project_lookup_keyword", None)
extracted_keyword = ""
@ -146,8 +149,11 @@ class ExecuteQueryProjectMixin:
*,
message_text: str,
message_intent: Optional[str],
source_policy: Optional[str] = None,
) -> bool:
_ = message_text
if str(source_policy or "").strip().lower() == "external":
return False
return str(message_intent or "").strip().lower() == "casual_chat"
@classmethod
@ -3999,8 +4005,11 @@ class ExecuteQueryProjectMixin:
]
if not plan_candidates:
return None, None
preferred_candidates = [
row for row in plan_candidates if int(row.id) in draft_by_topic_id
] or plan_candidates
topic = self._select_requirement_plan_topic(
candidates=plan_candidates,
candidates=preferred_candidates,
topic_key=topic_key,
root_message_id=root_message_id,
reply_to_message_id=reply_to_message_id,
@ -4186,6 +4195,7 @@ class ExecuteQueryProjectMixin:
self,
*,
message_text: str,
message_intent: Optional[str],
topic_switch: Optional[str],
has_requirement_anchor: bool,
topic: Optional[MountedTopicWorkspace],
@ -4197,6 +4207,16 @@ class ExecuteQueryProjectMixin:
return False
if self._extract_requirement_plan_choice(message_text):
return False
normalized_message_intent = str(message_intent or "").strip().lower()
if normalized_message_intent not in {
"requirement_plan",
"requirement_gathering",
"requirement_convergence",
"consult_requirement",
"request_draft_generation",
"request_draft_generation_from_context",
}:
return False
normalized_topic_switch = str(topic_switch or "").strip().lower()
nontrivial_seed = self._sanitize_requirement_plan_text(message_text).strip()
@ -5074,6 +5094,10 @@ class ExecuteQueryProjectMixin:
metadata = intent.metadata if isinstance(intent.metadata, dict) else {}
semantic_bundle = metadata.get("semantic_bundle") if isinstance(metadata.get("semantic_bundle"), dict) else {}
semantic_diagnostics = semantic_bundle.get("diagnostics") if isinstance(semantic_bundle.get("diagnostics"), dict) else {}
semantic_planner_degraded = bool(
metadata.get("semantic_planner_degraded") or semantic_diagnostics.get("semantic_planner_degraded")
)
degraded_context_source = str(metadata.get("context_source") or "").strip().lower() == "semantic_planner_degraded"
rag_request_context = (
metadata.get("rag_request_context")
if isinstance(metadata.get("rag_request_context"), dict)
@ -5169,6 +5193,12 @@ class ExecuteQueryProjectMixin:
or semantic_diagnostics.get("force_thinking_mode")
or ""
).strip() or None
if semantic_planner_degraded and degraded_context_source:
# Planner failure metadata is only an audit marker. Do not let stale
# degraded scene labels drive requirement routing for a fresh query.
message_intent = None
consultant_mode_preference = None
advise_subtype = None
normalized_scene_metadata = WorkflowSemanticPlanner.normalize_scene_metadata(
{
"message_intent": message_intent,
@ -5250,6 +5280,8 @@ class ExecuteQueryProjectMixin:
or needs_legacy_business_review_refresh
)
)
if semantic_planner_degraded:
needs_semantic_backfill = True
if not normalized_message_intent:
needs_semantic_backfill = True
elif normalized_message_intent in {"business_review", "requirement_plan"} and not str(
@ -5269,52 +5301,74 @@ class ExecuteQueryProjectMixin:
has_active_requirement_plan_workspace = (
active_requirement_plan_topic is not None and active_requirement_plan_draft is not None
)
semantic_bundle = await self._backfill_query_semantic_bundle(
message_text=message_text,
source_system=source_system,
metadata=metadata,
requested_project_id=requested_project_id if requested_project_id is not None else intent.project_id,
source_chat_id=source_chat_id,
source_sender_open_id=source_sender_open_id,
thread_context=thread_context,
conversation_key=conversation_key,
active_requirement_plan_topic=active_requirement_plan_topic if has_requirement_anchor else None,
active_requirement_plan_draft=active_requirement_plan_draft if has_requirement_anchor else None,
)
semantic_diagnostics = (
semantic_bundle.get("diagnostics") if isinstance(semantic_bundle.get("diagnostics"), dict) else {}
)
refreshed_message_intent = str(semantic_diagnostics.get("message_intent") or "").strip() or None
refreshed_consultant_mode = (
str(semantic_diagnostics.get("consultant_mode_preference") or "").strip().lower() or None
)
refreshed_advise_subtype = (
str(semantic_diagnostics.get("advise_subtype") or "").strip().lower() or None
)
refreshed_topic_switch = str(semantic_diagnostics.get("topic_switch") or "").strip().lower() or None
refreshed_thinking_mode = str(semantic_diagnostics.get("force_thinking_mode") or "").strip() or None
if refreshed_message_intent:
message_intent = refreshed_message_intent
if refreshed_consultant_mode:
consultant_mode_preference = refreshed_consultant_mode
if refreshed_advise_subtype:
advise_subtype = refreshed_advise_subtype
if refreshed_topic_switch:
topic_switch = refreshed_topic_switch
if refreshed_thinking_mode:
thinking_mode = refreshed_thinking_mode
metadata = {**metadata, "semantic_bundle": semantic_bundle}
normalized_scene_metadata = WorkflowSemanticPlanner.normalize_scene_metadata(
{
"message_intent": message_intent,
"consultant_mode_preference": consultant_mode_preference,
}
)
message_intent = str(normalized_scene_metadata.get("message_intent") or message_intent or "").strip() or None
consultant_mode_preference = (
str(normalized_scene_metadata.get("consultant_mode_preference") or consultant_mode_preference or "").strip().lower()
or None
)
try:
semantic_bundle = await self._backfill_query_semantic_bundle(
message_text=message_text,
source_system=source_system,
metadata=metadata,
requested_project_id=requested_project_id if requested_project_id is not None else intent.project_id,
source_chat_id=source_chat_id,
source_sender_open_id=source_sender_open_id,
thread_context=thread_context,
conversation_key=conversation_key,
active_requirement_plan_topic=active_requirement_plan_topic if has_requirement_anchor else None,
active_requirement_plan_draft=active_requirement_plan_draft if has_requirement_anchor else None,
)
except Exception as exc:
logger.warning(
"query_semantic_backfill_failed",
tenant_id=self.tenant_id,
source_system=source_system,
source_message_id=source_message_id,
semantic_planner_degraded=semantic_planner_degraded,
error=str(exc),
)
if isinstance(intent.metadata, dict):
intent.metadata["semantic_backfill_failed"] = True
intent.metadata["semantic_backfill_error"] = str(exc)[:240]
if semantic_planner_degraded and degraded_context_source and not has_active_requirement_plan_workspace:
message_intent = None
consultant_mode_preference = None
advise_subtype = None
else:
semantic_diagnostics = (
semantic_bundle.get("diagnostics") if isinstance(semantic_bundle.get("diagnostics"), dict) else {}
)
refreshed_message_intent = str(semantic_diagnostics.get("message_intent") or "").strip() or None
refreshed_consultant_mode = (
str(semantic_diagnostics.get("consultant_mode_preference") or "").strip().lower() or None
)
refreshed_advise_subtype = (
str(semantic_diagnostics.get("advise_subtype") or "").strip().lower() or None
)
refreshed_topic_switch = str(semantic_diagnostics.get("topic_switch") or "").strip().lower() or None
refreshed_thinking_mode = str(semantic_diagnostics.get("force_thinking_mode") or "").strip() or None
if refreshed_message_intent:
message_intent = refreshed_message_intent
if refreshed_consultant_mode:
consultant_mode_preference = refreshed_consultant_mode
if refreshed_advise_subtype:
advise_subtype = refreshed_advise_subtype
if refreshed_topic_switch:
topic_switch = refreshed_topic_switch
if refreshed_thinking_mode:
thinking_mode = refreshed_thinking_mode
metadata = {**metadata, "semantic_bundle": semantic_bundle}
normalized_scene_metadata = WorkflowSemanticPlanner.normalize_scene_metadata(
{
"message_intent": message_intent,
"consultant_mode_preference": consultant_mode_preference,
}
)
message_intent = str(normalized_scene_metadata.get("message_intent") or message_intent or "").strip() or None
consultant_mode_preference = (
str(
normalized_scene_metadata.get("consultant_mode_preference")
or consultant_mode_preference
or ""
).strip().lower()
or None
)
normalized_message_intent = str(message_intent or "").strip().lower()
if not has_active_requirement_plan_workspace and normalized_message_intent == "requirement_plan":
active_requirement_plan_topic, active_requirement_plan_draft = await self._find_active_requirement_plan_workspace(
@ -5346,6 +5400,7 @@ class ExecuteQueryProjectMixin:
)
if self._should_force_new_same_scope_requirement_plan(
message_text=message_text,
message_intent=message_intent,
topic_switch=topic_switch,
has_requirement_anchor=has_requirement_anchor,
topic=active_requirement_plan_topic,
@ -5489,7 +5544,11 @@ class ExecuteQueryProjectMixin:
)
direct_result["project_id"] = intent.project_id
return direct_result
if requirement_mode_locked and normalized_message_intent == "casual_chat":
if requirement_mode_locked and normalized_message_intent not in {
"requirement_plan",
"requirement_gathering",
"requirement_convergence",
}:
direct_result = self._build_direct_query_result(
answer=self._build_requirement_mode_locked_answer(card=locked_card),
scene_label="Mounted Runtime Plan",
@ -5506,9 +5565,48 @@ class ExecuteQueryProjectMixin:
consultant_mode_preference = (
str(consultant_mode_preference or "exploratory").strip().lower() or "exploratory"
)
route_preview: dict[str, Any] = {}
route_preview_resolver = getattr(rag_service, "preview_query_route_decision", None)
if callable(route_preview_resolver):
try:
route_preview = await route_preview_resolver(
question=message_text,
asked_via=asked_via,
project_id=requested_project_id if requested_project_id is not None else intent.project_id,
source_chat_id=source_chat_id,
source_message_id=source_message_id,
conversation_scope_key=conversation_key,
reply_to_message_id=str((thread_context or {}).get("reply_to_message_id") or "").strip() or None,
root_message_id=str((thread_context or {}).get("root_message_id") or "").strip() or None,
)
except Exception:
route_preview = {}
source_policy_hint = (
route_preview.get("query_semantics")
if isinstance(route_preview.get("query_semantics"), dict)
else {}
)
route_decision = (
route_preview.get("route_decision")
if isinstance(route_preview.get("route_decision"), dict)
else {}
)
source_policy = (
str(source_policy_hint.get("source_policy") or route_decision.get("route_mode") or "").strip().lower() or None
)
effective_source_policy = (
str(
source_policy_hint.get("effective_source_policy")
or route_decision.get("effective_source_policy")
or source_policy
or ""
).strip().lower()
or None
)
if self._should_use_lightweight_chat_answer(
message_text=message_text,
message_intent=message_intent,
source_policy=effective_source_policy,
):
direct_result = self._build_direct_query_result(
answer=self._build_lightweight_chat_answer(message_text=message_text),
@ -5587,6 +5685,7 @@ class ExecuteQueryProjectMixin:
message_text=message_text,
project_id=requested_project_id if requested_project_id is not None else intent.project_id,
message_intent=message_intent,
source_policy=effective_source_policy,
)
if project_lookup_result is not None:
return project_lookup_result
@ -5613,6 +5712,9 @@ class ExecuteQueryProjectMixin:
"message_intent": message_intent,
"mixed_signal_part": mixed_signal_part,
"consultant_mode_preference": consultant_mode_preference,
"source_policy": source_policy,
"precomputed_query_semantics": source_policy_hint if isinstance(source_policy_hint, dict) and source_policy_hint else None,
"route_decision": route_decision if isinstance(route_decision, dict) and route_decision else None,
}
if session_id is not None:
rag_kwargs["session_id"] = int(session_id)

6
backend/app/services/ai_pmo/draft_confirm_handler.py
View File

@ -1504,6 +1504,12 @@ class DraftConfirmHandler:
mounted=(draft_row, topic_row),
)
continue
if anchor_context and not self._matches_mounted_draft_anchor(
draft=draft_row,
topic=topic_row,
anchor_context=anchor_context,
):
continue
score = 0
if normalized_channel and str(getattr(topic_row, "channel_ref", "") or "").strip() == normalized_channel:
score += 30

2
backend/app/services/ai_pmo/workflow_semantic_planner/service.py
View File

@ -40,6 +40,8 @@ class WorkflowSemanticPlanner:
}
_SCENE_CONSULTANT_MODE_ALIASES = {
"structured_analysis": "structured_analysis_first",
"requirement_analysis": "structured_analysis_first",
"requirements_analysis": "structured_analysis_first",
"collaborative": "exploratory",
}

21
backend/app/services/external_query_service/__init__.py Normal file
View File

@ -0,0 +1,21 @@
from __future__ import annotations
from .gateway import ExternalQueryGateway
from .planner import ExternalQueryPlanner
from .registry import ExternalProviderRegistry
from .schemas import (
ExternalAdapterResult,
ExternalEvidenceItem,
ExternalQueryExecution,
ExternalQueryPlan,
)
__all__ = [
"ExternalAdapterResult",
"ExternalEvidenceItem",
"ExternalProviderRegistry",
"ExternalQueryExecution",
"ExternalQueryGateway",
"ExternalQueryPlan",
"ExternalQueryPlanner",
]

6
backend/app/services/external_query_service/adapters/__init__.py Normal file
View File

@ -0,0 +1,6 @@
from __future__ import annotations
from .live_search import LiveSearchFallbackAdapter
from .mcp import MCPExternalAdapter
__all__ = ["LiveSearchFallbackAdapter", "MCPExternalAdapter"]

151
backend/app/services/external_query_service/adapters/live_search.py Normal file
View File

@ -0,0 +1,151 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any, Awaitable, Callable
from ..schemas import ExternalAdapterResult, ExternalEvidenceItem, ExternalQueryPlan
class LiveSearchFallbackAdapter:
adapter_name = "live_search"
supported_query_kinds = {"weather", "news", "market", "generic_web", "unknown"}
def __init__(
self,
*,
build_queries_callable: Callable[[str], list[str]] | None = None,
search_results_callable: Callable[..., Awaitable[list[dict[str, str]]]] | None = None,
fetch_text_callable: Callable[[str], Awaitable[str]] | None = None,
chunk_text_callable: Callable[..., list[str]] | None = None,
score_search_result_callable: Callable[..., float] | None = None,
score_chunk_callable: Callable[..., float] | None = None,
max_pages: int = 3,
) -> None:
self.build_queries_callable = build_queries_callable
self.search_results_callable = search_results_callable
self.fetch_text_callable = fetch_text_callable
self.chunk_text_callable = chunk_text_callable
self.score_search_result_callable = score_search_result_callable
self.score_chunk_callable = score_chunk_callable
self.max_pages = max(1, int(max_pages or 3))
async def execute(self, plan: ExternalQueryPlan) -> ExternalAdapterResult:
if not all(
callable(item)
for item in (
self.search_results_callable,
self.fetch_text_callable,
self.chunk_text_callable,
self.score_search_result_callable,
self.score_chunk_callable,
)
):
return ExternalAdapterResult(
adapter_name=self.adapter_name,
status="unavailable",
evidence=[],
meta={"reason": "live_search_callbacks_missing"},
)
queries = [str(item or "").strip() for item in list(plan.search_queries or []) if str(item or "").strip()]
if not queries and callable(self.build_queries_callable):
try:
queries = [
str(item or "").strip()
for item in list(self.build_queries_callable(plan.topic or "") or [])
if str(item or "").strip()
]
except Exception:
queries = []
if not queries:
return ExternalAdapterResult(adapter_name=self.adapter_name, status="empty", evidence=[], meta={"query": ""})
search_results: list[dict[str, str]] = []
selected_query = ""
for query in queries:
try:
search_results = await self.search_results_callable(
query=query,
max_results=max(3, int(plan.result_limit) * 2),
)
except Exception:
search_results = []
if search_results:
selected_query = query
break
ranked = sorted(
search_results,
key=lambda row: float(
self.score_search_result_callable(
question=plan.topic or selected_query or queries[0],
title=str(row.get("title") or ""),
url=str(row.get("url") or ""),
)
or 0.0
),
reverse=True,
)
evidence: list[ExternalEvidenceItem] = []
fetch_attempts = 0
candidate_limit = min(len(ranked), self.max_pages + 2)
for row in ranked[:candidate_limit]:
url = str(row.get("url") or "").strip()
title = str(row.get("title") or "").strip()
if not url:
continue
fetch_attempts += 1
try:
text = await self.fetch_text_callable(url)
except Exception:
continue
chunks = list(self.chunk_text_callable(text, chunk_size=900, overlap=120) or [])
if not chunks:
continue
best_chunk = ""
best_score = 0.0
for chunk in chunks[:6]:
score = float(
self.score_chunk_callable(
question=plan.topic or selected_query or queries[0],
title=title,
content=chunk,
url=url,
)
or 0.0
)
if score > best_score:
best_score = score
best_chunk = str(chunk or "").strip()
if not best_chunk:
continue
evidence.append(
ExternalEvidenceItem(
source_type="url",
source_label=title or url,
title=title[:255],
target_url=url,
snippet=best_chunk[:1800],
retrieved_at=datetime.now(timezone.utc).isoformat(),
freshness_score=round(best_score, 4),
authority_score=round(best_score, 4),
grounded_fields=["snippet"],
raw_metadata={
"query": selected_query or queries[0],
"provider": self.adapter_name,
"title": title[:255],
},
)
)
return ExternalAdapterResult(
adapter_name=self.adapter_name,
status="success" if evidence else "empty",
evidence=evidence[: max(1, int(plan.result_limit))],
meta={
"query": selected_query or (queries[0] if queries else ""),
"search_result_count": len(search_results),
"fetched_pages": fetch_attempts,
},
)

129
backend/app/services/external_query_service/adapters/mcp.py Normal file
View File

@ -0,0 +1,129 @@
from __future__ import annotations
from typing import Any, Awaitable, Callable
from ..schemas import ExternalAdapterResult, ExternalEvidenceItem, ExternalQueryPlan
class MCPExternalAdapter:
adapter_name = "mcp"
supported_query_kinds = {"model", "company", "generic_web", "unknown"}
def __init__(
self,
*,
search_callable: Callable[[ExternalQueryPlan], Awaitable[dict[str, Any] | ExternalAdapterResult]] | None = None,
fetch_text_callable: Callable[[str], Awaitable[str]] | None = None,
chunk_text_callable: Callable[..., list[str]] | None = None,
score_chunk_callable: Callable[..., float] | None = None,
max_pages: int = 3,
) -> None:
self.search_callable = search_callable
self.fetch_text_callable = fetch_text_callable
self.chunk_text_callable = chunk_text_callable
self.score_chunk_callable = score_chunk_callable
self.max_pages = max(1, int(max_pages or 3))
async def execute(self, plan: ExternalQueryPlan) -> ExternalAdapterResult:
if not callable(self.search_callable):
return ExternalAdapterResult(
adapter_name=self.adapter_name,
status="unavailable",
evidence=[],
meta={"reason": "mcp_adapter_unavailable"},
)
result = await self.search_callable(plan)
if isinstance(result, ExternalAdapterResult):
validated = result
else:
payload = dict(result or {}) if isinstance(result, dict) else {}
payload.setdefault("adapter_name", self.adapter_name)
validated = ExternalAdapterResult.model_validate(payload)
evidence, fetched_pages = await self._hydrate_evidence_from_target_pages(
plan=plan,
evidence=list(validated.evidence or []),
)
meta = dict(validated.meta or {})
if fetched_pages:
meta["fetched_pages"] = int(fetched_pages)
return validated.model_copy(update={"evidence": evidence, "meta": meta})
def _page_fetch_ready(self) -> bool:
return all(
callable(item)
for item in (
self.fetch_text_callable,
self.chunk_text_callable,
self.score_chunk_callable,
)
)
async def _hydrate_evidence_from_target_pages(
self,
*,
plan: ExternalQueryPlan,
evidence: list[ExternalEvidenceItem],
) -> tuple[list[ExternalEvidenceItem], int]:
if not evidence or not self._page_fetch_ready():
return evidence, 0
hydrated: list[ExternalEvidenceItem] = []
fetch_attempts = 0
query = str((plan.search_queries[:1] or [plan.topic])[0] or "").strip() or str(plan.topic or "").strip()
for index, item in enumerate(evidence):
target_url = str(item.target_url or "").strip()
if index >= self.max_pages or (
not target_url.startswith("http://") and not target_url.startswith("https://")
):
hydrated.append(item)
continue
fetch_attempts += 1
try:
text = await self.fetch_text_callable(target_url)
except Exception:
hydrated.append(item)
continue
chunks = list(self.chunk_text_callable(text, chunk_size=900, overlap=120) or [])
if not chunks:
hydrated.append(item)
continue
best_chunk = ""
best_score = 0.0
title = str(item.title or "").strip()
for chunk in chunks[:6]:
score = float(
self.score_chunk_callable(
question=query or plan.topic or "",
title=title,
content=chunk,
url=target_url,
)
or 0.0
)
if score > best_score:
best_score = score
best_chunk = str(chunk or "").strip()
if not best_chunk:
hydrated.append(item)
continue
grounded_fields = [str(field or "").strip() for field in list(item.grounded_fields or []) if str(field or "").strip()]
if "page_content_snippet" not in grounded_fields:
grounded_fields.append("page_content_snippet")
raw_metadata = dict(item.raw_metadata or {})
original_snippet = str(item.snippet or "").strip()
if original_snippet and not str(raw_metadata.get("search_snippet") or "").strip():
raw_metadata["search_snippet"] = original_snippet[:1800]
raw_metadata["page_chunk_score"] = round(best_score, 4)
raw_metadata["target_page_fetched"] = True
hydrated.append(
item.model_copy(
update={
"snippet": best_chunk[:1800],
"freshness_score": round(max(float(item.freshness_score or 0.0), best_score), 4),
"grounded_fields": grounded_fields,
"raw_metadata": raw_metadata,
}
)
)
return hydrated, fetch_attempts

271
backend/app/services/external_query_service/gateway.py Normal file
View File

@ -0,0 +1,271 @@
from __future__ import annotations
from typing import Any
from urllib.parse import urlparse
from .registry import ExternalProviderRegistry
from .schemas import ExternalAdapterResult, ExternalEvidenceItem, ExternalQueryExecution, ExternalQueryPlan
class ExternalQueryGateway:
def __init__(
self,
*,
planner: Any,
registry: ExternalProviderRegistry,
) -> None:
self.planner = planner
self.registry = registry
async def execute(
self,
*,
question: str,
query_semantics: dict[str, Any] | None = None,
) -> ExternalQueryExecution:
planned = await self.planner.build_plan(
question=question,
query_semantics=query_semantics,
)
planner_meta: dict[str, Any] = {}
if isinstance(planned, tuple) and len(planned) == 2:
plan, planner_meta = planned
else:
plan = planned
if not isinstance(plan, ExternalQueryPlan):
plan = ExternalQueryPlan.model_validate(plan)
adapter_attempts: list[dict[str, Any]] = []
selected = ExternalAdapterResult(adapter_name="none", status="empty", evidence=[], meta={})
selected_verifier = self._verify_evidence_bundle(plan=plan, evidence=selected.evidence)
fallback_used = False
for index, adapter in enumerate(self._rank_adapters(plan=plan)):
adapter_name = str(getattr(adapter, "adapter_name", "") or f"adapter_{index}").strip() or f"adapter_{index}"
result = await adapter.execute(plan)
if not isinstance(result, ExternalAdapterResult):
result = ExternalAdapterResult.model_validate(result)
verifier = self._verify_evidence_bundle(plan=plan, evidence=result.evidence)
adapter_attempts.append(
{
"adapter": adapter_name,
"status": str(result.status or "").strip() or "empty",
"evidence_count": len(result.evidence),
"verifier_ok": bool(verifier.get("ok")),
"verifier_reason": verifier.get("reason_code"),
}
)
if result.evidence and bool(verifier.get("ok")):
selected = result
selected_verifier = verifier
fallback_used = index > 0
break
if selected.adapter_name == "none":
selected = result
selected_verifier = verifier
meta = dict(selected.meta or {})
meta["fallback_used"] = bool(fallback_used)
meta["adapter_attempts"] = adapter_attempts
return ExternalQueryExecution(
plan=plan,
adapter_name=selected.adapter_name,
evidence=list(selected.evidence or []),
verifier=selected_verifier,
planner_meta=planner_meta,
meta=meta,
)
@staticmethod
def _normalize_match_text(value: Any) -> str:
raw = str(value or "").strip().lower()
if not raw:
return ""
chars: list[str] = []
for char in raw:
if char.isalnum():
chars.append(char)
return "".join(chars)
@classmethod
def _normalize_anchor_term(cls, value: Any) -> str:
return cls._normalize_match_text(value)[:48]
@staticmethod
def _common_prefix_size(left: str, right: str) -> int:
limit = min(len(left), len(right))
idx = 0
while idx < limit and left[idx] == right[idx]:
idx += 1
return idx
@classmethod
def _build_char_windows(cls, value: str, *, size: int = 2) -> set[str]:
normalized = cls._normalize_match_text(value)
if not normalized:
return set()
if len(normalized) <= size:
return {normalized}
return {normalized[idx : idx + size] for idx in range(len(normalized) - size + 1)}
@classmethod
def _tokenize_match_text(cls, value: Any) -> list[str]:
raw = str(value or "").strip().lower()
if not raw:
return []
tokens: list[str] = []
current: list[str] = []
for char in raw:
if char.isalnum():
current.append(char)
continue
if current:
token = "".join(current)
if token:
tokens.append(token)
current = []
if current:
token = "".join(current)
if token:
tokens.append(token)
normalized = cls._normalize_match_text(raw)
if normalized and normalized not in tokens:
tokens.append(normalized)
return tokens
@classmethod
def _score_anchor_against_token(cls, anchor: str, token: str) -> float:
if not anchor or not token:
return 0.0
if anchor == token:
return 1.0
if anchor in token:
return max(0.82, min(0.98, len(anchor) / max(1, len(token))))
if token in anchor and len(token) >= 2:
return max(0.58, len(token) / max(1, len(anchor)))
prefix_size = cls._common_prefix_size(anchor, token)
prefix_ratio = prefix_size / max(1, min(len(anchor), len(token)))
anchor_windows = cls._build_char_windows(anchor)
token_windows = cls._build_char_windows(token)
window_overlap = (
len(anchor_windows & token_windows) / max(1, len(anchor_windows))
if anchor_windows and token_windows
else 0.0
)
score = 0.0
if prefix_size >= 2:
score = max(score, 0.42 + min(0.3, prefix_ratio * 0.4))
if window_overlap > 0:
score = max(score, 0.35 + min(0.35, window_overlap * 0.5))
return round(min(score, 0.98), 4)
@classmethod
def _evidence_match_tokens(cls, item: ExternalEvidenceItem) -> list[str]:
values = [
str(item.source_label or ""),
str(item.title or ""),
str(item.snippet or ""),
str(item.target_url or ""),
str(urlparse(str(item.target_url or "").strip()).path or ""),
]
tokens: list[str] = []
seen: set[str] = set()
for value in values:
for token in cls._tokenize_match_text(value):
if token and token not in seen:
seen.add(token)
tokens.append(token)
return tokens
@classmethod
def _score_anchor_against_evidence(cls, anchor: str, evidence: list[ExternalEvidenceItem]) -> float:
best = 0.0
for item in evidence:
for token in cls._evidence_match_tokens(item):
best = max(best, cls._score_anchor_against_token(anchor, token))
return round(best, 4)
@classmethod
def _adapter_support_level(cls, *, plan: ExternalQueryPlan, adapter: Any) -> int:
supported = getattr(adapter, "supported_query_kinds", None)
if not supported:
return 1
query_kind = str(plan.query_kind or "").strip().lower()
normalized = {str(item or "").strip().lower() for item in supported if str(item or "").strip()}
return 0 if query_kind in normalized else 2
def _rank_adapters(self, *, plan: ExternalQueryPlan) -> list[Any]:
ranked: list[tuple[int, int, Any]] = []
for index, adapter in enumerate(self._ensure_registry_items(plan=plan)):
ranked.append((self._adapter_support_level(plan=plan, adapter=adapter), index, adapter))
ranked.sort(key=lambda item: (item[0], item[1]))
return [item[2] for item in ranked]
def _ensure_registry_items(self, *, plan: ExternalQueryPlan) -> list[Any]:
_ = plan
return list(self.registry.iter_adapters())
@classmethod
def _build_plan_anchor_terms(cls, plan: ExternalQueryPlan) -> list[str]:
terms: list[str] = []
for item in list(plan.subject_entities or []):
normalized = cls._normalize_anchor_term(item)
if len(normalized) >= 2 and normalized not in terms:
terms.append(normalized)
location_hint = cls._normalize_anchor_term(plan.location_hint)
if len(location_hint) >= 2 and location_hint not in terms:
terms.append(location_hint)
return terms[:6]
@classmethod
def _verify_evidence_bundle(cls, *, plan: ExternalQueryPlan, evidence: list[ExternalEvidenceItem]) -> dict[str, Any]:
if not evidence:
return {
"ok": False,
"reason_code": "missing_external_grounding",
"missing_fields": ["external_evidence"],
}
anchor_terms = cls._build_plan_anchor_terms(plan)
anchor_hit = not anchor_terms
for item in evidence:
target_url = str(item.target_url or "").strip().lower()
source_type = str(item.source_type or "").strip().lower()
if source_type not in {"url", "mcp_search"}:
return {
"ok": False,
"reason_code": "external_source_type_invalid",
"missing_fields": ["external_source_type"],
}
if not target_url.startswith("http://") and not target_url.startswith("https://"):
return {
"ok": False,
"reason_code": "external_target_url_missing",
"missing_fields": ["target_url"],
}
anchor_scores = [cls._score_anchor_against_evidence(anchor, evidence) for anchor in anchor_terms]
strong_hits = sum(1 for score in anchor_scores if score >= 0.8)
medium_hits = sum(1 for score in anchor_scores if score >= 0.55)
bundle_score = (
round(sum(anchor_scores) / max(1, len(anchor_scores)), 4)
if anchor_scores
else 1.0
)
if anchor_scores:
anchor_hit = strong_hits >= 1 or medium_hits == len(anchor_scores) or bundle_score >= 0.6
if not anchor_hit:
return {
"ok": False,
"reason_code": "external_evidence_off_topic",
"missing_fields": ["plan_anchor_terms"],
"bundle_score": bundle_score,
"anchor_scores": anchor_scores,
}
return {
"ok": True,
"reason_code": None,
"missing_fields": [],
"bundle_score": bundle_score,
"anchor_scores": anchor_scores,
}

163
backend/app/services/external_query_service/planner.py Normal file
View File

@ -0,0 +1,163 @@
from __future__ import annotations
import json
import re
from typing import Any, Callable
from ...config.ai_prompts import PROMPT_EXTERNAL_QUERY_PLAN
from .schemas import ExternalQueryPlan
class ExternalQueryPlanner:
FUNCTION_NAME = "external_query_plan"
def __init__(
self,
*,
llm_service: Any,
db: Any,
tenant_id: int,
query_builder: Callable[[str], list[str]] | None = None,
) -> None:
self.llm_service = llm_service
self.db = db
self.tenant_id = int(tenant_id or 1)
self.query_builder = query_builder
async def build_plan(
self,
*,
question: str,
query_semantics: dict[str, Any] | None = None,
) -> tuple[ExternalQueryPlan, dict[str, Any]]:
fallback_plan = self._build_fallback_plan(
question=question,
query_semantics=query_semantics,
)
generator = getattr(self.llm_service, "generate_structured_object", None)
if not callable(generator):
return fallback_plan, {"planner": "heuristic", "fallback_taken": True}
prompt = PROMPT_EXTERNAL_QUERY_PLAN.format(
question=str(question or "").strip()[:1200],
query_semantics=json.dumps(query_semantics or {}, ensure_ascii=False)[:4000],
fallback_queries=json.dumps(fallback_plan.search_queries, ensure_ascii=False),
).strip()
try:
structured = await generator(
prompt=prompt,
db=self.db,
tenant_id=self.tenant_id,
function_name=self.FUNCTION_NAME,
use_prompt_profile=False,
temperature=0.0,
response_mode="fast",
)
payload = getattr(structured, "payload", None)
if payload is None and hasattr(structured, "model_dump"):
payload = structured.model_dump()
if not isinstance(payload, dict):
raise ValueError("external_query_plan_payload_not_dict")
merged = fallback_plan.model_dump()
for key, value in payload.items():
if value in (None, "", [], {}):
continue
merged[key] = value
plan = ExternalQueryPlan.model_validate(merged)
if not plan.search_queries:
plan.search_queries = list(fallback_plan.search_queries)
if not plan.preferred_domains:
plan.preferred_domains = list(fallback_plan.preferred_domains)
if not plan.topic:
plan.topic = fallback_plan.topic
return plan, {"planner": "llm_structured", "fallback_taken": False}
except Exception as exc:
return fallback_plan, {
"planner": "heuristic",
"fallback_taken": True,
"reason": str(exc)[:200],
}
def _build_fallback_plan(
self,
*,
question: str,
query_semantics: dict[str, Any] | None = None,
) -> ExternalQueryPlan:
text = re.sub(r"\s+", " ", str(question or "")).strip()
semantics = query_semantics if isinstance(query_semantics, dict) else {}
query_profile = semantics.get("query_profile") if isinstance(semantics.get("query_profile"), dict) else {}
object_mentions = [
str(item or "").strip()
for item in list(semantics.get("object_mentions") or [])
if str(item or "").strip()
]
query_type = str(semantics.get("query_type") or "").strip().lower()
if query_type == "compare":
answer_contract = "compare_with_sources"
elif bool(query_profile.get("needs_freshness")):
answer_contract = "summary_with_sources"
else:
answer_contract = "direct_fact"
topic = text[:120] or "外部查询"
search_queries = self._build_search_queries(text)
preferred_domains = self._infer_preferred_domains(text)
subject_entities = object_mentions[:6]
location_hint = None
time_range_hint = self._infer_time_range_hint(text)
requires_multi_source = answer_contract == "compare_with_sources"
clarify_on_ambiguity = False
result_limit = 3 if requires_multi_source or bool(query_profile.get("needs_freshness")) else 2
return ExternalQueryPlan(
plan_version="v1",
query_kind="generic_web" if text else "unknown",
answer_contract=answer_contract,
freshness_required=bool(query_profile.get("needs_freshness")),
topic=topic,
subject_entities=subject_entities,
location_hint=location_hint,
time_range_hint=time_range_hint,
preferred_domains=preferred_domains,
search_queries=search_queries,
result_limit=result_limit,
requires_multi_source=requires_multi_source,
clarify_on_ambiguity=clarify_on_ambiguity,
)
def _build_search_queries(self, question: str) -> list[str]:
if callable(self.query_builder):
try:
queries = self.query_builder(question)
normalized = [str(item or "").strip() for item in list(queries or []) if str(item or "").strip()]
if normalized:
return normalized[:5]
except Exception:
pass
text = re.sub(r"\s+", " ", str(question or "")).strip()
return [text] if text else []
@staticmethod
def _infer_preferred_domains(question: str) -> list[str]:
normalized = str(question or "").lower()
domains: list[str] = []
if any(token in normalized for token in ("codex", "gpt", "chatgpt", "openai")):
domains.extend(["openai.com", "developers.openai.com"])
if any(token in normalized for token in ("copilot", "github")):
domains.extend(["github.com", "docs.github.com"])
uniq: list[str] = []
for domain in domains:
if domain not in uniq:
uniq.append(domain)
return uniq[:4]
@staticmethod
def _infer_time_range_hint(question: str) -> str | None:
text = str(question or "")
for pattern in (r"未来\d+天", r"未来一周", r"今天", r"截至今天", r"截至现在", r"目前", r"最新"):
match = re.search(pattern, text)
if match:
return str(match.group(0) or "").strip() or None
return None

11
backend/app/services/external_query_service/registry.py Normal file
View File

@ -0,0 +1,11 @@
from __future__ import annotations
from typing import Any
class ExternalProviderRegistry:
def __init__(self, *, adapters: list[Any] | None = None) -> None:
self._adapters = list(adapters or [])
def iter_adapters(self) -> list[Any]:
return list(self._adapters)

62
backend/app/services/external_query_service/schemas.py Normal file
View File

@ -0,0 +1,62 @@
from __future__ import annotations
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field
ExternalQueryKind = Literal["weather", "news", "company", "model", "market", "generic_web", "unknown"]
ExternalAnswerContract = Literal["direct_fact", "summary_with_sources", "compare_with_sources", "clarify"]
class ExternalQueryPlan(BaseModel):
model_config = ConfigDict(extra="ignore")
plan_version: str = "v1"
query_kind: ExternalQueryKind = "unknown"
answer_contract: ExternalAnswerContract = "summary_with_sources"
freshness_required: bool = False
topic: str = ""
subject_entities: list[str] = Field(default_factory=list)
location_hint: str | None = None
time_range_hint: str | None = None
preferred_domains: list[str] = Field(default_factory=list)
search_queries: list[str] = Field(default_factory=list)
result_limit: int = Field(default=3, ge=1, le=5)
requires_multi_source: bool = False
clarify_on_ambiguity: bool = False
class ExternalEvidenceItem(BaseModel):
model_config = ConfigDict(extra="ignore")
source_type: str = "url"
source_label: str = ""
title: str = ""
target_url: str = ""
snippet: str = ""
retrieved_at: str | None = None
freshness_score: float = 0.0
authority_score: float = 0.0
grounded_fields: list[str] = Field(default_factory=list)
raw_metadata: dict[str, Any] = Field(default_factory=dict)
class ExternalAdapterResult(BaseModel):
model_config = ConfigDict(extra="ignore")
adapter_name: str
status: str = "empty"
evidence: list[ExternalEvidenceItem] = Field(default_factory=list)
meta: dict[str, Any] = Field(default_factory=dict)
class ExternalQueryExecution(BaseModel):
model_config = ConfigDict(extra="ignore")
plan: ExternalQueryPlan
adapter_name: str
evidence: list[ExternalEvidenceItem] = Field(default_factory=list)
verifier: dict[str, Any] = Field(default_factory=dict)
planner_meta: dict[str, Any] = Field(default_factory=dict)
meta: dict[str, Any] = Field(default_factory=dict)

2
backend/app/services/feedback_center/mappers.py
View File

@ -200,6 +200,7 @@ def map_qa_feedback_to_record_payload(*, qa_feedback: Any, qa_record: Any) -> di
"sources": _jsonable(getattr(qa_record, "sources", None) or []),
"citations": _jsonable(getattr(qa_record, "citations", None) or []),
"sources_used": _jsonable(getattr(qa_record, "sources_used", None) or []),
"routing_snapshot": _jsonable(getattr(qa_record, "routing_snapshot", None) or {}),
}
after_value = {
"rating": rating,
@ -233,6 +234,7 @@ def map_qa_feedback_to_record_payload(*, qa_feedback: Any, qa_record: Any) -> di
"answer_mode": getattr(qa_record, "answer_mode", None),
"grounding_status": getattr(qa_record, "grounding_status", None),
"policy_version": getattr(qa_record, "policy_version", None),
"routing_snapshot": _jsonable(getattr(qa_record, "routing_snapshot", None) or {}),
"citations": _jsonable(getattr(qa_record, "citations", None) or []),
"sources_used": _jsonable(getattr(qa_record, "sources_used", None) or []),
"evidence_pack": _jsonable(getattr(qa_record, "evidence_pack", None) or {}),

26
backend/app/services/integrations/feishu_integration_service/mixin_card_action.py
View File

@ -379,13 +379,28 @@ class FeishuCardActionMixin:
synthetic_text = "重出一版"
if not synthetic_text:
return {"code": 0, "msg": "card_action_ignored", "data": {"reason": "missing_synthetic_text"}}
kernel = AIPMOKernelService(self.db, tenant_id=self.tenant_id)
result = await kernel.process_ingress(
source_system="ai_pmo_ui",
source_message_id=(
use_feishu_context = bool(chat_id and source_message_id)
ingress_source_system = "feishu" if use_feishu_context else "ai_pmo_ui"
ingress_source_message_id = (
f"feishu-card-plan-{int(work_item_id)}-{action}-{int(time.time() * 1000)}"
if use_feishu_context
else (
source_message_id
or f"feishu-card-plan-{int(work_item_id)}-{action}-{int(time.time() * 1000)}"
),
)
)
ingress_options = None
if use_feishu_context:
ingress_options = {
"thread_context": {
"reply_to_message_id": str(source_message_id),
"root_message_id": str(source_message_id),
}
}
kernel = AIPMOKernelService(self.db, tenant_id=self.tenant_id)
result = await kernel.process_ingress(
source_system=ingress_source_system,
source_message_id=ingress_source_message_id,
message_text=synthetic_text,
sender=f"{operator_sender}(card)",
channel_ref=chat_id,
@ -393,6 +408,7 @@ class FeishuCardActionMixin:
captured_at=None,
chat_type="manual",
project_hint_id=None,
ingress_options=ingress_options,
)
if hasattr(self.db, "commit") and callable(getattr(self.db, "commit")):
await self.db.commit()

8
backend/app/services/llm_service/mixin_cache_circuit.py
View File

@ -181,9 +181,13 @@ class LLMCacheCircuitMixin:
return
now = time.monotonic()
state = dict(cls._provider_circuit_state.get(str(circuit_key)) or {})
consecutive = int(state.get("consecutive_failures") or 0) + 1
fail_fast = cls._is_fail_fast_exception(exc)
if fail_fast:
consecutive = CIRCUIT_BREAKER_FAILURE_THRESHOLD
else:
consecutive = int(state.get("consecutive_failures") or 0) + 1
open_until = float(state.get("open_until") or 0.0)
if consecutive >= CIRCUIT_BREAKER_FAILURE_THRESHOLD:
if fail_fast or consecutive >= CIRCUIT_BREAKER_FAILURE_THRESHOLD:
open_until = now + CIRCUIT_BREAKER_COOLDOWN_SECONDS
cls._provider_circuit_state[str(circuit_key)] = {
"consecutive_failures": consecutive,

66
backend/app/services/llm_service/mixin_chat_openai.py
View File

@ -9,8 +9,30 @@ class LLMEmptyContentError(ValueError):
class LLMChatOpenAIMixin:
@staticmethod
def _append_direct_answer_retry_instruction(messages: list[dict[str, str]]) -> None:
retry_instruction = "请直接输出答案"
def _is_provider_billing_blocked_http_error(*, status_code: int, body_preview: str) -> bool:
if int(status_code or 0) not in {400, 401, 402, 403}:
return False
lowered = str(body_preview or "").strip().lower()
if not lowered:
return False
return (
"arrearage" in lowered
or "overdue-payment" in lowered
or ("access denied" in lowered and "good standing" in lowered)
or "provider_billing_blocked" in lowered
)
@staticmethod
def _append_direct_answer_retry_instruction(
messages: list[dict[str, str]],
*,
structured_output: bool = False,
) -> None:
retry_instruction = (
"请直接输出最终 JSON 对象，不要解释，不要 Markdown，不要把正文放在 reasoning_content。"
if structured_output
else "请直接输出答案"
)
for idx in range(len(messages) - 1, -1, -1):
row = messages[idx]
if not isinstance(row, dict):
@ -45,6 +67,7 @@ class LLMChatOpenAIMixin:
request_started_at_perf: float,
request_trace_id: str,
control_plane_agent_timeout: Optional[int],
routing_meta: dict[str, Any] | None = None,
) -> ChatCompletionResult:
endpoint = f"{base_url.rstrip('/')}/chat/completions"
headers = {
@ -127,6 +150,11 @@ class LLMChatOpenAIMixin:
),
)
status_code = int(resp.status_code)
if self._is_provider_billing_blocked_http_error(
status_code=status_code,
body_preview=body_preview,
):
raise ValueError(f"provider_billing_blocked:{provider_name}:{body_preview}")
can_retry_http_status = (
http_status_retry_budget > 0
and self._is_retryable_http_status(status_code)
@ -185,14 +213,27 @@ class LLMChatOpenAIMixin:
if self._contains_json_container(normalized_reasoning):
candidate = self._extract_json_candidate(normalized_reasoning)
if candidate:
_runtime_log(
"LLM",
(
"结构化reasoning_content回退："
f"function={function_label}, provider={provider_name}"
),
)
text = candidate
if self._call_is_invalid_structured_output(
function_name=function_label,
response_text=candidate,
routing_meta=routing_meta,
):
_runtime_log(
"LLM",
(
"结构化reasoning_content候选无效，继续空content重试："
f"function={function_label}, provider={provider_name}"
),
)
else:
_runtime_log(
"LLM",
(
"结构化reasoning_content回退："
f"function={function_label}, provider={provider_name}"
),
)
text = candidate
if text:
break
@ -209,7 +250,10 @@ class LLMChatOpenAIMixin:
)
if not direct_answer_retry_used:
direct_answer_retry_used = True
self._append_direct_answer_retry_instruction(messages)
self._append_direct_answer_retry_instruction(
messages,
structured_output=structured_output,
)
_runtime_log(
"LLM",
(

1
backend/app/services/llm_service/mixin_chat_transport.py
View File

@ -129,6 +129,7 @@ class LLMChatTransportMixin:
request_started_at_perf=request_started_at_perf,
request_trace_id=request_trace_id,
control_plane_agent_timeout=control_plane_agent_timeout,
routing_meta=normalized_routing_meta,
)
if use_provider_circuit_breaker:
self._record_provider_success(circuit_key)

81
backend/app/services/llm_service/mixin_task_helpers.py
View File

@ -3,6 +3,47 @@ from __future__ import annotations
from .shared import *
class LLMTaskHelpersMixin:
@staticmethod
def _extract_embedding_vector(payload: Any) -> list[float]:
if not isinstance(payload, dict):
return []
if isinstance(payload.get("embedding"), list):
return [float(item) for item in payload.get("embedding") or []]
embeddings = payload.get("embeddings")
if isinstance(embeddings, list) and embeddings and isinstance(embeddings[0], list):
return [float(item) for item in embeddings[0]]
data = payload.get("data")
if isinstance(data, list) and data and isinstance(data[0], dict):
embedding = data[0].get("embedding")
if isinstance(embedding, list):
return [float(item) for item in embedding]
return []
async def _request_openai_embedding(self, *, endpoint: str, headers: dict[str, str], payload: dict[str, Any]) -> list[float]:
async with httpx.AsyncClient(timeout=self.request_timeout, trust_env=False) as client:
resp = await client.post(endpoint, headers=headers, json=payload)
resp.raise_for_status()
return self._extract_embedding_vector(resp.json())
async def _request_ollama_embedding(self, *, host: str, model: str, text: str) -> list[float]:
base_url = str(host or os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")).rstrip("/")
headers = {"Content-Type": "application/json"}
candidates = [
(f"{base_url}/api/embed", {"model": model, "input": text}),
(f"{base_url}/api/embeddings", {"model": model, "prompt": text}),
]
for endpoint, payload in candidates:
try:
async with httpx.AsyncClient(timeout=self.request_timeout, trust_env=False) as client:
resp = await client.post(endpoint, headers=headers, json=payload)
resp.raise_for_status()
vector = self._extract_embedding_vector(resp.json())
if vector:
return vector
except Exception:
continue
return []
async def generate_task_breakdown_async(self, description: str, **kwargs: Any) -> list[dict]:
if not description.strip():
return []
@ -27,14 +68,18 @@ class LLMTaskHelpersMixin:
]
async def get_embedding_async(self, text: str, **kwargs: Any) -> list[float]:
clean_text = str(text or "").strip()
if not clean_text:
return []
db = kwargs.get("db") or kwargs.get("db_session")
tenant_id = int(kwargs.get("tenant_id") or 1)
if db is None:
async with AsyncSessionLocal() as session:
return await self.get_embedding_async(text, db=session, tenant_id=tenant_id)
return await self.get_embedding_async(clean_text, db=session, tenant_id=tenant_id)
engine: dict[str, Any] = {}
try:
engine = await self._resolve_active_engine(db, tenant_id=tenant_id)
provider = str(engine.get("provider") or "")
provider = str(engine.get("source_provider") or engine.get("provider") or "").strip().lower()
if provider == "openai":
endpoint = f"{str(engine.get('base_url') or '').rstrip('/')}/embeddings"
headers = {
@ -43,18 +88,29 @@ class LLMTaskHelpersMixin:
}
payload = {
"model": str(engine.get("model") or "text-embedding-3-small"),
"input": text,
"input": clean_text,
}
async with httpx.AsyncClient(timeout=self.request_timeout, trust_env=False) as client:
resp = await client.post(endpoint, headers=headers, json=payload)
resp.raise_for_status()
data = resp.json()
return list(data.get("data", [{}])[0].get("embedding") or [])
vector = await self._request_openai_embedding(endpoint=endpoint, headers=headers, payload=payload)
if vector:
return vector
if provider == "ollama":
vector = await self._request_ollama_embedding(
host=str(engine.get("base_url") or os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")),
model=str(engine.get("model") or os.getenv("OLLAMA_EMBED_MODEL", "nomic-embed-text:latest")),
text=clean_text,
)
if vector:
return vector
except Exception:
pass
# DeepSeek/GLM 及失败场景：回退伪向量，保持功能可用。
seed = hashlib.md5(text.encode("utf-8")).digest()
return [((seed[i % 16] / 255.0) * 2 - 1) for i in range(64)]
engine = dict(engine) if isinstance(engine, dict) else {}
vector = await self._request_ollama_embedding(
host=str(os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")),
model=str(os.getenv("OLLAMA_EMBED_MODEL", "nomic-embed-text:latest")),
text=clean_text,
)
if vector:
return vector
return []
async def extract_context_insights_async(self, text: str, **kwargs: Any) -> dict:
prompt = (
@ -102,4 +158,3 @@ class LLMTaskHelpersMixin:
"passed": passed,
"reason": "basic_stub_check_passed" if passed else "invalid_input",
}

34
backend/app/services/llm_service/shared_part_01.py
View File

@ -122,6 +122,10 @@ FAIL_FAST_ERROR_MARKERS = (
"返回空内容",
"empty_response",
"provider_circuit_open",
"provider_billing_blocked",
"arrearage",
"overdue-payment",
"account is in good standing",
)
CIRCUIT_BREAKER_FAILURE_THRESHOLD = 2
CIRCUIT_BREAKER_COOLDOWN_SECONDS = 300.0
@ -190,6 +194,36 @@ STRUCTURED_OBJECT_SCHEMAS: dict[str, dict[str, Any]] = {
"uncertainties": ("list",),
},
},
"external_query_plan": {
"required_keys": (
"plan_version",
"query_kind",
"answer_contract",
"freshness_required",
"topic",
"search_queries",
"result_limit",
"requires_multi_source",
"clarify_on_ambiguity",
),
"type_checks": {
"plan_version": "str",
"query_kind": "str",
"answer_contract": "str",
"freshness_required": "bool",
"topic": "str",
"search_queries": "list",
"result_limit": "number",
"requires_multi_source": "bool",
"clarify_on_ambiguity": "bool",
},
"optional_type_checks": {
"subject_entities": ("list",),
"preferred_domains": ("list",),
"location_hint": ("str_or_none",),
"time_range_hint": ("str_or_none",),
},
},
"signal_planner_analyze": {
"required_keys": ("summary", "priority", "execution_plan", "risks", "confidence"),
"type_checks": {

3
backend/app/services/mcp_config/__init__.py Normal file
View File

@ -0,0 +1,3 @@
from .service import McpConfigService
__all__ = ["McpConfigService"]

87
backend/app/services/mcp_config/catalog.py Normal file
View File

@ -0,0 +1,87 @@
from __future__ import annotations
from .schemas import McpFieldSpec, McpTemplateSpec
def build_template_catalog() -> list[McpTemplateSpec]:
return [
McpTemplateSpec(
id="microsoft_learn",
name="Microsoft Learn MCP",
description="官方远程 MCP，无需 Token，可直接接入 Microsoft Learn 文档检索。",
transport="streamable_http",
execution_mode="backend_direct",
icon="book-open",
recommended=True,
docs_url="https://github.com/MicrosoftDocs/mcp",
fields=[],
command_hint="保存后由 PMFlow 后端直接连接远程 MCP，不要求客户机器安装额外 MCP host。",
verify_hint="重新校验时，PMFlow 会从后端执行 MCP 握手与 tools/list 探测。",
),
McpTemplateSpec(
id="github",
name="GitHub MCP",
description="官方 GitHub MCP，面向仓库、Issue、PR 等开发资产。",
transport="stdio",
execution_mode="local_bridge",
icon="github",
docs_url="https://github.com/modelcontextprotocol/servers",
fields=[
McpFieldSpec(
key="token_env_var_name",
label="本地 Token 环境变量名",
field_type="text",
required=True,
placeholder="GITHUB_PAT",
description="PMFlow 只保存环境变量名，不保存明文 Token。",
default_value="GITHUB_PAT",
),
],
command_hint="需要 PMFlow Local Bridge 或兼容 MCP host 在客户本机承载 stdio server。",
verify_hint="确认本机 bridge/host 已接入后，再回到这里做手动确认。",
),
McpTemplateSpec(
id="filesystem",
name="Filesystem MCP",
description="官方 reference server，面向受控目录的本地文件访问。",
transport="stdio",
execution_mode="local_bridge",
icon="folder-open",
docs_url="https://github.com/modelcontextprotocol/servers",
fields=[
McpFieldSpec(
key="allowed_paths",
label="允许目录",
field_type="path_list",
required=True,
placeholder="/absolute/path/one",
description="每行一个绝对路径，只暴露给 MCP 允许访问的目录。",
default_value=[],
),
],
command_hint="需要 PMFlow Local Bridge 或兼容 MCP host 在客户本机启动 filesystem server。",
verify_hint="确认本机 bridge/host 已接入并暴露目录后，再回到这里做手动确认。",
),
McpTemplateSpec(
id="playwright",
name="Playwright MCP",
description="官方 Playwright MCP，适合浏览器自动化和页面抓取。",
transport="stdio",
execution_mode="local_bridge",
icon="monitor-play",
docs_url="https://playwright.dev/java/agents/network-and-security",
fields=[
McpFieldSpec(
key="output_dir",
label="输出目录",
field_type="text",
required=False,
placeholder="~/.pmflow/mcp/playwright-output",
description="用于保存 Playwright MCP 输出工件的目录。",
default_value="~/.pmflow/mcp/playwright-output",
),
],
command_hint="需要 PMFlow Local Bridge 或兼容 MCP host 在客户本机启动 Playwright MCP。",
verify_hint="确认本机 bridge/host 已接入并能产出浏览器工件后，再回到这里做手动确认。",
),
]

149
backend/app/services/mcp_config/runtime.py Normal file
View File

@ -0,0 +1,149 @@
from __future__ import annotations
import json
from typing import Any, Callable
import httpx
from ...utils.timezone import now_cn_naive
from ..external_query_service.schemas import ExternalEvidenceItem
class MicrosoftLearnMCPRuntime:
def __init__(
self,
*,
endpoint_url: str = "https://learn.microsoft.com/api/mcp",
client_factory: Callable[[], httpx.AsyncClient] | None = None,
) -> None:
self.endpoint_url = str(endpoint_url or "https://learn.microsoft.com/api/mcp").strip()
self.client_factory = client_factory or (
lambda: httpx.AsyncClient(timeout=20.0, trust_env=False, follow_redirects=True)
)
async def probe(self) -> dict[str, Any]:
checked_at = now_cn_naive().isoformat()
try:
session_id, _ = await self._initialize()
_, tools_payload = await self._post_jsonrpc(
{
"jsonrpc": "2.0",
"id": 2,
"method": "tools/list",
"params": {},
},
session_id=session_id,
)
tools = list((tools_payload.get("result") or {}).get("tools") or [])
tool_names = [str(item.get("name") or "").strip() for item in tools if str(item.get("name") or "").strip()]
return {
"status": "ready",
"checked_at": checked_at,
"message": "PMFlow backend can directly reach the remote MCP server.",
"tool_names": tool_names,
}
except Exception as exc:
return {
"status": "error",
"checked_at": checked_at,
"message": f"PMFlow backend probe failed: {exc}",
"tool_names": [],
}
async def search(self, *, query: str, result_limit: int = 3) -> list[ExternalEvidenceItem]:
session_id, _ = await self._initialize()
_, payload = await self._post_jsonrpc(
{
"jsonrpc": "2.0",
"id": 3,
"method": "tools/call",
"params": {
"name": "microsoft_docs_search",
"arguments": {
"query": str(query or "").strip(),
},
},
},
session_id=session_id,
)
text_payload = ""
for item in list((payload.get("result") or {}).get("content") or []):
if str(item.get("type") or "").strip() == "text":
text_payload = str(item.get("text") or "").strip()
if text_payload:
break
parsed = json.loads(text_payload) if text_payload else {}
results = list(parsed.get("results") or [])
retrieved_at = now_cn_naive().isoformat()
evidence: list[ExternalEvidenceItem] = []
for raw in results[: max(1, int(result_limit or 3))]:
target_url = str(raw.get("contentUrl") or "").strip()
if not target_url.startswith("http://") and not target_url.startswith("https://"):
continue
title = str(raw.get("title") or "").strip()
snippet = str(raw.get("content") or "").strip()
evidence.append(
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Microsoft Learn MCP",
title=title,
target_url=target_url,
snippet=snippet[:1800],
retrieved_at=retrieved_at,
freshness_score=0.86,
authority_score=0.98,
grounded_fields=["microsoft_docs_search"],
raw_metadata={
"provider": "microsoft_learn_mcp",
"result_id": str(raw.get("id") or "").strip() or None,
},
)
)
return evidence
async def _initialize(self) -> tuple[str | None, dict[str, Any]]:
return await self._post_jsonrpc(
{
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": "2025-03-26",
"capabilities": {},
"clientInfo": {
"name": "pmflow",
"version": "0.1.0",
},
},
}
)
async def _post_jsonrpc(
self,
payload: dict[str, Any],
*,
session_id: str | None = None,
) -> tuple[str | None, dict[str, Any]]:
headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
if session_id:
headers["mcp-session-id"] = session_id
async with self.client_factory() as client:
response = await client.post(self.endpoint_url, headers=headers, json=payload)
response.raise_for_status()
next_session = str(response.headers.get("mcp-session-id") or session_id or "").strip() or None
body = response.text
parsed = self._extract_payload(body)
return next_session, parsed
@staticmethod
def _extract_payload(body: str) -> dict[str, Any]:
data_lines: list[str] = []
for line in str(body or "").splitlines():
if line.startswith("data:"):
data_lines.append(line[len("data:") :].lstrip())
if not data_lines:
raise ValueError("MCP streamable_http response missing data payload")
return json.loads("\n".join(data_lines))

62
backend/app/services/mcp_config/schemas.py Normal file
View File

@ -0,0 +1,62 @@
from __future__ import annotations
from typing import Literal
from pydantic import BaseModel, Field
McpClientApplyState = Literal["pending_local_apply", "user_confirmed", "not_required"]
McpExecutionMode = Literal["backend_direct", "local_bridge"]
McpRuntimeProbeStatus = Literal["unknown", "ready", "error", "local_bridge_required"]
class McpFieldSpec(BaseModel):
key: str
label: str
field_type: Literal["text", "path_list", "readonly"] = "text"
required: bool = False
placeholder: str | None = None
description: str | None = None
default_value: str | list[str] | None = None
class McpCommandPreview(BaseModel):
add_command: str
verify_command: str
expected_output: str
class McpRuntimeProbe(BaseModel):
status: McpRuntimeProbeStatus = "unknown"
checked_at: str | None = None
message: str | None = None
tool_names: list[str] = Field(default_factory=list)
class McpTemplateSpec(BaseModel):
id: str
name: str
description: str
transport: Literal["stdio", "streamable_http"]
execution_mode: McpExecutionMode
icon: str
recommended: bool = False
docs_url: str | None = None
fields: list[McpFieldSpec] = Field(default_factory=list)
command_hint: str | None = None
verify_hint: str | None = None
class McpServerConfigPayload(BaseModel):
id: str
name: str
display_name: str
template_id: str
transport: Literal["stdio", "streamable_http"]
execution_mode: McpExecutionMode = "local_bridge"
params: dict[str, object] = Field(default_factory=dict)
preview: McpCommandPreview
client_apply_state: McpClientApplyState = "pending_local_apply"
runtime_probe: McpRuntimeProbe = Field(default_factory=McpRuntimeProbe)
created_at: str
updated_at: str
confirmed_at: str | None = None

374
backend/app/services/mcp_config/service.py Normal file
View File

@ -0,0 +1,374 @@
from __future__ import annotations
import re
from typing import Any
from uuid import uuid4
from sqlalchemy import and_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...models import Integration
from ...utils.timezone import now_cn_naive
from ..external_query_service.schemas import ExternalEvidenceItem
from .catalog import build_template_catalog
from .runtime import MicrosoftLearnMCPRuntime
from .schemas import McpCommandPreview, McpRuntimeProbe, McpServerConfigPayload, McpTemplateSpec
INTEGRATION_TYPE = "mcp_template_center"
_VALID_ENV_VAR = re.compile(r"^[A-Z_][A-Z0-9_]*$")
_VALID_NAME_CHARS = re.compile(r"[^a-z0-9_-]+")
class McpConfigService:
def __init__(self, db: AsyncSession, *, tenant_id: int) -> None:
self.db = db
self.tenant_id = int(tenant_id)
self._catalog = {item.id: item for item in build_template_catalog()}
async def get_catalog(self) -> dict[str, Any]:
items = [item.model_dump(mode="json") for item in self._catalog.values()]
recommended = next((item.id for item in self._catalog.values() if item.recommended), "microsoft_learn")
return {
"recommended_template_id": recommended,
"items": items,
}
async def list_servers(self) -> list[dict[str, Any]]:
_, bundle = await self._get_or_init_bundle()
items = [
McpServerConfigPayload.model_validate(self._normalize_server_record(item)).model_dump(mode="json")
for item in bundle.get("servers", [])
if isinstance(item, dict)
]
return sorted(items, key=lambda item: str(item.get("updated_at") or ""), reverse=True)
async def create_server(
self,
*,
name: str,
display_name: str | None,
template_id: str,
params: dict[str, Any] | None,
) -> dict[str, Any]:
row, bundle = await self._get_or_init_bundle()
template = self._get_template(template_id)
cleaned_name = self._normalize_server_name(name)
servers = [item for item in bundle.get("servers", []) if isinstance(item, dict)]
if any(str(item.get("name") or "") == cleaned_name for item in servers):
raise ValueError(f"MCP 配置名已存在：{cleaned_name}")
cleaned_params = self._validate_params(template, params or {})
preview = self._build_preview(template=template, name=cleaned_name, params=cleaned_params)
now_text = now_cn_naive().isoformat()
item = McpServerConfigPayload(
id=f"mcp_{uuid4().hex[:12]}",
name=cleaned_name,
display_name=str(display_name or template.name).strip() or template.name,
template_id=template.id,
transport=template.transport,
execution_mode=template.execution_mode,
params=cleaned_params,
preview=preview,
client_apply_state=self._default_client_apply_state(template),
runtime_probe=self._default_runtime_probe(template),
created_at=now_text,
updated_at=now_text,
confirmed_at=None,
).model_dump(mode="json")
servers.append(item)
bundle["servers"] = servers
row.config = bundle
await self.db.flush()
return item
async def update_server(
self,
*,
server_id: str,
name: str | None = None,
display_name: str | None = None,
template_id: str | None = None,
params: dict[str, Any] | None = None,
) -> dict[str, Any]:
row, bundle = await self._get_or_init_bundle()
servers = [self._normalize_server_record(item) for item in bundle.get("servers", []) if isinstance(item, dict)]
target = self._find_server(servers, server_id=server_id)
if target is None:
raise LookupError("MCP 配置不存在")
next_template = self._get_template(str(template_id or target.get("template_id") or ""))
next_name = self._normalize_server_name(str(name or target.get("name") or ""))
if any(
str(item.get("id") or "") != str(server_id)
and str(item.get("name") or "") == next_name
for item in servers
):
raise ValueError(f"MCP 配置名已存在：{next_name}")
params_input = params if params is not None else dict(target.get("params") or {})
cleaned_params = self._validate_params(next_template, params_input)
preview = self._build_preview(template=next_template, name=next_name, params=cleaned_params)
name_changed = next_name != str(target.get("name") or "")
template_changed = next_template.id != str(target.get("template_id") or "")
params_changed = cleaned_params != dict(target.get("params") or {})
target["name"] = next_name
target["display_name"] = str(display_name or target.get("display_name") or next_template.name).strip() or next_template.name
target["template_id"] = next_template.id
target["transport"] = next_template.transport
target["execution_mode"] = next_template.execution_mode
target["params"] = cleaned_params
target["preview"] = preview.model_dump(mode="json")
target["updated_at"] = now_cn_naive().isoformat()
if name_changed or template_changed or params_changed:
target["client_apply_state"] = self._default_client_apply_state(next_template)
target["confirmed_at"] = None
target["runtime_probe"] = self._default_runtime_probe(next_template).model_dump(mode="json")
bundle["servers"] = servers
row.config = bundle
await self.db.flush()
return McpServerConfigPayload.model_validate(target).model_dump(mode="json")
async def validate_server(self, *, server_id: str) -> dict[str, Any]:
row, bundle = await self._get_or_init_bundle()
servers = [self._normalize_server_record(item) for item in bundle.get("servers", []) if isinstance(item, dict)]
target = self._find_server(servers, server_id=server_id)
if target is None:
raise LookupError("MCP 配置不存在")
template = self._get_template(str(target.get("template_id") or ""))
cleaned_params = self._validate_params(template, dict(target.get("params") or {}))
preview = self._build_preview(
template=template,
name=str(target.get("name") or ""),
params=cleaned_params,
)
target["params"] = cleaned_params
target["preview"] = preview.model_dump(mode="json")
if template.execution_mode == "backend_direct":
target["runtime_probe"] = (await self._probe_backend_runtime(template=template, params=cleaned_params))
else:
target["runtime_probe"] = self._default_runtime_probe(template).model_dump(mode="json")
bundle["servers"] = servers
row.config = bundle
await self.db.flush()
return {
"server_id": str(server_id),
"is_valid": True,
"preview": preview.model_dump(mode="json"),
"runtime_probe": dict(target.get("runtime_probe") or {}),
}
async def confirm_client_apply(self, *, server_id: str) -> dict[str, Any]:
row, bundle = await self._get_or_init_bundle()
servers = [self._normalize_server_record(item) for item in bundle.get("servers", []) if isinstance(item, dict)]
target = self._find_server(servers, server_id=server_id)
if target is None:
raise LookupError("MCP 配置不存在")
template = self._get_template(str(target.get("template_id") or ""))
if template.execution_mode == "backend_direct":
raise ValueError("后端直连模板不需要“已在本机执行”确认")
confirmed_at = now_cn_naive().isoformat()
target["client_apply_state"] = "user_confirmed"
target["confirmed_at"] = confirmed_at
target["updated_at"] = confirmed_at
bundle["servers"] = servers
row.config = bundle
await self.db.flush()
return McpServerConfigPayload.model_validate(target).model_dump(mode="json")
async def get_preferred_backend_direct_server(self, *, template_id: str | None = None) -> dict[str, Any] | None:
_, bundle = await self._get_or_init_bundle()
servers = [self._normalize_server_record(item) for item in bundle.get("servers", []) if isinstance(item, dict)]
ordered = sorted(servers, key=lambda item: str(item.get("updated_at") or ""), reverse=True)
for item in ordered:
if str(item.get("execution_mode") or "") != "backend_direct":
continue
if template_id and str(item.get("template_id") or "") != str(template_id):
continue
return McpServerConfigPayload.model_validate(item).model_dump(mode="json")
return None
async def search_external(self, *, query: str, result_limit: int = 3) -> list[ExternalEvidenceItem]:
query_text = str(query or "").strip()
if not query_text:
return []
server = await self.get_preferred_backend_direct_server(template_id="microsoft_learn")
if not server:
return []
template = self._get_template(str(server.get("template_id") or ""))
if template.id != "microsoft_learn":
return []
runtime = MicrosoftLearnMCPRuntime()
return await runtime.search(query=query_text, result_limit=max(1, int(result_limit or 3)))
async def _get_or_init_bundle(self) -> tuple[Integration, dict[str, Any]]:
row = (
await self.db.execute(
select(Integration).where(
and_(
Integration.tenant_id == self.tenant_id,
Integration.type == INTEGRATION_TYPE,
)
)
)
).scalar_one_or_none()
if row is None:
row = Integration(
tenant_id=self.tenant_id,
type=INTEGRATION_TYPE,
is_active=True,
config={"version": 1, "servers": []},
)
self.db.add(row)
await self.db.flush()
bundle = dict(row.config or {}) if isinstance(row.config, dict) else {}
if not isinstance(bundle.get("servers"), list):
bundle["servers"] = []
if not isinstance(bundle.get("version"), int):
bundle["version"] = 1
if bundle != row.config:
row.config = bundle
await self.db.flush()
return row, bundle
def _get_template(self, template_id: str) -> McpTemplateSpec:
template = self._catalog.get(str(template_id or ""))
if template is None:
raise ValueError(f"未知 MCP 模板：{template_id}")
return template
def _normalize_server_record(self, item: dict[str, Any]) -> dict[str, Any]:
normalized = dict(item or {})
template = self._get_template(str(normalized.get("template_id") or ""))
normalized["transport"] = template.transport
normalized["execution_mode"] = template.execution_mode
if template.execution_mode == "backend_direct":
normalized["client_apply_state"] = "not_required"
else:
state = str(normalized.get("client_apply_state") or "").strip()
normalized["client_apply_state"] = state if state in {"pending_local_apply", "user_confirmed"} else "pending_local_apply"
if not isinstance(normalized.get("runtime_probe"), dict):
normalized["runtime_probe"] = self._default_runtime_probe(template).model_dump(mode="json")
if not isinstance(normalized.get("preview"), dict):
normalized["preview"] = self._build_preview(
template=template,
name=str(normalized.get("name") or ""),
params=dict(normalized.get("params") or {}),
).model_dump(mode="json")
return normalized
@staticmethod
def _default_client_apply_state(template: McpTemplateSpec) -> str:
return "not_required" if template.execution_mode == "backend_direct" else "pending_local_apply"
@staticmethod
def _default_runtime_probe(template: McpTemplateSpec) -> McpRuntimeProbe:
if template.execution_mode == "backend_direct":
return McpRuntimeProbe(
status="unknown",
checked_at=None,
message="保存后由 PMFlow 后端直连；点击“重新校验”可执行 MCP 握手探测。",
tool_names=[],
)
return McpRuntimeProbe(
status="local_bridge_required",
checked_at=None,
message="该模板依赖 PMFlow Local Bridge 或兼容 MCP host 在客户本机承载。",
tool_names=[],
)
@staticmethod
def _normalize_server_name(value: str) -> str:
normalized = _VALID_NAME_CHARS.sub("-", str(value or "").strip().lower())
normalized = re.sub(r"-{2,}", "-", normalized).strip("-_")
if not normalized:
raise ValueError("MCP 配置名不能为空")
return normalized[:80]
@staticmethod
def _find_server(servers: list[dict[str, Any]], *, server_id: str) -> dict[str, Any] | None:
for item in servers:
if str(item.get("id") or "") == str(server_id or ""):
return item
return None
def _validate_params(self, template: McpTemplateSpec, params: dict[str, Any]) -> dict[str, Any]:
raw = dict(params or {})
if template.id == "microsoft_learn":
return {}
if template.id == "github":
for forbidden_key in ("access_token", "token", "personal_access_token", "pat"):
if str(raw.get(forbidden_key) or "").strip():
raise ValueError("GitHub MCP 只接受 Token 环境变量名，不要直接填写明文 token")
env_var_name = str(raw.get("token_env_var_name") or "").strip().upper()
if not env_var_name:
raise ValueError("token_env_var_name 不能为空")
if not _VALID_ENV_VAR.match(env_var_name):
raise ValueError("token_env_var_name 必须是合法的环境变量名")
return {"token_env_var_name": env_var_name}
if template.id == "filesystem":
value = raw.get("allowed_paths")
paths: list[str] = []
if isinstance(value, str):
paths = [item.strip() for item in value.splitlines() if item.strip()]
elif isinstance(value, list):
paths = [str(item).strip() for item in value if str(item).strip()]
if not paths:
raise ValueError("allowed_paths 至少需要一个目录")
return {"allowed_paths": paths}
if template.id == "playwright":
output_dir = str(raw.get("output_dir") or "~/.pmflow/mcp/playwright-output").strip()
return {"output_dir": output_dir}
raise ValueError(f"未知模板校验规则：{template.id}")
def _build_preview(self, *, template: McpTemplateSpec, name: str, params: dict[str, Any]) -> McpCommandPreview:
if template.id == "microsoft_learn":
return McpCommandPreview(
add_command="无需本机命令。保存后由 PMFlow 后端直接连接 Microsoft Learn MCP。",
verify_command="点击“重新校验”后，PMFlow 会从后端执行 MCP initialize 和 tools/list。",
expected_output="校验成功后，你会看到后端可用工具，例如 microsoft_docs_search、microsoft_docs_fetch。",
)
if template.id == "github":
env_name = str(params["token_env_var_name"])
return McpCommandPreview(
add_command=(
f"需要在客户本机的 PMFlow Local Bridge 或兼容 MCP host 中接入 GitHub MCP，"
f"并引用环境变量 {env_name}。"
),
verify_command="本机 bridge/host 接好后，再回到这里执行“标记为已在本机执行”。",
expected_output=f"bridge/host 中应能看到 GitHub MCP 已启用，并通过环境变量 {env_name} 读取 token。",
)
if template.id == "filesystem":
quoted_paths = " ".join(f'"{item}"' for item in params["allowed_paths"])
return McpCommandPreview(
add_command=(
"需要在客户本机的 PMFlow Local Bridge 或兼容 MCP host 中启动 Filesystem MCP，"
f"允许目录：{quoted_paths}"
),
verify_command="本机 bridge/host 接好后，再回到这里执行“标记为已在本机执行”。",
expected_output=f"bridge/host 中应能看到 Filesystem MCP 已启用，并暴露目录 {quoted_paths}。",
)
if template.id == "playwright":
output_dir = str(params["output_dir"])
return McpCommandPreview(
add_command=(
"需要在客户本机的 PMFlow Local Bridge 或兼容 MCP host 中启动 Playwright MCP，"
f"输出目录：\"{output_dir}\""
),
verify_command="本机 bridge/host 接好后，再回到这里执行“标记为已在本机执行”。",
expected_output=f"bridge/host 中应能看到 Playwright MCP 已启用，输出目录为 {output_dir}。",
)
raise ValueError(f"未知模板命令生成规则：{template.id}")
async def _probe_backend_runtime(self, *, template: McpTemplateSpec, params: dict[str, Any]) -> dict[str, Any]:
_ = params
if template.id == "microsoft_learn":
return await MicrosoftLearnMCPRuntime().probe()
return {
"status": "error",
"checked_at": now_cn_naive().isoformat(),
"message": f"模板 {template.id} 暂未实现后端直连 runtime。",
"tool_names": [],
}

40
backend/app/services/project_knowledge_graph_service/mixin_event_queue.py
View File

@ -6,6 +6,7 @@ _WORK_ITEM_DELTA_CHANNEL = "work_item_delta"
_WORK_ITEM_DELTA_OFFSET_KEY = "last_work_item_updated_at"
_LEGACY_FEISHU_DELTA_CHANNEL = "feishu_delta"
_LEGACY_FEISHU_DELTA_OFFSET_KEY = "last_feishu_item_updated_at"
_SEMANTIC_FACT_EXTRACT_CHANNEL = "semantic_fact_extract"
class PKGEventQueueMixin:
@ -99,6 +100,8 @@ class PKGEventQueueMixin:
try:
if str(ev.channel) in {_WORK_ITEM_DELTA_CHANNEL, _LEGACY_FEISHU_DELTA_CHANNEL}:
await self._process_delta_event(ev)
elif str(ev.channel) == _SEMANTIC_FACT_EXTRACT_CHANNEL:
await self._process_semantic_fact_extract_event(ev)
ev.status = "done"
ev.error_message = None
ev.processed_at = _utc_now_naive()
@ -275,3 +278,40 @@ class PKGEventQueueMixin:
source_ref=source_ref,
metadata={"work_item_id": int(work_item.id)},
)
await self.enqueue_ingest_event(
channel=_SEMANTIC_FACT_EXTRACT_CHANNEL,
event_key=f"project:{int(work_item.project_id)}",
payload={
"entity_type": "project",
"entity_id": int(work_item.project_id),
"operation": "extract",
"trigger_type": "work_item_delta",
"work_item_id": int(work_item.id),
"tenant_id": int(self.tenant_id),
"source_updated_at": work_item.updated_at.isoformat() if work_item.updated_at else None,
},
status="pending",
)
async def _process_semantic_fact_extract_event(self, event: IngestEvent) -> None:
from ..semantic_fact_service import SemanticFactService
payload = event.payload if isinstance(event.payload, dict) else {}
entity_type = str(payload.get("entity_type") or "").strip().lower()
entity_id = payload.get("entity_id")
operation = str(payload.get("operation") or "extract").strip().lower() or "extract"
trigger_type = str(payload.get("trigger_type") or "event_queue").strip() or "event_queue"
if entity_id is None:
return
service = SemanticFactService(self.db, tenant_id=self.tenant_id)
if operation == "deactivate":
await service.deactivate_entity_facts(
entity_type=entity_type,
entity_id=int(entity_id),
)
return
await service.extract_entity_facts(
entity_type=entity_type,
entity_id=int(entity_id),
trigger_type=trigger_type,
)

4
backend/app/services/semantic_fact_service/__init__.py Normal file
View File

@ -0,0 +1,4 @@
from .service import SemanticFactService
from .hooks import register_semantic_fact_hooks
__all__ = ["SemanticFactService", "register_semantic_fact_hooks"]

152
backend/app/services/semantic_fact_service/hooks.py Normal file
View File

@ -0,0 +1,152 @@
from __future__ import annotations
from sqlalchemy import and_, event, select
from sqlalchemy.orm import Session
from ...models.ingest_event import IngestEvent
from ...models.knowledge_base_doc import KnowledgeBaseDoc
from ...models.project import Project
from ...models.signal import Signal
from ...models.task import Task
from ...models.truth_node import TruthNode
from ...models.user import User
from .shared import (
ENTITY_TYPE_TO_TRUTH_NODE_TYPE,
SEMANTIC_FACT_EXTRACT_CHANNEL,
TRUTH_NODE_TYPE_TO_ENTITY_TYPE,
_normalized_entity_type,
_status_is_inactive,
)
_SEMANTIC_FACT_INFO_KEY = "_pmflow_semantic_fact_events"
_SEMANTIC_FACT_APPLYING_KEY = "_pmflow_semantic_fact_applying"
def _truth_node_entity_type(node: TruthNode) -> str:
return TRUTH_NODE_TYPE_TO_ENTITY_TYPE.get(str(getattr(node, "node_type", "") or "").strip().lower(), "")
def _build_payload(obj: object, *, deleted: bool) -> dict | None:
entity_type = ""
if isinstance(obj, Project):
entity_type = "project"
elif isinstance(obj, Task):
entity_type = "task"
elif isinstance(obj, Signal):
entity_type = "signal"
elif isinstance(obj, KnowledgeBaseDoc):
entity_type = "document"
elif isinstance(obj, User):
entity_type = "user"
elif isinstance(obj, TruthNode):
entity_type = _truth_node_entity_type(obj)
if not entity_type:
return None
entity_id = getattr(obj, "id", None)
tenant_id = getattr(obj, "tenant_id", None)
if entity_id is None or tenant_id is None:
return None
operation = "deactivate" if deleted else "extract"
if not deleted:
if isinstance(obj, User):
operation = "extract" if bool(getattr(obj, "is_active", True)) and not _status_is_inactive(getattr(obj, "status", "")) else "deactivate"
elif _status_is_inactive(getattr(obj, "status", "")):
operation = "deactivate"
updated_at = getattr(obj, "updated_at", None)
return {
"event_key": f"{_normalized_entity_type(entity_type)}:{int(entity_id)}",
"payload": {
"entity_type": _normalized_entity_type(entity_type),
"entity_id": int(entity_id),
"operation": operation,
"trigger_type": "orm_commit",
"tenant_id": int(tenant_id),
"source_updated_at": updated_at.isoformat() if updated_at is not None else None,
},
}
def register_semantic_fact_hooks() -> None:
if getattr(register_semantic_fact_hooks, "_registered", False):
return
@event.listens_for(Session, "after_flush")
def _collect_semantic_fact_events(session: Session, _flush_context) -> None: # pragma: no cover
if bool(session.info.get(_SEMANTIC_FACT_APPLYING_KEY)):
return
payload_map = session.info.get(_SEMANTIC_FACT_INFO_KEY)
if not isinstance(payload_map, dict):
payload_map = {}
session.info[_SEMANTIC_FACT_INFO_KEY] = payload_map
objects = set(session.new).union(session.dirty).union(session.deleted)
deleted_objects = set(session.deleted)
for obj in objects:
if isinstance(obj, IngestEvent):
continue
payload = _build_payload(obj, deleted=obj in deleted_objects)
if not payload:
continue
payload_map[str(payload["event_key"])] = payload["payload"]
@event.listens_for(Session, "after_flush_postexec")
def _upsert_semantic_fact_events(session: Session, _flush_context) -> None: # pragma: no cover
if bool(session.info.get(_SEMANTIC_FACT_APPLYING_KEY)):
return
payload_map = session.info.get(_SEMANTIC_FACT_INFO_KEY)
if not isinstance(payload_map, dict) or not payload_map:
return
session.info[_SEMANTIC_FACT_APPLYING_KEY] = True
try:
for event_key, payload in payload_map.items():
tenant_id = int(payload.get("tenant_id") or 0)
if tenant_id <= 0:
continue
row = session.execute(
select(IngestEvent).where(
and_(
IngestEvent.tenant_id == tenant_id,
IngestEvent.channel == SEMANTIC_FACT_EXTRACT_CHANNEL,
IngestEvent.event_key == str(event_key),
)
)
).scalar_one_or_none()
if row is None:
session.add(
IngestEvent(
tenant_id=tenant_id,
channel=SEMANTIC_FACT_EXTRACT_CHANNEL,
event_key=str(event_key),
status="pending",
payload=payload,
retry_count=0,
error_message=None,
processed_at=None,
)
)
else:
row.status = "pending"
row.payload = payload
row.retry_count = 0
row.error_message = None
row.processed_at = None
payload_map.clear()
finally:
session.info[_SEMANTIC_FACT_APPLYING_KEY] = False
@event.listens_for(Session, "after_commit")
def _clear_semantic_fact_events_after_commit(session: Session) -> None: # pragma: no cover
session.info.pop(_SEMANTIC_FACT_INFO_KEY, None)
session.info.pop(_SEMANTIC_FACT_APPLYING_KEY, None)
@event.listens_for(Session, "after_rollback")
def _clear_semantic_fact_events_after_rollback(session: Session) -> None: # pragma: no cover
session.info.pop(_SEMANTIC_FACT_INFO_KEY, None)
session.info.pop(_SEMANTIC_FACT_APPLYING_KEY, None)
setattr(register_semantic_fact_hooks, "_registered", True)
__all__ = ["register_semantic_fact_hooks"]

567
backend/app/services/semantic_fact_service/mixin_admin.py Normal file
View File

@ -0,0 +1,567 @@
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from .shared import *
class SemanticFactAdminMixin:
async def _list_project_snapshots(self, *, limit: int) -> list[dict[str, Any]]:
rows = (
await self.db.execute(
select(Project)
.where(Project.tenant_id == self.tenant_id)
.order_by(Project.updated_at.desc(), Project.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": "project",
"entity_id": int(row.id),
"display_name": str(row.title or row.name or f"项目#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": not _status_is_inactive(row.status),
"summary": str(row.description or "").strip(),
}
for row in rows
]
async def _list_task_snapshots(self, *, limit: int) -> list[dict[str, Any]]:
rows = (
await self.db.execute(
select(Task)
.where(Task.tenant_id == self.tenant_id)
.order_by(Task.updated_at.desc(), Task.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": "task",
"entity_id": int(row.id),
"display_name": str(row.title or f"任务#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": not _status_is_inactive(row.status),
"summary": str(row.description or "").strip(),
}
for row in rows
]
async def _list_signal_snapshots(self, *, limit: int) -> list[dict[str, Any]]:
rows = (
await self.db.execute(
select(Signal)
.where(Signal.tenant_id == self.tenant_id)
.order_by(Signal.updated_at.desc(), Signal.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": "signal",
"entity_id": int(row.id),
"display_name": str(row.title or row.summary or f"信号#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": not _status_is_inactive(row.status),
"summary": str(row.summary or row.content or "").strip(),
}
for row in rows
]
async def _list_document_snapshots(self, *, limit: int) -> list[dict[str, Any]]:
rows = (
await self.db.execute(
select(KnowledgeBaseDoc)
.where(KnowledgeBaseDoc.tenant_id == self.tenant_id)
.order_by(KnowledgeBaseDoc.updated_at.desc(), KnowledgeBaseDoc.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": "document",
"entity_id": int(row.id),
"display_name": str(row.title or f"文档#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": not _status_is_inactive(row.status),
"summary": f"{str(row.file_type or '').strip()} / {str(row.source_type or '').strip()}",
}
for row in rows
]
async def _list_user_snapshots(self, *, limit: int) -> list[dict[str, Any]]:
rows = (
await self.db.execute(
select(User)
.where(User.tenant_id == self.tenant_id)
.order_by(User.updated_at.desc(), User.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": "user",
"entity_id": int(row.id),
"display_name": str(row.name or row.username or f"成员#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": bool(row.is_active) and not _status_is_inactive(row.status),
"summary": f"{str(row.role or '').strip()} / {str(row.username or '').strip()}",
}
for row in rows
]
async def _list_truth_node_snapshots(self, *, entity_type: str, limit: int) -> list[dict[str, Any]]:
truth_node_type = ENTITY_TYPE_TO_TRUTH_NODE_TYPE[entity_type]
rows = (
await self.db.execute(
select(TruthNode)
.where(
and_(
TruthNode.tenant_id == self.tenant_id,
TruthNode.node_type == truth_node_type,
)
)
.order_by(TruthNode.updated_at.desc(), TruthNode.id.desc())
.limit(limit)
)
).scalars().all()
return [
{
"entity_type": entity_type,
"entity_id": int(row.id),
"display_name": str(row.title or f"{entity_type}#{int(row.id)}").strip(),
"object_status": str(row.status or "").strip(),
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"is_extractable": not _status_is_inactive(row.status),
"summary": str((row.content or {}).get("summary") or "").strip() if isinstance(row.content, dict) else "",
}
for row in rows
]
async def list_object_snapshots(
self,
*,
entity_types: Optional[list[str]] = None,
limit_per_type: int = 200,
) -> list[dict[str, Any]]:
normalized_types = [
_normalized_entity_type(item)
for item in list(entity_types or sorted(SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES))
if _normalized_entity_type(item) in SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES
]
if not normalized_types:
normalized_types = sorted(SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES)
limit = max(1, min(int(limit_per_type), 1000))
items: list[dict[str, Any]] = []
for entity_type in normalized_types:
if entity_type == "project":
items.extend(await self._list_project_snapshots(limit=limit))
elif entity_type == "task":
items.extend(await self._list_task_snapshots(limit=limit))
elif entity_type == "signal":
items.extend(await self._list_signal_snapshots(limit=limit))
elif entity_type == "document":
items.extend(await self._list_document_snapshots(limit=limit))
elif entity_type == "user":
items.extend(await self._list_user_snapshots(limit=limit))
elif entity_type in {"requirement", "release", "review"}:
items.extend(await self._list_truth_node_snapshots(entity_type=entity_type, limit=limit))
return items
async def _load_object_metadata_maps(
self,
*,
items: list[dict[str, Any]],
) -> tuple[dict[tuple[str, int], int], dict[tuple[str, int], list[str]], dict[tuple[str, int], dict[str, Any]], dict[tuple[str, int], dict[str, Any]]]:
if not items:
return {}, {}, {}, {}
keys = {(str(item["entity_type"]), int(item["entity_id"])) for item in items}
entity_types = sorted({entity_type for entity_type, _entity_id in keys})
fact_rows = (
await self.db.execute(
select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type.in_(entity_types),
SemanticFact.is_active.is_(True),
)
)
)
).scalars().all()
fact_counts: dict[tuple[str, int], int] = {}
fact_texts: dict[tuple[str, int], list[str]] = {}
for row in fact_rows:
key = (str(row.entity_type), int(row.entity_id))
if key not in keys:
continue
fact_counts[key] = int(fact_counts.get(key) or 0) + 1
fact_texts.setdefault(key, []).append(
" ".join(
part for part in [
str(row.fact_key or "").strip(),
str(row.fact_value_text or "").strip(),
str(row.fact_value_norm or "").strip(),
] if part
)
)
run_rows = (
await self.db.execute(
select(SemanticFactExtractionRun)
.where(
and_(
SemanticFactExtractionRun.tenant_id == self.tenant_id,
SemanticFactExtractionRun.entity_type.in_(entity_types),
)
)
.order_by(SemanticFactExtractionRun.id.desc())
)
).scalars().all()
latest_runs: dict[tuple[str, int], dict[str, Any]] = {}
for row in run_rows:
key = (str(row.entity_type), int(row.entity_id))
if key not in keys or key in latest_runs:
continue
latest_runs[key] = {
"id": int(row.id),
"status": str(row.status or "").strip(),
"trigger_type": str(row.trigger_type or "").strip(),
"facts_written": int(row.facts_written or 0),
"failure_reason": str(row.failure_reason or "").strip() or None,
"extract_version": str(row.extract_version or "").strip() or None,
"started_at": str(row.started_at or "").strip() or None,
"finished_at": str(row.finished_at or "").strip() or None,
}
event_rows = (
await self.db.execute(
select(IngestEvent).where(
and_(
IngestEvent.tenant_id == self.tenant_id,
IngestEvent.channel == SEMANTIC_FACT_EXTRACT_CHANNEL,
)
)
)
).scalars().all()
latest_events: dict[tuple[str, int], dict[str, Any]] = {}
for row in event_rows:
payload = row.payload if isinstance(row.payload, dict) else {}
key = (
_normalized_entity_type(payload.get("entity_type") or ""),
int(payload.get("entity_id") or 0),
)
if key not in keys:
continue
latest_events[key] = {
"id": int(row.id),
"status": str(row.status or "").strip(),
"retry_count": int(row.retry_count or 0),
"error_message": str(row.error_message or "").strip() or None,
"processed_at": row.processed_at.isoformat() if row.processed_at else None,
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
"created_at": row.created_at.isoformat() if row.created_at else None,
"payload": payload,
}
return fact_counts, fact_texts, latest_runs, latest_events
@staticmethod
def _resolve_last_status(*, event: Optional[dict[str, Any]], run: Optional[dict[str, Any]]) -> str:
if event and str(event.get("status") or "").strip():
return str(event.get("status") or "").strip()
if run and str(run.get("status") or "").strip():
return str(run.get("status") or "").strip()
return "idle"
async def list_admin_objects(
self,
*,
entity_types: Optional[list[str]] = None,
q: Optional[str] = None,
status: Optional[str] = None,
limit_per_type: int = 200,
) -> list[dict[str, Any]]:
items = await self.list_object_snapshots(entity_types=entity_types, limit_per_type=limit_per_type)
fact_counts, fact_texts, latest_runs, latest_events = await self._load_object_metadata_maps(items=items)
query_text = str(q or "").strip().lower()
status_filter = str(status or "").strip().lower()
results: list[dict[str, Any]] = []
for item in items:
key = (str(item["entity_type"]), int(item["entity_id"]))
last_run = latest_runs.get(key)
last_event = latest_events.get(key)
last_status = self._resolve_last_status(event=last_event, run=last_run)
result = {
**item,
"facts_count": int(fact_counts.get(key) or 0),
"last_extracted_at": (
str((last_run or {}).get("finished_at") or (last_run or {}).get("started_at") or "").strip() or None
),
"last_status": last_status,
"queue_event": last_event,
"last_run": last_run,
}
if query_text:
haystack = " ".join(
[
str(item.get("display_name") or ""),
str(item.get("summary") or ""),
str(item.get("entity_id") or ""),
" ".join(fact_texts.get(key) or []),
]
).lower()
if query_text not in haystack:
continue
if status_filter and status_filter not in {"all", last_status.lower()}:
continue
results.append(result)
results.sort(
key=lambda row: (
_parse_iso_datetime(row.get("last_extracted_at")) or _parse_iso_datetime(row.get("updated_at")) or datetime(1970, 1, 1, tzinfo=UTC),
int(row.get("entity_id") or 0),
),
reverse=True,
)
return results
async def get_admin_detail(
self,
*,
entity_type: str,
entity_id: int,
) -> dict[str, Any]:
snapshot = await self._load_entity_snapshot(entity_type=entity_type, entity_id=int(entity_id))
if snapshot is None:
raise ValueError("semantic_fact_entity_not_found")
facts = (
await self.db.execute(
select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type == _normalized_entity_type(entity_type),
SemanticFact.entity_id == int(entity_id),
SemanticFact.is_active.is_(True),
)
).order_by(SemanticFact.confidence.desc(), SemanticFact.updated_at.desc(), SemanticFact.id.desc())
)
).scalars().all()
runs = (
await self.db.execute(
select(SemanticFactExtractionRun).where(
and_(
SemanticFactExtractionRun.tenant_id == self.tenant_id,
SemanticFactExtractionRun.entity_type == _normalized_entity_type(entity_type),
SemanticFactExtractionRun.entity_id == int(entity_id),
)
).order_by(SemanticFactExtractionRun.id.desc()).limit(10)
)
).scalars().all()
event = (
await self.db.execute(
select(IngestEvent).where(
and_(
IngestEvent.tenant_id == self.tenant_id,
IngestEvent.channel == SEMANTIC_FACT_EXTRACT_CHANNEL,
IngestEvent.event_key == f"{_normalized_entity_type(entity_type)}:{int(entity_id)}",
)
)
)
).scalar_one_or_none()
return {
"object": {
"entity_type": str(snapshot["entity_type"]),
"entity_id": int(snapshot["entity_id"]),
"display_name": str(snapshot["display_name"]),
"object_status": str(snapshot.get("object_status") or ""),
"updated_at": snapshot.get("updated_at"),
"is_extractable": bool(snapshot.get("is_extractable")),
"source_payload": snapshot.get("source_payload") or {},
},
"facts": [
{
"id": int(row.id),
"fact_namespace": str(row.fact_namespace or "").strip(),
"fact_key": str(row.fact_key or "").strip(),
"fact_value_text": str(row.fact_value_text or "").strip(),
"fact_value_norm": str(row.fact_value_norm or "").strip(),
"value_type": str(row.value_type or "").strip(),
"confidence": float(row.confidence or 0.0),
"evidence_excerpt": str(row.evidence_excerpt or "").strip() or None,
"freshness_at": str(row.freshness_at or "").strip() or None,
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
}
for row in facts
],
"recent_runs": [
{
"id": int(row.id),
"status": str(row.status or "").strip(),
"trigger_type": str(row.trigger_type or "").strip(),
"facts_written": int(row.facts_written or 0),
"failure_reason": str(row.failure_reason or "").strip() or None,
"extract_version": str(row.extract_version or "").strip() or None,
"started_at": str(row.started_at or "").strip() or None,
"finished_at": str(row.finished_at or "").strip() or None,
}
for row in runs
],
"queue_event": (
{
"id": int(event.id),
"status": str(event.status or "").strip(),
"retry_count": int(event.retry_count or 0),
"error_message": str(event.error_message or "").strip() or None,
"processed_at": event.processed_at.isoformat() if event.processed_at else None,
"updated_at": event.updated_at.isoformat() if event.updated_at else None,
"payload": event.payload if isinstance(event.payload, dict) else {},
}
if event is not None
else None
),
}
async def enqueue_backfill(
self,
*,
entity_types: Optional[list[str]] = None,
only_missing: bool = True,
limit_per_type: int = 200,
) -> dict[str, Any]:
items = await self.list_object_snapshots(entity_types=entity_types, limit_per_type=limit_per_type)
by_type: dict[str, int] = {}
existing_pairs: set[tuple[str, int]] = set()
if only_missing:
fact_rows = (
await self.db.execute(
select(SemanticFact.entity_type, SemanticFact.entity_id).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.is_active.is_(True),
)
)
)
).all()
existing_pairs = {
(str(entity_type), int(entity_id))
for entity_type, entity_id in fact_rows
}
total = 0
for item in items:
key = (str(item["entity_type"]), int(item["entity_id"]))
if not bool(item.get("is_extractable")):
continue
if only_missing and key in existing_pairs:
continue
await self.enqueue_entity_event(
entity_type=str(item["entity_type"]),
entity_id=int(item["entity_id"]),
operation="extract",
trigger_type="backfill",
extra_payload={
"source_updated_at": str(item.get("updated_at") or "").strip() or None,
},
)
by_type[str(item["entity_type"])] = int(by_type.get(str(item["entity_type"])) or 0) + 1
total += 1
return {
"enqueued": total,
"only_missing": bool(only_missing),
"by_type": by_type,
}
async def get_admin_overview(self) -> dict[str, Any]:
items = await self.list_object_snapshots(limit_per_type=500)
active_fact_total = int(
(
await self.db.execute(
select(func.count(SemanticFact.id)).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.is_active.is_(True),
)
)
)
).scalar_one()
or 0
)
event_rows = (
await self.db.execute(
select(IngestEvent).where(
and_(
IngestEvent.tenant_id == self.tenant_id,
IngestEvent.channel == SEMANTIC_FACT_EXTRACT_CHANNEL,
)
).order_by(IngestEvent.updated_at.desc(), IngestEvent.id.desc())
)
).scalars().all()
pending_rows = [row for row in event_rows if str(row.status or "").strip() in {"pending", "retry"}]
failed_rows = [row for row in event_rows if str(row.status or "").strip() == "dead_letter"]
running_count = int(
(
await self.db.execute(
select(func.count(SemanticFactExtractionRun.id)).where(
and_(
SemanticFactExtractionRun.tenant_id == self.tenant_id,
SemanticFactExtractionRun.status == "running",
)
)
)
).scalar_one()
or 0
)
all_runs = (
await self.db.execute(
select(SemanticFactExtractionRun).where(
SemanticFactExtractionRun.tenant_id == self.tenant_id
)
)
).scalars().all()
cutoff = datetime.now(UTC) - timedelta(hours=24)
success_24h = 0
failed_24h = 0
for row in all_runs:
pivot = _parse_iso_datetime(row.finished_at or row.started_at)
if pivot is None or pivot < cutoff:
continue
status = str(row.status or "").strip()
if status == "completed":
success_24h += 1
elif status == "failed":
failed_24h += 1
return {
"kpis": {
"object_total": len(items),
"active_fact_total": active_fact_total,
"success_24h": success_24h,
"failed_24h": failed_24h,
},
"queue": {
"pending": len(pending_rows),
"processing": running_count,
"failed": len(failed_rows),
"oldest_pending_at": pending_rows[-1].created_at.isoformat() if pending_rows else None,
"recent_failures": [
{
"event_key": str(row.event_key or ""),
"status": str(row.status or "").strip(),
"error_message": str(row.error_message or "").strip() or None,
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
}
for row in failed_rows[:5]
],
},
"entity_types": sorted(SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES),
}
__all__ = ["SemanticFactAdminMixin"]

370
backend/app/services/semantic_fact_service/mixin_extract.py Normal file
View File

@ -0,0 +1,370 @@
from __future__ import annotations
from .shared import *
class SemanticFactExtractMixin:
async def _create_extraction_run(
self,
*,
entity_type: str,
entity_id: int,
trigger_type: str,
input_fingerprint: str,
) -> SemanticFactExtractionRun:
run = SemanticFactExtractionRun(
tenant_id=self.tenant_id,
entity_type=_normalized_entity_type(entity_type),
entity_id=int(entity_id),
trigger_type=str(trigger_type or "").strip() or "manual_retry",
status="running",
input_fingerprint=input_fingerprint,
facts_written=0,
extract_version=SEMANTIC_FACT_EXTRACT_VERSION,
started_at=_utc_now_iso(),
)
self.db.add(run)
await self.db.flush()
return run
async def _load_project_snapshot(self, *, entity_id: int) -> Optional[dict[str, Any]]:
project = (
await self.db.execute(
select(Project).where(
and_(
Project.tenant_id == self.tenant_id,
Project.id == int(entity_id),
)
)
)
).scalar_one_or_none()
if project is None:
return None
return {
"entity_type": "project",
"entity_id": int(project.id),
"display_name": str(project.title or project.name or f"项目#{int(project.id)}").strip(),
"object_status": str(project.status or "").strip(),
"updated_at": project.updated_at.isoformat() if project.updated_at else None,
"is_extractable": not _status_is_inactive(project.status),
"source_payload": {
"project_id": int(project.id),
"name": str(project.name or "").strip(),
"title": str(project.title or "").strip(),
"description": str(project.description or "").strip(),
"status": str(project.status or "").strip(),
"assigned_role": str(project.assigned_role or "").strip(),
},
}
async def _load_task_snapshot(self, *, entity_id: int) -> Optional[dict[str, Any]]:
task = (
await self.db.execute(
select(Task).where(
and_(
Task.tenant_id == self.tenant_id,
Task.id == int(entity_id),
)
)
)
).scalar_one_or_none()
if task is None:
return None
return {
"entity_type": "task",
"entity_id": int(task.id),
"display_name": str(task.title or f"任务#{int(task.id)}").strip(),
"object_status": str(task.status or "").strip(),
"updated_at": task.updated_at.isoformat() if task.updated_at else None,
"is_extractable": not _status_is_inactive(task.status),
"source_payload": {
"task_id": int(task.id),
"project_id": int(task.project_id),
"truth_node_id": int(task.truth_node_id),
"assignee_id": int(task.assignee_id) if task.assignee_id is not None else None,
"title": str(task.title or "").strip(),
"description": str(task.description or "").strip(),
"status": str(task.status or "").strip(),
"priority": str(task.priority or "").strip(),
"required_skill_tag_ids": list(task.required_skill_tag_ids or []),
},
}
async def _load_signal_snapshot(self, *, entity_id: int) -> Optional[dict[str, Any]]:
signal = (
await self.db.execute(
select(Signal).where(
and_(
Signal.tenant_id == self.tenant_id,
Signal.id == int(entity_id),
)
)
)
).scalar_one_or_none()
if signal is None:
return None
return {
"entity_type": "signal",
"entity_id": int(signal.id),
"display_name": str(signal.title or signal.summary or f"信号#{int(signal.id)}").strip(),
"object_status": str(signal.status or "").strip(),
"updated_at": signal.updated_at.isoformat() if signal.updated_at else None,
"is_extractable": not _status_is_inactive(signal.status),
"source_payload": {
"signal_id": int(signal.id),
"title": str(signal.title or "").strip(),
"summary": str(signal.summary or "").strip(),
"content": str(signal.content or "").strip(),
"status": str(signal.status or "").strip(),
"priority": str(signal.priority or "").strip(),
"signal_type": str(signal.signal_type or "").strip(),
"source": str(signal.source or "").strip(),
"matched_project_id": int(signal.matched_project_id) if signal.matched_project_id is not None else None,
"processed": bool(signal.processed),
},
}
async def _load_document_snapshot(self, *, entity_id: int) -> Optional[dict[str, Any]]:
doc = (
await self.db.execute(
select(KnowledgeBaseDoc).where(
and_(
KnowledgeBaseDoc.tenant_id == self.tenant_id,
KnowledgeBaseDoc.id == int(entity_id),
)
)
)
).scalar_one_or_none()
if doc is None:
return None
chunk_rows = (
await self.db.execute(
select(KnowledgeBaseChunk).where(
and_(
KnowledgeBaseChunk.tenant_id == self.tenant_id,
KnowledgeBaseChunk.doc_id == int(doc.id),
)
).order_by(KnowledgeBaseChunk.chunk_index.asc()).limit(3)
)
).scalars().all()
preview_text = "\n".join(str(row.content or "").strip() for row in chunk_rows if str(row.content or "").strip())
return {
"entity_type": "document",
"entity_id": int(doc.id),
"display_name": str(doc.title or f"文档#{int(doc.id)}").strip(),
"object_status": str(doc.status or "").strip(),
"updated_at": doc.updated_at.isoformat() if doc.updated_at else None,
"is_extractable": not _status_is_inactive(doc.status),
"source_payload": {
"doc_id": int(doc.id),
"title": str(doc.title or "").strip(),
"file_type": str(doc.file_type or "").strip(),
"source_type": str(doc.source_type or "").strip(),
"file_url": str(doc.file_url or "").strip(),
"status": str(doc.status or "").strip(),
"preview_text": preview_text[:2000],
},
}
async def _load_user_snapshot(self, *, entity_id: int) -> Optional[dict[str, Any]]:
user = (
await self.db.execute(
select(User).where(
and_(
User.tenant_id == self.tenant_id,
User.id == int(entity_id),
)
)
)
).scalar_one_or_none()
if user is None:
return None
return {
"entity_type": "user",
"entity_id": int(user.id),
"display_name": str(user.name or user.username or f"成员#{int(user.id)}").strip(),
"object_status": str(user.status or "").strip(),
"updated_at": user.updated_at.isoformat() if user.updated_at else None,
"is_extractable": bool(user.is_active) and not _status_is_inactive(user.status),
"source_payload": {
"user_id": int(user.id),
"username": str(user.username or "").strip(),
"name": str(user.name or "").strip(),
"role": str(user.role or "").strip(),
"status": str(user.status or "").strip(),
"is_active": bool(user.is_active),
"bio": str(user.bio or "").strip(),
"assigned_domains": list(user.assigned_domains or []),
},
}
async def _load_truth_node_snapshot(self, *, entity_type: str, entity_id: int) -> Optional[dict[str, Any]]:
truth_node_type = ENTITY_TYPE_TO_TRUTH_NODE_TYPE.get(_normalized_entity_type(entity_type))
if not truth_node_type:
return None
node = (
await self.db.execute(
select(TruthNode).where(
and_(
TruthNode.tenant_id == self.tenant_id,
TruthNode.id == int(entity_id),
TruthNode.node_type == truth_node_type,
)
)
)
).scalar_one_or_none()
if node is None:
return None
normalized_entity_type = TRUTH_NODE_TYPE_TO_ENTITY_TYPE.get(str(node.node_type or "").strip().lower(), "")
if not normalized_entity_type:
return None
return {
"entity_type": normalized_entity_type,
"entity_id": int(node.id),
"display_name": str(node.title or f"{normalized_entity_type}#{int(node.id)}").strip(),
"object_status": str(node.status or "").strip(),
"updated_at": node.updated_at.isoformat() if node.updated_at else None,
"is_extractable": not _status_is_inactive(node.status),
"source_payload": {
"node_id": int(node.id),
"node_type": str(node.node_type or "").strip(),
"project_id": int(node.project_id) if node.project_id is not None else None,
"signal_id": int(node.signal_id) if node.signal_id is not None else None,
"title": str(node.title or "").strip(),
"status": str(node.status or "").strip(),
"version": int(node.version or 1),
"content": node.content if isinstance(node.content, dict) else {},
},
}
async def _load_entity_snapshot(
self,
*,
entity_type: str,
entity_id: int,
) -> Optional[dict[str, Any]]:
normalized_entity_type = _normalized_entity_type(entity_type)
if normalized_entity_type == "project":
return await self._load_project_snapshot(entity_id=int(entity_id))
if normalized_entity_type == "task":
return await self._load_task_snapshot(entity_id=int(entity_id))
if normalized_entity_type == "signal":
return await self._load_signal_snapshot(entity_id=int(entity_id))
if normalized_entity_type == "document":
return await self._load_document_snapshot(entity_id=int(entity_id))
if normalized_entity_type == "user":
return await self._load_user_snapshot(entity_id=int(entity_id))
if normalized_entity_type in {"requirement", "release", "review"}:
return await self._load_truth_node_snapshot(entity_type=normalized_entity_type, entity_id=int(entity_id))
raise ValueError(f"unsupported_semantic_fact_entity_type:{normalized_entity_type}")
def _render_extraction_prompt(self, *, entity_type: str, source_payload: dict[str, Any]) -> str:
normalized_entity_type = _normalized_entity_type(entity_type)
prompt_template = SEMANTIC_FACT_PROMPTS.get(normalized_entity_type)
if prompt_template is None:
raise ValueError(f"unsupported_semantic_fact_prompt:{normalized_entity_type}")
payload_text = json.dumps(source_payload, ensure_ascii=False, sort_keys=True)
if normalized_entity_type == "project":
return prompt_template.replace("<<project_payload>>", payload_text).strip()
return prompt_template.replace("<<entity_payload>>", payload_text).strip()
async def extract_entity_facts(
self,
*,
entity_type: str,
entity_id: int,
trigger_type: str,
) -> Optional[SemanticFactExtractionRun]:
from ..llm_service import llm_service
normalized_entity_type = _normalized_entity_type(entity_type)
snapshot = await self._load_entity_snapshot(
entity_type=normalized_entity_type,
entity_id=int(entity_id),
)
if snapshot is None:
await self.deactivate_entity_facts(entity_type=normalized_entity_type, entity_id=int(entity_id))
return None
if not bool(snapshot.get("is_extractable")):
await self.deactivate_entity_facts(entity_type=normalized_entity_type, entity_id=int(entity_id))
return None
source_payload = dict(snapshot.get("source_payload") or {})
fingerprint = hashlib.sha1(
json.dumps(source_payload, ensure_ascii=False, sort_keys=True).encode("utf-8")
).hexdigest()
run = await self._create_extraction_run(
entity_type=normalized_entity_type,
entity_id=int(entity_id),
trigger_type=trigger_type,
input_fingerprint=fingerprint,
)
try:
prompt = self._render_extraction_prompt(
entity_type=normalized_entity_type,
source_payload=source_payload,
)
extraction = await llm_service.generate_structured_object(
prompt,
db=self.db,
tenant_id=self.tenant_id,
function_name=SEMANTIC_FACT_FUNCTION_NAMES[normalized_entity_type],
temperature=0.0,
max_tokens=1200,
)
payload = extraction.payload if isinstance(extraction.payload, dict) else {}
fact_rows = list(payload.get("facts") or [])
await self.deactivate_entity_facts(
entity_type=normalized_entity_type,
entity_id=int(entity_id),
)
written = 0
for item in fact_rows:
if not isinstance(item, dict):
continue
fact_key = str(item.get("fact_key") or "").strip()
fact_value_text = str(item.get("fact_value_text") or "").strip()
if not fact_key or not fact_value_text:
continue
await self.upsert_fact(
entity_type=normalized_entity_type,
entity_id=int(entity_id),
fact_namespace=str(item.get("fact_namespace") or "general").strip() or "general",
fact_key=fact_key,
fact_value_text=fact_value_text,
fact_value_norm=str(item.get("fact_value_norm") or "").strip() or None,
value_type=str(item.get("value_type") or "keyword").strip() or "keyword",
confidence=float(item.get("confidence") or 0.0),
evidence_source_type=normalized_entity_type,
evidence_source_id=int(entity_id),
evidence_excerpt=str(item.get("evidence_excerpt") or "").strip() or None,
extract_version=SEMANTIC_FACT_EXTRACT_VERSION,
freshness_at=str(snapshot.get("updated_at") or "").strip() or None,
)
written += 1
run.status = "completed"
run.facts_written = written
run.failure_reason = None
run.finished_at = _utc_now_iso()
await self.db.flush()
return run
except Exception as exc:
run.status = "failed"
run.failure_reason = str(exc)[:1000]
run.finished_at = _utc_now_iso()
await self.db.flush()
raise
async def extract_project_facts(self, *, project_id: int, trigger_type: str) -> SemanticFactExtractionRun:
run = await self.extract_entity_facts(
entity_type="project",
entity_id=int(project_id),
trigger_type=trigger_type,
)
if run is None:
raise ValueError(f"project_not_found:{int(project_id)}")
return run
__all__ = ["SemanticFactExtractMixin"]

198
backend/app/services/semantic_fact_service/mixin_query.py Normal file
View File

@ -0,0 +1,198 @@
from __future__ import annotations
from .shared import *
class SemanticFactQueryMixin:
async def upsert_fact(
self,
*,
entity_type: str,
entity_id: int,
fact_namespace: str,
fact_key: str,
fact_value_text: str,
fact_value_norm: Optional[str],
value_type: str,
confidence: float,
evidence_source_type: Optional[str],
evidence_source_id: Optional[int],
evidence_excerpt: Optional[str],
extract_version: Optional[str],
freshness_at: Optional[str],
) -> SemanticFact:
normalized_fact_value = _normalize_fact_value(fact_value_norm or fact_value_text)
row = (
await self.db.execute(
select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type == str(entity_type or "").strip(),
SemanticFact.entity_id == int(entity_id),
SemanticFact.fact_key == str(fact_key or "").strip(),
SemanticFact.fact_value_norm == normalized_fact_value,
)
)
)
).scalar_one_or_none()
if row is None:
row = SemanticFact(
tenant_id=self.tenant_id,
entity_type=str(entity_type or "").strip(),
entity_id=int(entity_id),
fact_namespace=str(fact_namespace or "").strip() or "general",
fact_key=str(fact_key or "").strip(),
fact_value_text=str(fact_value_text or "").strip(),
fact_value_norm=normalized_fact_value,
value_type=str(value_type or "").strip() or "keyword",
confidence=max(0.0, min(1.0, float(confidence or 0.0))),
evidence_source_type=str(evidence_source_type or "").strip() or None,
evidence_source_id=int(evidence_source_id) if evidence_source_id is not None else None,
evidence_excerpt=str(evidence_excerpt or "").strip() or None,
extract_version=str(extract_version or "").strip() or None,
freshness_at=str(freshness_at or "").strip() or None,
is_active=True,
)
self.db.add(row)
await self.db.flush()
return row
row.fact_namespace = str(fact_namespace or "").strip() or row.fact_namespace
row.fact_value_text = str(fact_value_text or "").strip() or row.fact_value_text
row.value_type = str(value_type or "").strip() or row.value_type
row.confidence = max(0.0, min(1.0, float(confidence or 0.0)))
row.evidence_source_type = str(evidence_source_type or "").strip() or row.evidence_source_type
row.evidence_source_id = int(evidence_source_id) if evidence_source_id is not None else row.evidence_source_id
row.evidence_excerpt = str(evidence_excerpt or "").strip() or row.evidence_excerpt
row.extract_version = str(extract_version or "").strip() or row.extract_version
row.freshness_at = str(freshness_at or "").strip() or row.freshness_at
row.is_active = True
await self.db.flush()
return row
async def deactivate_entity_facts(
self,
*,
entity_type: str,
entity_id: int,
) -> int:
normalized_entity_type = _normalized_entity_type(entity_type)
rows = (
await self.db.execute(
select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type == normalized_entity_type,
SemanticFact.entity_id == int(entity_id),
SemanticFact.is_active.is_(True),
)
)
)
).scalars().all()
for row in rows:
row.is_active = False
await self.db.flush()
return len(rows)
async def enqueue_entity_event(
self,
*,
entity_type: str,
entity_id: int,
operation: str,
trigger_type: str,
extra_payload: Optional[dict[str, Any]] = None,
) -> IngestEvent:
normalized_entity_type = _normalized_entity_type(entity_type)
event_key = f"{normalized_entity_type}:{int(entity_id)}"
payload = {
"entity_type": normalized_entity_type,
"entity_id": int(entity_id),
"operation": str(operation or "extract").strip() or "extract",
"trigger_type": str(trigger_type or "manual_retry").strip() or "manual_retry",
"tenant_id": int(self.tenant_id),
}
if isinstance(extra_payload, dict):
for key, value in extra_payload.items():
payload[str(key)] = value
row = (
await self.db.execute(
select(IngestEvent).where(
and_(
IngestEvent.tenant_id == self.tenant_id,
IngestEvent.channel == SEMANTIC_FACT_EXTRACT_CHANNEL,
IngestEvent.event_key == event_key,
)
)
)
).scalar_one_or_none()
if row is None:
row = IngestEvent(
tenant_id=self.tenant_id,
channel=SEMANTIC_FACT_EXTRACT_CHANNEL,
event_key=event_key,
status="pending",
payload=payload,
retry_count=0,
error_message=None,
processed_at=None,
)
self.db.add(row)
else:
row.status = "pending"
row.payload = payload
row.retry_count = 0
row.error_message = None
row.processed_at = None
await self.db.flush()
return row
async def query_facts(
self,
*,
entity_type: str,
query_type: str,
filters: Optional[list[dict[str, Any]]] = None,
entity_id: Optional[int] = None,
limit: int = 20,
) -> list[SemanticFact]:
_ = query_type
stmt = select(SemanticFact).where(
and_(
SemanticFact.tenant_id == self.tenant_id,
SemanticFact.entity_type == _normalized_entity_type(entity_type),
SemanticFact.is_active.is_(True),
)
)
if entity_id is not None:
stmt = stmt.where(SemanticFact.entity_id == int(entity_id))
for item in list(filters or []):
if not isinstance(item, dict):
continue
fact_key = str(item.get("fact_key") or item.get("field") or "").strip()
if fact_key:
stmt = stmt.where(SemanticFact.fact_key == fact_key)
raw_value = str(item.get("value") or item.get("normalized_value") or "").strip()
normalized_value = _normalize_fact_value(item.get("normalized_value") or raw_value)
if normalized_value:
stmt = stmt.where(
or_(
SemanticFact.fact_value_norm == normalized_value,
func.lower(SemanticFact.fact_value_text) == raw_value.lower(),
)
)
rows = (
await self.db.execute(
stmt.order_by(SemanticFact.confidence.desc(), SemanticFact.updated_at.desc(), SemanticFact.id.desc()).limit(
max(1, int(limit))
)
)
).scalars().all()
return list(rows)
__all__ = ["SemanticFactQueryMixin"]

19
backend/app/services/semantic_fact_service/service.py Normal file
View File

@ -0,0 +1,19 @@
from __future__ import annotations
from .mixin_admin import SemanticFactAdminMixin
from .mixin_extract import SemanticFactExtractMixin
from .mixin_query import SemanticFactQueryMixin
from .shared import AsyncSession
class SemanticFactService(
SemanticFactAdminMixin,
SemanticFactExtractMixin,
SemanticFactQueryMixin,
):
def __init__(self, db: AsyncSession, *, tenant_id: int) -> None:
self.db = db
self.tenant_id = int(tenant_id)
__all__ = ["SemanticFactService"]

162
backend/app/services/semantic_fact_service/shared.py Normal file
View File

@ -0,0 +1,162 @@
from __future__ import annotations
import hashlib
import json
import re
from datetime import UTC, datetime
from typing import Any, Optional
from sqlalchemy import and_, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from ...config.ai_prompts import (
PROMPT_SEMANTIC_FACT_EXTRACT_DOCUMENT,
PROMPT_SEMANTIC_FACT_EXTRACT_PROJECT,
PROMPT_SEMANTIC_FACT_EXTRACT_RELEASE,
PROMPT_SEMANTIC_FACT_EXTRACT_REQUIREMENT,
PROMPT_SEMANTIC_FACT_EXTRACT_REVIEW,
PROMPT_SEMANTIC_FACT_EXTRACT_SIGNAL,
PROMPT_SEMANTIC_FACT_EXTRACT_TASK,
PROMPT_SEMANTIC_FACT_EXTRACT_USER,
)
from ...models.ingest_event import IngestEvent
from ...models.knowledge_base_chunk import KnowledgeBaseChunk
from ...models.knowledge_base_doc import KnowledgeBaseDoc
from ...models.project import Project
from ...models.semantic_fact import SemanticFact
from ...models.semantic_fact_extraction_run import SemanticFactExtractionRun
from ...models.signal import Signal
from ...models.task import Task
from ...models.truth_node import TruthNode
from ...models.user import User
SEMANTIC_FACT_EXTRACT_VERSION = "semantic-facts-v1"
SEMANTIC_FACT_EXTRACT_CHANNEL = "semantic_fact_extract"
SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES = {
"project",
"task",
"signal",
"document",
"user",
"requirement",
"release",
"review",
}
SEMANTIC_FACT_INACTIVE_STATUSES = {
"archived",
"closed",
"deleted",
"disabled",
"inactive",
"offline",
"已关闭",
"已归档",
}
TRUTH_NODE_TYPE_TO_ENTITY_TYPE = {
"charter": "requirement",
"release": "release",
"review": "review",
}
ENTITY_TYPE_TO_TRUTH_NODE_TYPE = {
value: key for key, value in TRUTH_NODE_TYPE_TO_ENTITY_TYPE.items()
}
SEMANTIC_FACT_PROMPTS = {
"project": PROMPT_SEMANTIC_FACT_EXTRACT_PROJECT,
"task": PROMPT_SEMANTIC_FACT_EXTRACT_TASK,
"signal": PROMPT_SEMANTIC_FACT_EXTRACT_SIGNAL,
"document": PROMPT_SEMANTIC_FACT_EXTRACT_DOCUMENT,
"user": PROMPT_SEMANTIC_FACT_EXTRACT_USER,
"requirement": PROMPT_SEMANTIC_FACT_EXTRACT_REQUIREMENT,
"release": PROMPT_SEMANTIC_FACT_EXTRACT_RELEASE,
"review": PROMPT_SEMANTIC_FACT_EXTRACT_REVIEW,
}
SEMANTIC_FACT_FUNCTION_NAMES = {
entity_type: f"semantic_fact_extract_{entity_type}"
for entity_type in SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES
}
def _utc_now_iso() -> str:
return datetime.now(UTC).replace(microsecond=0).isoformat()
def _normalize_fact_value(value: Any) -> str:
text = str(value or "").strip().lower()
if not text:
return ""
text = text.replace("a股", "a股")
text = re.sub(r"\s+", "_", text)
text = re.sub(r"[^a-z0-9\u4e00-\u9fff_]+", "_", text)
text = re.sub(r"_+", "_", text).strip("_")
return text
def _normalized_entity_type(entity_type: Any) -> str:
value = str(entity_type or "").strip().lower()
if value == "doc":
return "document"
return value
def _normalized_status(value: Any) -> str:
return str(value or "").strip().lower()
def _status_is_inactive(value: Any) -> bool:
return _normalized_status(value) in SEMANTIC_FACT_INACTIVE_STATUSES
def _parse_iso_datetime(value: Any) -> Optional[datetime]:
text = str(value or "").strip()
if not text:
return None
try:
parsed = datetime.fromisoformat(text.replace("Z", "+00:00"))
except ValueError:
return None
if parsed.tzinfo is None:
return parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
__all__ = [
"AsyncSession",
"ENTITY_TYPE_TO_TRUTH_NODE_TYPE",
"IngestEvent",
"KnowledgeBaseChunk",
"KnowledgeBaseDoc",
"Project",
"PROMPT_SEMANTIC_FACT_EXTRACT_DOCUMENT",
"PROMPT_SEMANTIC_FACT_EXTRACT_PROJECT",
"PROMPT_SEMANTIC_FACT_EXTRACT_RELEASE",
"PROMPT_SEMANTIC_FACT_EXTRACT_REQUIREMENT",
"PROMPT_SEMANTIC_FACT_EXTRACT_REVIEW",
"PROMPT_SEMANTIC_FACT_EXTRACT_SIGNAL",
"PROMPT_SEMANTIC_FACT_EXTRACT_TASK",
"PROMPT_SEMANTIC_FACT_EXTRACT_USER",
"SEMANTIC_FACT_EXTRACT_VERSION",
"SEMANTIC_FACT_EXTRACT_CHANNEL",
"SEMANTIC_FACT_FUNCTION_NAMES",
"SEMANTIC_FACT_INACTIVE_STATUSES",
"SEMANTIC_FACT_PROMPTS",
"SEMANTIC_FACT_SUPPORTED_ENTITY_TYPES",
"SemanticFact",
"SemanticFactExtractionRun",
"Signal",
"Task",
"TRUTH_NODE_TYPE_TO_ENTITY_TYPE",
"TruthNode",
"User",
"_normalize_fact_value",
"_normalized_entity_type",
"_normalized_status",
"_parse_iso_datetime",
"_status_is_inactive",
"_utc_now_iso",
"and_",
"func",
"hashlib",
"json",
"or_",
"select",
]

0
backend/backend/tmp_acceptance/feishu_weather2_run_20260416_1000.json Normal file
View File

1651
backend/tests/test_adaptive_rag_mainline_contract.py
View File

File diff suppressed because it is too large Load Diff

729
backend/tests/test_ai_pmo_ingress_mainline.py
View File

@ -1522,7 +1522,7 @@ async def test_terminate_requirement_plan_scope_cancels_orphan_pending_work_item
@pytest.mark.asyncio
async def test_process_ingress_terminates_stale_requirement_scope_before_active_context_followup(monkeypatch) -> None:
async def test_process_ingress_does_not_terminate_requirement_scope_before_active_context_followup(monkeypatch) -> None:
cleanup_calls: list[dict[str, object]] = []
class _FakeDB:
@ -1563,8 +1563,8 @@ async def test_process_ingress_terminates_stale_requirement_scope_before_active_
self.active_context_called = True
_ = kwargs
assert self.prepared_session is not None
assert str(getattr(self.prepared_session, "last_intent", "") or "") == "query.answer"
assert getattr(self.prepared_session, "current_project_id", None) is None
assert str(getattr(self.prepared_session, "last_intent", "") or "") == "requirement_plan"
assert getattr(self.prepared_session, "current_project_id", None) == 99
return {
"handled": True,
"status": "answered",
@ -1575,11 +1575,11 @@ async def test_process_ingress_terminates_stale_requirement_scope_before_active_
async def _maybe_handle_draft_confirmation_ingress(self, **kwargs):
_ = kwargs
raise AssertionError("stale scope cleanup should happen before draft confirmation routing")
raise AssertionError("active context followup should short-circuit before draft confirmation routing")
async def _prepare_mixed_sidecar_context(self, **kwargs):
_ = kwargs
raise AssertionError("stale scope cleanup should short-circuit before query sidecar")
raise AssertionError("active context followup should short-circuit before query sidecar")
async def _fake_terminate_requirement_plan_scope(self, **kwargs):
cleanup_calls.append(kwargs)
@ -1603,9 +1603,7 @@ async def test_process_ingress_terminates_stale_requirement_scope_before_active_
ingress_options={"thread_context": {"reply_to_message_id": ""}},
)
assert cleanup_calls
assert cleanup_calls[0]["source_message_id"] == "s-msg-terminate-1"
assert cleanup_calls[0]["source_system"] == "feishu"
assert cleanup_calls == []
assert kernel.active_context_called is True
assert result["direct_reply_text"] == "普通问答结果"
@ -2538,6 +2536,8 @@ async def test_active_requirement_followup_prefers_current_requirement_context(m
assert result is not None
assert result["handled"] is True
assert "信号输入" in result["direct_reply_text"]
assert result["reply_template_key"] == "requirement_plan_followup"
assert result["work_item_id"] == 81
assert captured["packet_kwargs"]["latest_user_message"] == "信号源呢？"
assert captured["packet_kwargs"]["ingress_context"]["candidate_kind"] == "active_context_followup"
assert captured["direct_kwargs"]["message_text"] == "信号源呢？"
@ -2619,6 +2619,71 @@ async def test_active_requirement_choice_protocol_followup_bypasses_semantic_dec
assert captured["direct_kwargs"]["message_text"] == "我选C 继续"
@pytest.mark.asyncio
async def test_active_requirement_exit_request_suspends_bound_requirement_context_before_semantic_decision(
monkeypatch,
) -> None:
flush_mock = AsyncMock()
topic = SimpleNamespace(
id=181,
status="active",
plan_status="collecting",
pending_actions=[],
metadata_json={"plan_mode": True},
)
draft = SimpleNamespace(id=182, state="collecting")
async def _executor_find_active_requirement_plan_workspace(**kwargs):
_ = kwargs
return topic, draft
class FakeKernel(mixin_entry.KernelIngressFlowEntryMixin):
def __init__(self) -> None:
self.db = SimpleNamespace(flush=flush_mock)
self.tenant_id = 1
self.executor = SimpleNamespace(
_find_active_requirement_plan_workspace=_executor_find_active_requirement_plan_workspace,
)
class BombSemanticContextAssembler:
def __init__(self, db) -> None:
self.db = db
async def assemble(self, **kwargs):
_ = kwargs
raise AssertionError("explicit exit should bypass semantic decision assembly")
class BombSemanticDecisionService:
def __init__(self, _llm_service) -> None:
pass
async def decide(self, *, packet):
_ = packet
raise AssertionError("explicit exit should bypass semantic decision")
monkeypatch.setattr(mixin_entry, "SemanticContextAssembler", BombSemanticContextAssembler)
monkeypatch.setattr(mixin_entry, "SemanticDecisionService", BombSemanticDecisionService)
kernel = FakeKernel()
result = await kernel._maybe_handle_active_context_followup(
source_system="feishu",
source_message_id="msg-plan-exit-1",
channel_ref="chat-active",
source_sender_open_id="ou_active",
message_text="退出需求模式",
thread_context=None,
sender="tester",
)
assert result is not None
assert result["handled"] is True
assert "先暂停" in result["direct_reply_text"]
assert result["reply_template_key"] == "requirement_mode_suspended"
assert result["workflow_route"] == mixin_entry.ROUTE_REQUIREMENT_PLAN_COLLECT
assert result["topic_workspace_id"] == 181
flush_mock.assert_awaited_once()
@pytest.mark.asyncio
async def test_active_requirement_followup_uses_builder_result_to_persist_requirement_context(monkeypatch) -> None:
captured: dict[str, object] = {}
@ -3088,6 +3153,7 @@ async def test_find_active_requirement_plan_workspace_includes_draft_pending_con
assert found_topic is not None
assert found_draft is not None
assert int(found_topic.id) == int(topic.id)
assert str(found_topic.plan_status) == "draft_pending_confirm"
assert str(found_draft.state) == "draft_pending_confirm"
@ -3151,8 +3217,76 @@ async def test_find_active_requirement_plan_workspace_recovers_plan_topic_marked
assert found_topic is not None
assert found_draft is not None
assert int(found_topic.id) == int(topic.id)
assert str(found_topic.plan_status) == "options_ready"
assert str(found_draft.state) == "collecting"
@pytest.mark.asyncio
async def test_find_active_requirement_plan_workspace_prefers_live_draft_topic_over_newer_idle_scope_match(
requirement_plan_db_session,
) -> None:
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = requirement_plan_db_session
self.tenant_id = 7
executor = FakeExecutor()
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id="chat-plan-prefer-live-draft",
source_sender_open_id="ou-plan-prefer-live-draft",
sender=None,
)
live_topic = MountedTopicWorkspace(
tenant_id=7,
topic_key="feishu:chat-plan-prefer-live-draft:message:msg-plan-root",
source_system="feishu",
channel_ref="chat-plan-prefer-live-draft",
conversation_scope_key=conversation_scope_key,
mode="plan",
status="active",
plan_status="collecting",
metadata_json={"plan_mode": True},
)
requirement_plan_db_session.add(live_topic)
await requirement_plan_db_session.flush()
live_draft = MountedDraftWorkspace(
tenant_id=7,
topic_workspace_id=int(live_topic.id),
state="collecting",
draft_version=0,
requirement_card={"title": "采购库存报表助手"},
draft_payload=None,
required_confirmations=[],
)
requirement_plan_db_session.add(live_draft)
await requirement_plan_db_session.flush()
idle_query_topic = MountedTopicWorkspace(
tenant_id=7,
topic_key="feishu:chat-plan-prefer-live-draft:message:msg-later-query",
source_system="feishu",
channel_ref="chat-plan-prefer-live-draft",
conversation_scope_key=conversation_scope_key,
mode="chat",
status="active",
plan_status="idle",
metadata_json={"plan_mode": True},
)
requirement_plan_db_session.add(idle_query_topic)
await requirement_plan_db_session.flush()
found_topic, found_draft = await executor._find_active_requirement_plan_workspace(
source_system="feishu",
source_message_id="msg-followup-exit",
source_chat_id="chat-plan-prefer-live-draft",
sender=None,
source_sender_open_id="ou-plan-prefer-live-draft",
thread_context=None,
conversation_key=conversation_scope_key,
)
assert found_topic is not None
assert found_draft is not None
assert int(found_topic.id) == int(live_topic.id)
assert int(found_draft.id) == int(live_draft.id)
def test_world_model_session_project_inheritance_keeps_meaningful_followup() -> None:
@ -3382,6 +3516,19 @@ def test_mounted_runtime_query_requirement_consultation_structured_analysis_uses
assert mode == "plan"
def test_mounted_runtime_query_consult_requirement_requirements_analysis_uses_plan_mode() -> None:
mode = mixin_entry.KernelIngressFlowEntryMixin._resolve_mounted_runtime_mode_override(
workflow_route="query.answer",
route_metadata={
"message_intent": "consult_requirement",
"consultant_mode_preference": "requirements_analysis",
},
reply_template_key=None,
)
assert mode == "plan"
def test_mounted_runtime_query_requirement_gathering_stepwise_converge_uses_plan_mode() -> None:
mode = mixin_entry.KernelIngressFlowEntryMixin._resolve_mounted_runtime_mode_override(
workflow_route="query.answer",
@ -3972,11 +4119,11 @@ async def test_kernel_intent_resolve_degrades_when_semantic_planner_fails(
assert source == "semantic_planner_degraded"
assert intent.name == "query.read_only"
assert intent.metadata["workflow_route"] == "query.answer"
assert intent.metadata["message_intent"] == "consult_requirement"
assert intent.metadata["consultant_mode_preference"] == "clarify"
assert intent.metadata["context_source"] == "semantic_planner_degraded"
assert intent.metadata["semantic_planner_degraded"] is True
assert "upstream_empty_or_invalid_response" in str(intent.metadata["semantic_planner_error"])
assert "message_intent" not in intent.metadata
assert "consultant_mode_preference" not in intent.metadata
assert intent.signal_title == ""
@ -4259,6 +4406,39 @@ def test_reply_renderer_formats_project_delete_pending_and_success() -> None:
assert "已删除项目#12《重复项目》" in success["text"]
def test_reply_renderer_preserves_requirement_plan_followup_template_for_query_intent() -> None:
renderer = ReplyRenderer(
state_waiting_manual="waiting_manual",
state_failed="failed",
receipt_phase_action="action",
receipt_phase_pending="pending",
receipt_phase_error="error",
)
intent = CanonicalIntent(
name="query.read_only",
confidence=0.97,
project_id=None,
signal_title="",
signal_summary="",
metadata={},
)
rendered = renderer.render(
work_state="succeeded",
intent=intent,
decision_record={"approval_requirement": {}},
execution_result={
"status": "succeeded",
"workflow_route": "requirement_plan.collect",
"answer": "A. 轻量工具\nB. 协同平台\nC. 能力底座\n\n⭐ 当前我更推荐：A",
"reply_template_key": "requirement_plan_followup",
},
)
assert rendered["template_key"] == "requirement_plan_followup"
assert rendered["text"].startswith("A. 轻量工具")
@pytest.mark.asyncio
async def test_feishu_outbox_service_enqueue_preserves_requirement_plan_text_above_2000_chars(
feishu_outbox_db_session,
@ -4823,6 +5003,83 @@ async def test_execute_query_intent_short_circuits_casual_chat_without_rag(monke
assert str(result["rag_result"]["query_type"]) == "chat"
@pytest.mark.asyncio
async def test_execute_query_intent_external_source_policy_bypasses_casual_chat_shortcut(monkeypatch) -> None:
captured: dict[str, object] = {}
class FakeAdaptiveRAGService:
def __init__(self, db, tenant_id: int):
_ = (db, tenant_id)
async def preview_query_route_decision(self, **kwargs):
captured["route_preview_kwargs"] = dict(kwargs)
return {
"query_semantics": {
"source_policy": "uncertain",
"effective_source_policy": "external",
"query_type": "status",
"retrieval_domains": ["url"],
"query_profile": {
"needs_private_context": False,
"needs_freshness": True,
"referenced_objects": ["weather"],
},
},
"route_decision": {
"route_mode": "external",
"effective_source_policy": "external",
"decision_origin": "external_probe",
"reply_contract": {"mode": "explicit_source", "head": "外部实时查询：", "tail_hint": None},
"probe_snapshot_summary": {
"internal": {"config_hit_count": 0},
"external": {"trusted_domain_hit_count": 1, "answerable_page_hit_count": 1},
},
},
}
async def answer_question(self, **kwargs):
captured["answer_kwargs"] = dict(kwargs)
return {
"answer": "广州荔湾区今天多云，午后有阵雨。",
"sources": [{"source": "url", "label": "Weather", "score": 0.93, "metadata": {"url": "https://weather.example"}}],
"query_type": "status",
}
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = object()
self.tenant_id = 3
monkeypatch.setattr(mixin_execute_query_project, "AdaptiveRAGService", FakeAdaptiveRAGService)
executor = FakeExecutor()
intent = SimpleNamespace(
project_id=None,
metadata={
"semantic_bundle": {
"diagnostics": {
"message_intent": "casual_chat",
"consultant_mode_preference": "brainstorm",
}
}
},
)
result = await executor._execute_query_intent(
intent=intent,
message_text="今天荔湾区天气如何？",
operator_user_id=999,
source_system="feishu",
source_message_id="msg-chat-weather-1",
)
assert result["status"] == "succeeded"
assert "广州荔湾区今天多云" in str(result["answer"] or "")
answer_kwargs = captured["answer_kwargs"]
assert isinstance(answer_kwargs, dict)
assert answer_kwargs["source_policy"] == "uncertain"
assert answer_kwargs["route_decision"]["effective_source_policy"] == "external"
@pytest.mark.asyncio
async def test_execute_query_intent_backfills_semantic_bundle_for_casual_chat_without_legacy_helpers(
monkeypatch,
@ -6799,6 +7056,101 @@ async def test_execute_query_intent_locked_requirement_mode_redirects_casual_cha
assert str(result["rag_result"]["query_type"]) == "plan"
@pytest.mark.asyncio
async def test_execute_query_intent_locked_requirement_mode_redirects_generic_query_to_requirement_notice(
monkeypatch,
requirement_plan_db_session,
) -> None:
class FakeAdaptiveRAGService:
def __init__(self, db, tenant_id: int):
_ = (db, tenant_id)
async def answer_question(self, **kwargs):
raise AssertionError(f"locked requirement mode should not fall through to generic RAG: {kwargs}")
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = requirement_plan_db_session
self.tenant_id = 3
_install_requirement_plan_query_bundle(
monkeypatch,
message_intent="query",
consultant_mode_preference="",
)
monkeypatch.setattr(mixin_execute_query_project, "AdaptiveRAGService", FakeAdaptiveRAGService)
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id="chat-lock-plan-query",
source_sender_open_id="ou-lock-plan-query",
sender=None,
)
topic = MountedTopicWorkspace(
tenant_id=3,
topic_key="feishu:chat-lock-plan-query:root:om-lock-plan-query-root",
source_system="feishu",
channel_ref="chat-lock-plan-query",
conversation_scope_key=conversation_scope_key,
mode="plan",
status="active",
plan_status="collecting",
metadata_json={"plan_mode": True},
root_message_id="om-lock-plan-query-root",
)
requirement_plan_db_session.add(topic)
await requirement_plan_db_session.flush()
requirement_plan_db_session.add(
MountedDraftWorkspace(
tenant_id=3,
topic_workspace_id=int(topic.id),
state="collecting",
draft_version=0,
requirement_card={
"plan_profile": "it_project",
"title": "采购库存报表助手",
"goal": "先把采购库存报表助手的首版需求收清楚。",
"target_user": "采购经理",
"question_round": 2,
"last_round_options": {
"field": "target_user",
"options": [
{"key": "A", "title": "只服务采购经理本人"},
{"key": "B", "title": "采购经理+仓管员"},
{"key": "C", "title": "采购经理+财务复核人"},
],
"recommendation": {"key": "A"},
},
},
required_confirmations=[],
)
)
await requirement_plan_db_session.flush()
executor = FakeExecutor()
intent = SimpleNamespace(
project_id=None,
metadata={
"source_chat_id": "chat-lock-plan-query",
"source_sender_open_id": "ou-lock-plan-query",
"message_intent": "query",
},
)
result = await executor._execute_query_intent(
intent=intent,
message_text="系统里面有哪些项目？",
operator_user_id=999,
source_system="feishu",
source_message_id="msg-lock-plan-query-1",
)
assert result["status"] == "succeeded"
assert "当前正在需求模式" in str(result["answer"] or "")
assert "退出需求模式" in str(result["answer"] or "")
assert str(result["workflow_route"]) == "requirement_plan.collect"
assert str(result["rag_result"]["query_type"]) == "plan"
@pytest.mark.asyncio
async def test_execute_query_intent_converts_query_answer_requirement_contract_into_requirement_plan_result(
monkeypatch,
@ -7419,6 +7771,270 @@ async def test_execute_query_intent_treats_requirement_consultation_structured_a
assert "Feature Catalog" not in str(result["answer"] or "")
@pytest.mark.asyncio
async def test_execute_query_intent_treats_consult_requirement_requirements_analysis_as_requirement_plan(
monkeypatch,
requirement_plan_db_session,
) -> None:
captured: dict[str, object] = {}
class FakeAdaptiveRAGService:
def __init__(self, db, tenant_id: int):
_ = (db, tenant_id)
async def answer_question(self, **kwargs):
raise AssertionError(f"requirements analysis should not enter heavy RAG: {kwargs}")
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = requirement_plan_db_session
self.tenant_id = 3
async def _fake_build_requirement_planning_direct_result(self, **kwargs): # noqa: ANN001
_ = self
captured["direct_kwargs"] = dict(kwargs)
return {
"status": "succeeded",
"answer": "可以，我们先不创建，先把这件事收成一版需求。",
"rag_result": {
"answer": "可以，我们先不创建，先把这件事收成一版需求。",
"query_type": "plan",
},
"reply_template_key": "requirement_plan_followup",
}
monkeypatch.setattr(
mixin_execute_query_project.ExecuteQueryProjectMixin,
"_build_requirement_planning_direct_result",
_fake_build_requirement_planning_direct_result,
raising=False,
)
monkeypatch.setattr(mixin_execute_query_project, "AdaptiveRAGService", FakeAdaptiveRAGService)
executor = FakeExecutor()
intent = SimpleNamespace(
project_id=None,
metadata={
"source_chat_id": "chat-consult-requirements-analysis",
"source_sender_open_id": "ou-consult-requirements-analysis",
"message_intent": "consult_requirement",
"consultant_mode_preference": "requirements_analysis",
"topic_switch": "new",
},
)
result = await executor._execute_query_intent(
intent=intent,
message_text="想跟你聊个需求，关于给采购经理提供可清理库存的报表。",
operator_user_id=999,
source_system="feishu",
source_message_id="msg-consult-requirements-analysis-1",
)
assert result["status"] == "succeeded"
assert str(result["rag_result"]["query_type"]) == "plan"
direct_kwargs = captured.get("direct_kwargs")
assert isinstance(direct_kwargs, dict)
assert str(direct_kwargs["message_text"]) == "想跟你聊个需求，关于给采购经理提供可清理库存的报表。"
assert str(direct_kwargs["source_chat_id"]) == "chat-consult-requirements-analysis"
assert "Feature Catalog" not in str(result["answer"] or "")
@pytest.mark.asyncio
async def test_execute_query_intent_semantic_planner_degraded_factual_query_falls_back_to_regular_qa(
monkeypatch,
requirement_plan_db_session,
) -> None:
captured: dict[str, object] = {}
class FakeAdaptiveRAGService:
def __init__(self, db, tenant_id: int):
_ = (db, tenant_id)
async def answer_question(self, **kwargs):
captured["rag_kwargs"] = dict(kwargs)
return {
"answer": "根据公开页面，目前走普通事实问答链。",
"query_type": "explain",
"sources": [{"source": "web", "label": "External", "score": 0.9}],
}
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = requirement_plan_db_session
self.tenant_id = 3
async def _failing_aplan_message(*, message_text: str, source_system: str, metadata=None):
_ = (message_text, source_system, metadata)
raise RuntimeError("upstream_empty_or_invalid_response")
async def _unexpected_requirement_direct_result(self, **kwargs): # noqa: ANN001
raise AssertionError(f"degraded factual query should not enter requirement plan direct path: {kwargs}")
monkeypatch.setattr(
WorkflowSemanticPlanner,
"aplan_message",
classmethod(lambda cls, **kwargs: _failing_aplan_message(**kwargs)),
)
monkeypatch.setattr(
mixin_execute_query_project.ExecuteQueryProjectMixin,
"_build_requirement_planning_direct_result",
_unexpected_requirement_direct_result,
raising=False,
)
monkeypatch.setattr(mixin_execute_query_project, "AdaptiveRAGService", FakeAdaptiveRAGService)
executor = FakeExecutor()
intent = SimpleNamespace(
project_id=None,
metadata={
"source_chat_id": "chat-degraded-factual",
"source_sender_open_id": "ou-degraded-factual",
"semantic_planner_degraded": True,
},
)
result = await executor._execute_query_intent(
intent=intent,
message_text="当前claude最新的模型用的是什么？",
operator_user_id=999,
source_system="feishu",
source_message_id="msg-degraded-factual-1",
)
assert result["status"] == "succeeded"
assert str(result["rag_result"]["query_type"]) == "explain"
assert "收成一版需求" not in str(result["answer"] or "")
assert "requirement_plan_context" not in result
assert isinstance(captured.get("rag_kwargs"), dict)
@pytest.mark.asyncio
async def test_execute_query_intent_semantic_planner_degraded_active_requirement_scope_stays_in_plan(
monkeypatch,
requirement_plan_db_session,
) -> None:
captured: dict[str, object] = {}
class FakeAdaptiveRAGService:
def __init__(self, db, tenant_id: int):
_ = (db, tenant_id)
async def answer_question(self, **kwargs):
raise AssertionError(f"active requirement scope should not enter heavy RAG: {kwargs}")
class FakeExecutor(mixin_execute_query_project.ExecuteQueryProjectMixin):
def __init__(self) -> None:
self.db = requirement_plan_db_session
self.tenant_id = 3
async def _failing_aplan_message(*, message_text: str, source_system: str, metadata=None):
_ = (message_text, source_system, metadata)
raise RuntimeError("upstream_empty_or_invalid_response")
async def _fake_build_requirement_planning_direct_result(self, **kwargs): # noqa: ANN001
_ = self
captured["direct_kwargs"] = dict(kwargs)
return {
"status": "succeeded",
"answer": "第二轮：继续收敛来源范围。",
"rag_result": {
"answer": "第二轮：继续收敛来源范围。",
"query_type": "plan",
},
"reply_template_key": "requirement_plan_followup",
}
monkeypatch.setattr(
WorkflowSemanticPlanner,
"aplan_message",
classmethod(lambda cls, **kwargs: _failing_aplan_message(**kwargs)),
)
monkeypatch.setattr(
mixin_execute_query_project.ExecuteQueryProjectMixin,
"_build_requirement_planning_direct_result",
_fake_build_requirement_planning_direct_result,
raising=False,
)
monkeypatch.setattr(mixin_execute_query_project, "AdaptiveRAGService", FakeAdaptiveRAGService)
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id="chat-degraded-plan",
source_sender_open_id="ou-degraded-plan",
sender=None,
)
topic = MountedTopicWorkspace(
tenant_id=3,
topic_key="feishu:chat-degraded-plan:root:om-degraded-plan-root",
source_system="feishu",
channel_ref="chat-degraded-plan",
conversation_scope_key=conversation_scope_key,
mode="plan",
status="active",
plan_status="collecting",
metadata_json={"plan_mode": True},
facts_json={"title": "Claude 模型信息卡片"},
)
requirement_plan_db_session.add(topic)
await requirement_plan_db_session.flush()
requirement_plan_db_session.add(
MountedDraftWorkspace(
tenant_id=3,
topic_workspace_id=int(topic.id),
state="collecting",
draft_version=0,
requirement_card={
"plan_profile": "it_project",
"title": "Claude 模型信息卡片",
"goal": "先把模型信息卡片需求收清楚。",
"project_nature": "轻量即用型工具",
"question_round": 1,
"last_round_options": {
"field": "project_nature",
"options": [
{"key": "A", "title": "轻量即用型工具"},
{"key": "B", "title": "业务协同平台"},
{"key": "C", "title": "AI能力底座模块"},
],
"recommendation": {"key": "A"},
},
"round_option_maps": {
"project_nature": {
"A": "轻量即用型工具",
"B": "业务协同平台",
"C": "AI能力底座模块",
}
},
},
required_confirmations=[],
)
)
await requirement_plan_db_session.flush()
executor = FakeExecutor()
intent = SimpleNamespace(
project_id=None,
metadata={
"source_chat_id": "chat-degraded-plan",
"source_sender_open_id": "ou-degraded-plan",
"semantic_planner_degraded": True,
},
)
result = await executor._execute_query_intent(
intent=intent,
message_text="A",
operator_user_id=999,
source_system="feishu",
source_message_id="msg-degraded-plan-1",
)
assert result["status"] == "succeeded"
assert str(result["workflow_route"]) == "requirement_plan.collect"
assert str(result["rag_result"]["query_type"]) == "plan"
assert isinstance(captured.get("direct_kwargs"), dict)
@pytest.mark.asyncio
async def test_execute_query_intent_promotes_requirement_followup_from_structured_round_context_without_render_markers(
monkeypatch,
@ -11068,6 +11684,7 @@ async def test_should_route_to_draft_confirmation_session_does_not_route_free_te
@pytest.mark.asyncio
async def test_external_freshness_followup_stays_in_query_answer_without_internal_target(
external_qa_db_session,
monkeypatch,
) -> None:
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id="chat-external-qa",
@ -11113,13 +11730,41 @@ async def test_external_freshness_followup_stays_in_query_answer_without_interna
self.db = external_qa_db_session
self.tenant_id = 1
async def _fake_aplan_message(cls, *, message_text: str, source_system: str, metadata=None):
_ = (cls, message_text, source_system, metadata)
return WorkflowSemanticPlanner.deserialize_bundle(
{
"bundle_id": "freshness-followup-query",
"semantic_version": "v1",
"source_system": "feishu",
"primary_workflow": "query_flow",
"workflows": [
{
"workflow_type": "query_flow",
"workflow_action": "answer",
"target_scope": "global",
"operation_codes": ["query.answer"],
"requires_confirmation": False,
"confidence": 0.91,
"clarification_needed": False,
"metadata": {},
}
],
"message_gate": "auto",
"reply_contract": {"mode": "text", "template_key": "workflow_bundle", "summary": "query_flow"},
"diagnostics": {"message_intent": "question"},
}
)
monkeypatch.setattr(WorkflowSemanticPlanner, "aplan_message", classmethod(_fake_aplan_message))
kernel = FakeKernel()
result = await kernel._maybe_handle_external_freshness_followup(
source_system="feishu",
source_message_id="om-external-followup-1",
channel_ref="chat-external-qa",
source_sender_open_id="ou-external-qa",
message_text="接入外部来源继续核实",
message_text="重新联网核实一下",
thread_context={"reply_to_message_id": "om-external-question-1"},
)
@ -11134,6 +11779,7 @@ async def test_external_freshness_followup_stays_in_query_answer_without_interna
@pytest.mark.asyncio
async def test_external_freshness_followup_allows_explicit_internal_source_request(
external_qa_db_session,
monkeypatch,
) -> None:
conversation_scope_key = build_feishu_conversation_scope_key(
source_chat_id="chat-external-qa",
@ -11179,6 +11825,34 @@ async def test_external_freshness_followup_allows_explicit_internal_source_reque
self.db = external_qa_db_session
self.tenant_id = 1
async def _fake_aplan_message(cls, *, message_text: str, source_system: str, metadata=None):
_ = (cls, message_text, source_system, metadata)
return WorkflowSemanticPlanner.deserialize_bundle(
{
"bundle_id": "freshness-followup-signal-source",
"semantic_version": "v1",
"source_system": "feishu",
"primary_workflow": "signal_source_flow",
"workflows": [
{
"workflow_type": "signal_source_flow",
"workflow_action": "onboard",
"target_scope": "global",
"operation_codes": ["signal_source.onboard"],
"requires_confirmation": False,
"confidence": 0.95,
"clarification_needed": False,
"metadata": {},
}
],
"message_gate": "auto",
"reply_contract": {"mode": "text", "template_key": "workflow_bundle", "summary": "signal_source_flow"},
"diagnostics": {"message_intent": "question"},
}
)
monkeypatch.setattr(WorkflowSemanticPlanner, "aplan_message", classmethod(_fake_aplan_message))
kernel = FakeKernel()
result = await kernel._maybe_handle_external_freshness_followup(
source_system="feishu",
@ -11301,13 +11975,40 @@ async def test_process_ingress_short_circuits_external_freshness_followup_before
async def _prepare_mixed_sidecar_context(self, **kwargs):
raise AssertionError("external freshness followup should not enter mixed sidecar")
async def _fake_aplan_message(cls, *, message_text: str, source_system: str, metadata=None):
_ = (cls, message_text, source_system, metadata)
return WorkflowSemanticPlanner.deserialize_bundle(
{
"bundle_id": "freshness-followup-query-process-ingress",
"semantic_version": "v1",
"source_system": "feishu",
"primary_workflow": "query_flow",
"workflows": [
{
"workflow_type": "query_flow",
"workflow_action": "answer",
"target_scope": "global",
"operation_codes": ["query.answer"],
"requires_confirmation": False,
"confidence": 0.93,
"clarification_needed": False,
"metadata": {},
}
],
"message_gate": "auto",
"reply_contract": {"mode": "text", "template_key": "workflow_bundle", "summary": "query_flow"},
"diagnostics": {"message_intent": "question"},
}
)
monkeypatch.setattr(mixin_entry, "MountedAIRuntimeWriteService", FakeMountedAIRuntimeWriteService)
monkeypatch.setattr(WorkflowSemanticPlanner, "aplan_message", classmethod(_fake_aplan_message))
kernel = FakeKernel()
result = await kernel.process_ingress(
source_system="feishu",
source_message_id="om-external-followup-3",
message_text="接入外部来源继续核实",
message_text="继续联网查最新结果",
sender="林宇航",
channel_ref="chat-external-qa",
source_sender_open_id="ou-external-qa",

186
backend/tests/test_ai_service_core_runtime.py
View File

@ -636,6 +636,192 @@ async def test_openai_compatible_chat_accepts_structured_json_from_reasoning_con
assert json.loads(result.text)["title"] == "AI直播复盘助手"
@pytest.mark.asyncio
async def test_openai_compatible_chat_ignores_invalid_structured_candidate_from_reasoning_content_and_retries(
monkeypatch,
) -> None:
calls: list[dict[str, object]] = []
class FakeResponse:
def __init__(self, payload: dict[str, object]) -> None:
self.status_code = 200
self._payload = payload
def json(self):
return self._payload
valid_payload = {
"topic_relation": "new_topic",
"mode_decision": "answer_only",
"followup_scope": "none",
"followup_action": "none",
"revision_note": "",
"draft_patch": {},
"workflow_plans": [
{
"workflow_type": "query_flow",
"workflow_action": "answer",
"target_scope": "global",
"operation_codes": ["query.answer"],
"requires_confirmation": False,
"confidence": 0.91,
"clarification_needed": False,
"metadata": {},
}
],
"scene_metadata": {},
"assistant_reply": "我会查询 Microsoft Learn 并返回最新官方链接。",
"requires_confirmation": False,
"requested_actions": ["workflow.bundle.compose"],
"confidence": 0.91,
"uncertainties": [],
}
class FakeClient:
def __init__(self, *args, **kwargs):
_ = (args, kwargs)
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, tb):
return False
async def post(self, *args, **kwargs):
calls.append(dict(kwargs.get("json") or {}))
if len(calls) == 1:
return FakeResponse(
{
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"reasoning_content": (
"1. 分析输入\n"
"packet.thread.recent_turns: []\n"
"packet.ingress.allowed_context_scopes: []\n"
"接下来应进入 query.answer。"
),
},
"finish_reason": "length",
}
]
}
)
return FakeResponse(
{
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": json.dumps(valid_payload, ensure_ascii=False),
"reasoning_content": "",
},
"finish_reason": "stop",
}
]
}
)
async def fake_sleep(_seconds: float):
return None
monkeypatch.setattr(mixin_chat_openai.httpx, "AsyncClient", FakeClient)
monkeypatch.setattr(mixin_chat_openai.asyncio, "sleep", fake_sleep)
result = await llm_service._chat_completion_via_openai_compatible(
key="test-key",
base_url="https://api.edgefn.net/v1",
model="GLM-5",
temperature=0.0,
max_tokens=512,
messages=[{"role": "user", "content": "帮我判断这句消息应该走哪个 workflow"}],
request_timeout=1.0,
structured_output=True,
function_label="pmflow_semantic_context_decision",
provider_name="openai",
is_control_plane_gateway_request=False,
control_plane_agents_timeout_seconds=120,
request_started_at_perf=time.perf_counter(),
request_trace_id="trace-invalid-reasoning-structured",
control_plane_agent_timeout=None,
routing_meta={"response_contract": "structured"},
)
assert len(calls) == 2
assert json.loads(result.text)["topic_relation"] == "new_topic"
assert "最终 JSON 对象" in str(calls[1]["messages"][-1]["content"])
@pytest.mark.asyncio
async def test_chat_completion_raw_opens_provider_circuit_immediately_on_billing_block(monkeypatch) -> None:
circuit_key = "glm|https://dashscope.aliyuncs.com/compatible-mode/v1|qwen-plus"
original_state = dict(llm_service._provider_circuit_state)
original_signature = llm_service._provider_circuit_config_signature
class FakeResponse:
status_code = 400
headers: dict[str, str] = {}
text = json.dumps(
{
"error": {
"message": "Access denied, please make sure your account is in good standing.",
"type": "Arrearage",
"code": "Arrearage",
}
},
ensure_ascii=False,
)
def json(self):
return json.loads(self.text)
def raise_for_status(self):
request = httpx.Request("POST", "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions")
response = httpx.Response(self.status_code, request=request, text=self.text)
raise httpx.HTTPStatusError("Client error '400 Bad Request'", request=request, response=response)
class FakeClient:
def __init__(self, *args, **kwargs):
_ = (args, kwargs)
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, tb):
return False
async def post(self, *args, **kwargs):
_ = (args, kwargs)
return FakeResponse()
monkeypatch.setattr(mixin_chat_openai.httpx, "AsyncClient", FakeClient)
try:
with pytest.raises(ValueError, match="provider_billing_blocked:glm"):
await llm_service._chat_completion_raw(
provider="glm",
prompt="输出 JSON",
system_prompt="你是 JSON 生成器",
api_key="glm-key",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
model="qwen-plus",
temperature=0.0,
max_tokens=64,
function_name="pmflow_semantic_context_decision",
response_mode="fast",
routing_meta={"response_contract": "structured"},
)
assert llm_service._is_provider_circuit_open(circuit_key) is True
finally:
llm_service._provider_circuit_state.clear()
llm_service._provider_circuit_state.update(original_state)
llm_service._provider_circuit_config_signature = original_signature
@pytest.mark.asyncio
async def test_openai_compatible_chat_retries_empty_reasoning_response_with_bigger_token_budget(monkeypatch) -> None:
calls: list[dict[str, object]] = []

106
backend/tests/test_draft_confirm_handler.py
View File

@ -1568,6 +1568,112 @@ async def test_resolve_active_mounted_draft_keeps_retryable_pending_confirm_with
assert result == (draft_row, topic_row)
@pytest.mark.asyncio
async def test_resolve_active_mounted_draft_does_not_fall_back_to_same_scope_row_when_explicit_anchor_misses(
) -> None:
class _FakeResult:
def __init__(self, rows) -> None:
self._rows = list(rows)
def all(self):
return list(self._rows)
draft_row = SimpleNamespace(
id=18,
topic_workspace_id=177,
active_work_item_id=None,
state=STATE_DRAFT_PENDING_CONFIRM,
latest_reply_message_id="om-stale-draft-reply-1",
latest_source_message_id="src-stale-draft-1",
)
topic_row = SimpleNamespace(
id=177,
tenant_id=1,
source_system="feishu",
mode="plan",
status="active",
channel_ref="chat-1",
conversation_scope_key="feishu:chat-1:ou_xxx",
plan_status=STATE_DRAFT_PENDING_CONFIRM,
root_message_id="msg-stale-draft-root",
thread_key="",
last_message_id="src-stale-draft-1",
updated_at=None,
)
class _FakeDB:
async def execute(self, query): # noqa: ARG002
return _FakeResult([(draft_row, topic_row)])
handler = DraftConfirmHandler(type("Kernel", (), {"db": _FakeDB(), "tenant_id": 1})())
result = await handler._resolve_active_mounted_draft(
sender="林宇航",
source_system="feishu",
channel_ref="chat-1",
source_sender_open_id="ou_xxx",
thread_context={
"reply_to_message_id": "om-current-plan-reply-1",
"root_message_id": "msg-current-plan-root",
},
)
assert result is None
@pytest.mark.asyncio
async def test_has_active_draft_session_does_not_lock_same_scope_when_explicit_anchor_targets_other_thread(
) -> None:
class _FakeResult:
def __init__(self, rows) -> None:
self._rows = list(rows)
def all(self):
return list(self._rows)
stale_draft = SimpleNamespace(
id=28,
topic_workspace_id=277,
active_work_item_id=None,
state=STATE_DRAFT_PENDING_CONFIRM,
latest_reply_message_id="om-stale-draft-reply-2",
latest_source_message_id="src-stale-draft-2",
)
stale_topic = SimpleNamespace(
id=277,
tenant_id=1,
source_system="feishu",
mode="plan",
status="active",
channel_ref="chat-1",
conversation_scope_key="feishu:chat-1:ou_xxx",
plan_status=STATE_DRAFT_PENDING_CONFIRM,
root_message_id="msg-stale-draft-root-2",
thread_key="",
last_message_id="src-stale-draft-2",
updated_at=None,
)
class _FakeDB:
async def execute(self, query): # noqa: ARG002
return _FakeResult([(stale_draft, stale_topic)])
handler = DraftConfirmHandler(type("Kernel", (), {"db": _FakeDB(), "tenant_id": 1})())
has_session = await handler.has_active_draft_session(
sender="林宇航",
source_system="feishu",
channel_ref="chat-1",
source_sender_open_id="ou_xxx",
thread_context={
"reply_to_message_id": "om-current-plan-reply-2",
"root_message_id": "msg-current-plan-root-2",
},
)
assert has_session is False
@pytest.mark.asyncio
async def test_resolve_pending_draft_falls_back_to_anchor_matched_pending_work_item_when_active_mounted_draft_has_no_active_work_item_id(
monkeypatch,

51
backend/tests/test_external_query_planner.py Normal file
View File

@ -0,0 +1,51 @@
from app.services.external_query_service.planner import ExternalQueryPlanner
def test_fallback_plan_stays_generic_for_fresh_weather_semantics() -> None:
planner = ExternalQueryPlanner(
llm_service=object(),
db=object(),
tenant_id=1,
)
plan = planner._build_fallback_plan(
question="帮我看看未来5天的广州天气如何",
query_semantics={
"query_type": "status",
"object_mentions": ["广州"],
"query_profile": {
"needs_freshness": True,
"referenced_objects": ["weather"],
},
},
)
assert plan.query_kind == "generic_web"
assert plan.answer_contract == "summary_with_sources"
assert plan.location_hint is None
assert plan.result_limit == 3
def test_fallback_plan_uses_compare_contract_without_object_kind_mapping() -> None:
planner = ExternalQueryPlanner(
llm_service=object(),
db=object(),
tenant_id=1,
)
plan = planner._build_fallback_plan(
question="对比 OpenAI 和 Anthropic 最新模型能力差异",
query_semantics={
"query_type": "compare",
"object_mentions": ["OpenAI", "Anthropic"],
"query_profile": {
"needs_freshness": True,
"referenced_objects": ["public_model"],
},
},
)
assert plan.query_kind == "generic_web"
assert plan.answer_contract == "compare_with_sources"
assert plan.requires_multi_source is True
assert plan.result_limit == 3

7
backend/tests/test_feishu_card_action_contract.py
View File

@ -277,7 +277,12 @@ async def test_handle_card_action_trigger_plan_choice_dispatches_synthetic_ingre
assert data.get("target_work_item_id") == 88
ingress_kwargs = called["ingress_kwargs"]
assert ingress_kwargs["message_text"] == "C"
assert ingress_kwargs["source_system"] == "ai_pmo_ui"
assert ingress_kwargs["source_system"] == "feishu"
assert str(ingress_kwargs["source_message_id"]).startswith("feishu-card-plan-88-plan_choice-")
ingress_options = ingress_kwargs.get("ingress_options") if isinstance(ingress_kwargs.get("ingress_options"), dict) else {}
thread_context = ingress_options.get("thread_context") if isinstance(ingress_options.get("thread_context"), dict) else {}
assert thread_context["reply_to_message_id"] == "om_plan_1"
assert thread_context["root_message_id"] == "om_plan_1"
assert called["reply_chat"] == "oc_plan_1"
assert isinstance(called["reply_card"], dict)
assert called["reply_card"]["header"]["title"]["content"] == "AI PMO 需求收敛"

269
backend/tests/test_mcp_config_api.py Normal file
View File

@ -0,0 +1,269 @@
from __future__ import annotations
from types import SimpleNamespace
import pytest
import pytest_asyncio
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
from sqlalchemy import JSON, select
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.api.mcp_api import routes as mcp_routes
from app.core.database import get_db
from app.dependencies.auth import get_current_user
from app.models.base import Base
from app.models.integration import Integration
from app.services.mcp_config import McpConfigService
def _use_sqlite_json_types(*tables) -> None:
for table in tables:
for column in table.columns:
if column.type.__class__.__name__ in {"JSONB", "ARRAY", "Vector"}:
column.type = JSON()
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
tables = [Integration.__table__]
_use_sqlite_json_types(*tables)
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: Base.metadata.create_all(sync_conn, tables=tables))
maker = async_sessionmaker(engine, expire_on_commit=False)
async with maker() as session:
yield session
await engine.dispose()
def _build_app(db_session, *, role: str, tenant_id: int = 1) -> FastAPI:
app = FastAPI()
app.include_router(mcp_routes.router)
async def _fake_get_db():
yield db_session
app.dependency_overrides[get_db] = _fake_get_db
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=7,
tenant_id=tenant_id,
role=role,
username="tester",
email="tester@example.com",
name="Tester",
)
return app
@pytest.mark.asyncio
async def test_mcp_admin_api_requires_manager_role(db_session) -> None:
app = _build_app(db_session, role="AI")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/mcp/admin/catalog")
assert response.status_code == 403
@pytest.mark.asyncio
async def test_mcp_admin_catalog_and_server_lifecycle(db_session, monkeypatch) -> None:
async def _fake_probe(self, *, template, params): # noqa: ANN001
_ = (self, template, params)
return {
"status": "ready",
"checked_at": "2026-04-15T12:01:00+08:00",
"message": "PMFlow backend can reach Microsoft Learn MCP.",
"tool_names": ["microsoft_docs_search", "microsoft_docs_fetch"],
}
monkeypatch.setattr(McpConfigService, "_probe_backend_runtime", _fake_probe)
app = _build_app(db_session, role="ADMIN")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
catalog_response = await client.get("/mcp/admin/catalog")
create_response = await client.post(
"/mcp/admin/servers",
json={
"name": "microsoft-learn",
"display_name": "Microsoft Learn MCP",
"template_id": "microsoft_learn",
"params": {},
},
)
created = create_response.json()["data"]
list_response = await client.get("/mcp/admin/servers")
validate_response = await client.post(f"/mcp/admin/servers/{created['id']}/validate")
update_response = await client.put(
f"/mcp/admin/servers/{created['id']}",
json={
"display_name": "Microsoft Learn Docs",
"params": {},
},
)
confirm_response = await client.post(f"/mcp/admin/servers/{created['id']}/confirm-client-apply")
assert catalog_response.status_code == 200
assert catalog_response.json()["code"] == 0
assert catalog_response.json()["data"]["recommended_template_id"] == "microsoft_learn"
assert catalog_response.json()["data"]["items"][0]["execution_mode"] == "backend_direct"
assert create_response.status_code == 200
assert create_response.json()["code"] == 0
assert created["execution_mode"] == "backend_direct"
assert created["client_apply_state"] == "not_required"
assert created["runtime_probe"]["status"] == "unknown"
assert list_response.status_code == 200
assert list_response.json()["data"]["items"][0]["name"] == "microsoft-learn"
assert validate_response.status_code == 200
assert validate_response.json()["code"] == 0
assert validate_response.json()["data"]["is_valid"] is True
assert validate_response.json()["data"]["preview"]["add_command"] == "无需本机命令。保存后由 PMFlow 后端直接连接 Microsoft Learn MCP。"
assert validate_response.json()["data"]["preview"]["verify_command"] == "点击“重新校验”后，PMFlow 会从后端执行 MCP initialize 和 tools/list。"
assert validate_response.json()["data"]["runtime_probe"]["status"] == "ready"
assert validate_response.json()["data"]["runtime_probe"]["tool_names"] == ["microsoft_docs_search", "microsoft_docs_fetch"]
assert update_response.status_code == 200
assert update_response.json()["code"] == 0
assert update_response.json()["data"]["display_name"] == "Microsoft Learn Docs"
assert confirm_response.status_code == 400
assert confirm_response.json()["code"] == 400
assert "后端直连" in confirm_response.json()["message"]
@pytest.mark.asyncio
async def test_mcp_admin_servers_are_tenant_scoped(db_session) -> None:
tenant1_app = _build_app(db_session, role="ADMIN", tenant_id=1)
tenant2_app = _build_app(db_session, role="ADMIN", tenant_id=2)
async with AsyncClient(transport=ASGITransport(app=tenant1_app), base_url="http://test") as client:
create_response = await client.post(
"/mcp/admin/servers",
json={
"name": "playwright",
"display_name": "Playwright MCP",
"template_id": "playwright",
"params": {"output_dir": "~/.codex/mcp/playwright-output"},
},
)
async with AsyncClient(transport=ASGITransport(app=tenant2_app), base_url="http://test") as client:
list_response = await client.get("/mcp/admin/servers")
assert create_response.status_code == 200
assert create_response.json()["code"] == 0
assert list_response.status_code == 200
assert list_response.json()["code"] == 0
assert list_response.json()["data"]["items"] == []
@pytest.mark.asyncio
async def test_mcp_admin_github_template_rejects_raw_token_and_only_persists_env_var_name(db_session) -> None:
app = _build_app(db_session, role="ADMIN")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
invalid_response = await client.post(
"/mcp/admin/servers",
json={
"name": "github",
"display_name": "GitHub MCP",
"template_id": "github",
"params": {
"token_env_var_name": "GITHUB_PAT",
"access_token": "ghp_secret_should_not_be_saved",
},
},
)
valid_response = await client.post(
"/mcp/admin/servers",
json={
"name": "github",
"display_name": "GitHub MCP",
"template_id": "github",
"params": {
"token_env_var_name": "GITHUB_PAT",
},
},
)
row = (
await db_session.execute(
select(Integration).where(
Integration.tenant_id == 1,
Integration.type == "mcp_template_center",
)
)
).scalar_one()
server = row.config["servers"][0]
assert invalid_response.status_code == 400
assert invalid_response.json()["code"] == 400
assert "token" in invalid_response.json()["message"].lower()
assert valid_response.status_code == 200
assert valid_response.json()["code"] == 0
assert valid_response.json()["data"]["execution_mode"] == "local_bridge"
assert server["params"]["token_env_var_name"] == "GITHUB_PAT"
assert "access_token" not in server["params"]
@pytest.mark.asyncio
async def test_mcp_admin_filesystem_requires_allowed_paths(db_session) -> None:
app = _build_app(db_session, role="ADMIN")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post(
"/mcp/admin/servers",
json={
"name": "filesystem",
"display_name": "Filesystem MCP",
"template_id": "filesystem",
"params": {},
},
)
assert response.status_code == 400
assert response.json()["code"] == 400
assert "allowed_paths" in response.json()["message"]
@pytest.mark.asyncio
async def test_local_bridge_template_uses_manual_instructions_and_can_be_confirmed(db_session) -> None:
app = _build_app(db_session, role="ADMIN")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
create_response = await client.post(
"/mcp/admin/servers",
json={
"name": "filesystem",
"display_name": "Filesystem MCP",
"template_id": "filesystem",
"params": {
"allowed_paths": ["/Users/yuuuu/Documents"],
},
},
)
created = create_response.json()["data"]
validate_response = await client.post(f"/mcp/admin/servers/{created['id']}/validate")
confirm_response = await client.post(f"/mcp/admin/servers/{created['id']}/confirm-client-apply")
assert create_response.status_code == 200
assert created["execution_mode"] == "local_bridge"
assert created["client_apply_state"] == "pending_local_apply"
assert "codex mcp add" not in created["preview"]["add_command"].lower()
assert "local bridge" in created["preview"]["add_command"].lower()
assert validate_response.status_code == 200
assert validate_response.json()["data"]["runtime_probe"]["status"] == "local_bridge_required"
assert confirm_response.status_code == 200
assert confirm_response.json()["data"]["client_apply_state"] == "user_confirmed"

448
backend/tests/test_mcp_external_query_bridge.py Normal file
View File

@ -0,0 +1,448 @@
from __future__ import annotations
from types import SimpleNamespace
import pytest
import pytest_asyncio
from sqlalchemy import JSON
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.models.base import Base
from app.models.integration import Integration
from app.services.adaptive_rag_service.mixin_answer_retrieval import AdaptiveRAGAnswerRetrievalMixin
from app.services.adaptive_rag_service.mixin_deep_guard_rules import AdaptiveRAGDeepGuardRulesMixin
from app.services.adaptive_rag_service.shared_part_02 import RetrievedItem
from app.services.external_query_service.adapters import MCPExternalAdapter
from app.services.external_query_service.gateway import ExternalQueryGateway
from app.services.external_query_service.registry import ExternalProviderRegistry
from app.services.external_query_service.schemas import ExternalAdapterResult, ExternalEvidenceItem, ExternalQueryPlan
from app.services.mcp_config import McpConfigService
from app.services.mcp_config.runtime import MicrosoftLearnMCPRuntime
def _use_sqlite_json_types(*tables) -> None:
for table in tables:
for column in table.columns:
if column.type.__class__.__name__ in {"JSONB", "ARRAY", "Vector"}:
column.type = JSON()
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
tables = [Integration.__table__]
_use_sqlite_json_types(*tables)
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: Base.metadata.create_all(sync_conn, tables=tables))
maker = async_sessionmaker(engine, expire_on_commit=False)
async with maker() as session:
yield session
await engine.dispose()
async def _seed_backend_direct_server(db_session) -> None:
service = McpConfigService(db_session, tenant_id=1)
await service.create_server(
name="microsoft-learn",
display_name="Microsoft Learn MCP",
template_id="microsoft_learn",
params={},
)
await db_session.commit()
@pytest.mark.asyncio
async def test_mcp_config_service_search_external_uses_backend_direct_server(db_session, monkeypatch) -> None:
await _seed_backend_direct_server(db_session)
async def _fake_search(self, *, query: str, result_limit: int = 3): # noqa: ANN001
_ = (self, query, result_limit)
return [
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Microsoft Learn MCP",
title="Azure OpenAI quickstart",
target_url="https://learn.microsoft.com/azure/ai-foundry/openai/chatgpt-quickstart",
snippet="Official Microsoft quickstart.",
retrieved_at="2026-04-15T12:30:00+08:00",
freshness_score=0.9,
authority_score=0.98,
grounded_fields=["microsoft_docs_search"],
raw_metadata={"provider": "microsoft_learn_mcp"},
)
]
monkeypatch.setattr(MicrosoftLearnMCPRuntime, "search", _fake_search)
evidence = await McpConfigService(db_session, tenant_id=1).search_external(
query="azure openai quickstart",
result_limit=3,
)
assert len(evidence) == 1
assert evidence[0].source_type == "mcp_search"
assert evidence[0].target_url == "https://learn.microsoft.com/azure/ai-foundry/openai/chatgpt-quickstart"
@pytest.mark.asyncio
async def test_external_query_gateway_prefers_backend_direct_mcp_when_available(db_session, monkeypatch) -> None:
await _seed_backend_direct_server(db_session)
async def _fake_search_external(self, *, query: str, result_limit: int = 3): # noqa: ANN001
_ = (self, query, result_limit)
return [
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Microsoft Learn MCP",
title="Azure OpenAI quickstart",
target_url="https://learn.microsoft.com/azure/ai-foundry/openai/chatgpt-quickstart",
snippet="Official Microsoft quickstart.",
retrieved_at="2026-04-15T12:30:00+08:00",
freshness_score=0.9,
authority_score=0.98,
grounded_fields=["microsoft_docs_search"],
raw_metadata={"provider": "microsoft_learn_mcp"},
)
]
monkeypatch.setattr(McpConfigService, "search_external", _fake_search_external)
class _FakeService(AdaptiveRAGAnswerRetrievalMixin, AdaptiveRAGDeepGuardRulesMixin):
def __init__(self, db):
self.db = db
self.tenant_id = 1
self.llm = None
service = _FakeService(db_session)
rows, meta = await service._execute_external_query_gateway(
question="Azure OpenAI quickstart",
query_semantics={},
result_limit=3,
max_pages=1,
)
assert len(rows) == 1
assert isinstance(rows[0], RetrievedItem)
assert rows[0].source == "mcp_search"
assert rows[0].metadata["external_adapter"] == "mcp"
assert meta["adapter"] == "mcp"
@pytest.mark.asyncio
async def test_external_query_gateway_falls_back_when_mcp_evidence_is_off_topic() -> None:
plan = ExternalQueryPlan(
plan_version="v1",
query_kind="weather",
answer_contract="summary_with_sources",
freshness_required=True,
topic="广州未来5天天气",
subject_entities=["广州"],
location_hint="广州",
time_range_hint="未来5天",
preferred_domains=[],
search_queries=["广州 未来5天天气"],
result_limit=3,
requires_multi_source=False,
clarify_on_ambiguity=False,
)
class _Planner:
async def build_plan(self, *, question: str, query_semantics: dict | None = None):
_ = (question, query_semantics)
return plan, {"planner": "test"}
class _MCPAdapter:
adapter_name = "mcp"
async def execute(self, current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
return ExternalAdapterResult(
adapter_name="mcp",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Microsoft Learn MCP",
title="Azure Maps Weather services",
target_url="https://learn.microsoft.com/azure/azure-maps/weather-services-concepts",
snippet="Learn how Azure Maps Weather APIs expose forecast data models and request options.",
retrieved_at="2026-04-15T12:30:00+08:00",
freshness_score=0.82,
authority_score=0.98,
grounded_fields=["microsoft_docs_search"],
raw_metadata={"provider": "microsoft_learn_mcp"},
)
],
meta={"provider": "mcp"},
)
class _LiveSearchAdapter:
adapter_name = "live_search"
async def execute(self, current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
return ExternalAdapterResult(
adapter_name="live_search",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="url",
source_label="Weather Example",
title="广州未来5天天气预报",
target_url="https://weather.example/guangzhou-5day",
snippet="广州未来5天有阵雨转多云，最高温31度，最低温24度。",
retrieved_at="2026-04-15T12:35:00+08:00",
freshness_score=0.91,
authority_score=0.76,
grounded_fields=["forecast"],
raw_metadata={"provider": "live_search"},
)
],
meta={"provider": "live_search"},
)
execution = await ExternalQueryGateway(
planner=_Planner(),
registry=ExternalProviderRegistry(adapters=[_MCPAdapter(), _LiveSearchAdapter()]),
).execute(
question="帮我看看未来5天的广州天气如何",
query_semantics={"query_profile": {"needs_freshness": True}},
)
assert execution.adapter_name == "live_search"
assert execution.meta["fallback_used"] is True
assert execution.verifier["ok"] is True
@pytest.mark.asyncio
async def test_mcp_adapter_fetches_target_page_content_after_search_result() -> None:
plan = ExternalQueryPlan(
plan_version="v1",
query_kind="weather",
answer_contract="summary_with_sources",
freshness_required=True,
topic="湛江市未来五天天气",
subject_entities=["湛江市"],
location_hint="湛江市",
time_range_hint="未来5天",
preferred_domains=["weather.example"],
search_queries=["湛江市未来五天天气预报"],
result_limit=3,
requires_multi_source=False,
clarify_on_ambiguity=False,
)
class _Planner:
async def build_plan(self, *, question: str, query_semantics: dict | None = None):
_ = (question, query_semantics)
return plan, {"planner": "test"}
async def _search_callable(current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
return ExternalAdapterResult(
adapter_name="mcp",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Weather MCP",
title="Forecast detail page",
target_url="https://weather.example/forecast/zhanjiang-five-day",
snippet="Landing page",
retrieved_at="2026-04-16T01:05:00+08:00",
freshness_score=0.6,
authority_score=0.86,
grounded_fields=["mcp_search"],
raw_metadata={"provider": "weather_mcp"},
)
],
meta={"provider": "weather_mcp"},
)
async def _fetch_text(url: str) -> str:
assert url == "https://weather.example/forecast/zhanjiang-five-day"
return "页面说明。湛江未来5天有阵雨转多云，最高温33度，最低温26度。末尾。"
def _chunk_text(text: str, *, chunk_size: int = 900, overlap: int = 120) -> list[str]:
_ = (text, chunk_size, overlap)
return ["页面说明。", "湛江未来5天有阵雨转多云，最高温33度，最低温26度。"]
def _score_chunk(*, question: str, title: str, content: str, url: str) -> float:
_ = (question, title, url)
return 0.94 if "湛江未来5天" in content else 0.08
execution = await ExternalQueryGateway(
planner=_Planner(),
registry=ExternalProviderRegistry(
adapters=[
MCPExternalAdapter(
search_callable=_search_callable,
fetch_text_callable=_fetch_text,
chunk_text_callable=_chunk_text,
score_chunk_callable=_score_chunk,
max_pages=1,
)
]
),
).execute(
question="湛江市未来五天的天气",
query_semantics={"query_profile": {"needs_freshness": True}},
)
assert execution.adapter_name == "mcp"
assert execution.verifier["ok"] is True
assert execution.evidence[0].snippet == "湛江未来5天有阵雨转多云，最高温33度，最低温26度。"
assert execution.evidence[0].raw_metadata["search_snippet"] == "Landing page"
assert "page_content_snippet" in execution.evidence[0].grounded_fields
@pytest.mark.asyncio
async def test_external_query_gateway_accepts_bundle_with_normalized_anchor_match() -> None:
plan = ExternalQueryPlan(
plan_version="v1",
query_kind="weather",
answer_contract="summary_with_sources",
freshness_required=True,
topic="湛江市未来五天天气",
subject_entities=["湛江市"],
location_hint="湛江市",
time_range_hint="未来5天",
preferred_domains=["weather.com.cn", "nmc.cn"],
search_queries=["湛江市未来五天天气预报"],
result_limit=3,
requires_multi_source=False,
clarify_on_ambiguity=False,
)
class _Planner:
async def build_plan(self, *, question: str, query_semantics: dict | None = None):
_ = (question, query_semantics)
return plan, {"planner": "test"}
class _LiveSearchAdapter:
adapter_name = "live_search"
async def execute(self, current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
return ExternalAdapterResult(
adapter_name="live_search",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="url",
source_label="中国天气网",
title="湛江天气预报,湛江7天天气预报,湛江15天天气预报,湛江天气查询",
target_url="https://www.weather.com.cn/weather/101281001.shtml",
snippet="湛江未来5天有阵雨转多云，最高温33度，最低温26度。",
retrieved_at="2026-04-16T00:30:00+08:00",
freshness_score=0.92,
authority_score=0.95,
grounded_fields=["forecast"],
raw_metadata={"provider": "live_search"},
)
],
meta={"provider": "live_search"},
)
execution = await ExternalQueryGateway(
planner=_Planner(),
registry=ExternalProviderRegistry(adapters=[_LiveSearchAdapter()]),
).execute(
question="湛江市未来五天的天气",
query_semantics={"query_profile": {"needs_freshness": True}},
)
assert execution.adapter_name == "live_search"
assert execution.verifier["ok"] is True
@pytest.mark.asyncio
async def test_external_query_gateway_prefers_weather_capable_adapter_before_doc_mcp() -> None:
plan = ExternalQueryPlan(
plan_version="v1",
query_kind="weather",
answer_contract="summary_with_sources",
freshness_required=True,
topic="湛江市未来五天天气",
subject_entities=["湛江市"],
location_hint="湛江市",
time_range_hint="未来5天",
preferred_domains=["weather.com.cn", "nmc.cn"],
search_queries=["湛江市未来五天天气预报"],
result_limit=3,
requires_multi_source=False,
clarify_on_ambiguity=False,
)
call_order: list[str] = []
class _Planner:
async def build_plan(self, *, question: str, query_semantics: dict | None = None):
_ = (question, query_semantics)
return plan, {"planner": "test"}
class _MCPAdapter:
adapter_name = "mcp"
supported_query_kinds = {"model", "company", "generic_web", "unknown"}
async def execute(self, current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
call_order.append("mcp")
return ExternalAdapterResult(
adapter_name="mcp",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="mcp_search",
source_label="Microsoft Learn MCP",
title="Azure Maps Weather service coverage",
target_url="https://learn.microsoft.com/azure/azure-maps/weather-coverage",
snippet="Weather service coverage.",
retrieved_at="2026-04-16T00:30:00+08:00",
freshness_score=0.82,
authority_score=0.98,
grounded_fields=["microsoft_docs_search"],
raw_metadata={"provider": "mcp"},
)
],
meta={"provider": "mcp"},
)
class _LiveSearchAdapter:
adapter_name = "live_search"
supported_query_kinds = {"weather", "news", "market", "generic_web", "unknown"}
async def execute(self, current_plan: ExternalQueryPlan) -> ExternalAdapterResult:
_ = current_plan
call_order.append("live_search")
return ExternalAdapterResult(
adapter_name="live_search",
status="success",
evidence=[
ExternalEvidenceItem(
source_type="url",
source_label="中国天气网",
title="湛江天气预报,湛江7天天气预报,湛江15天天气预报,湛江天气查询",
target_url="https://www.weather.com.cn/weather/101281001.shtml",
snippet="湛江未来5天有阵雨转多云，最高温33度，最低温26度。",
retrieved_at="2026-04-16T00:30:00+08:00",
freshness_score=0.92,
authority_score=0.95,
grounded_fields=["forecast"],
raw_metadata={"provider": "live_search"},
)
],
meta={"provider": "live_search"},
)
execution = await ExternalQueryGateway(
planner=_Planner(),
registry=ExternalProviderRegistry(adapters=[_MCPAdapter(), _LiveSearchAdapter()]),
).execute(
question="湛江市未来五天的天气",
query_semantics={"query_profile": {"needs_freshness": True}},
)
assert call_order == ["live_search"]
assert execution.adapter_name == "live_search"
assert execution.verifier["ok"] is True

120
backend/tests/test_mcp_runtime_service.py Normal file
View File

@ -0,0 +1,120 @@
from __future__ import annotations
import json
import httpx
import pytest
from app.services.mcp_config.runtime import MicrosoftLearnMCPRuntime
def _sse(payload: dict) -> httpx.Response:
body = f"event: message\ndata: {json.dumps(payload, ensure_ascii=False)}\n\n"
return httpx.Response(
200,
headers={"content-type": "text/event-stream", "mcp-session-id": "test-session"},
text=body,
)
@pytest.mark.asyncio
async def test_microsoft_learn_runtime_probe_initializes_and_lists_tools() -> None:
calls: list[dict] = []
def _handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content.decode("utf-8"))
calls.append(payload)
if payload["method"] == "initialize":
return _sse(
{
"jsonrpc": "2.0",
"id": payload["id"],
"result": {
"serverInfo": {"name": "Microsoft Learn MCP Server", "version": "1.0.0"},
},
}
)
if payload["method"] == "tools/list":
assert request.headers["mcp-session-id"] == "test-session"
return _sse(
{
"jsonrpc": "2.0",
"id": payload["id"],
"result": {
"tools": [
{"name": "microsoft_docs_search"},
{"name": "microsoft_docs_fetch"},
]
},
}
)
raise AssertionError(f"unexpected method: {payload['method']}")
runtime = MicrosoftLearnMCPRuntime(
endpoint_url="https://learn.microsoft.com/api/mcp",
client_factory=lambda: httpx.AsyncClient(transport=httpx.MockTransport(_handler)),
)
probe = await runtime.probe()
assert probe["status"] == "ready"
assert probe["tool_names"] == ["microsoft_docs_search", "microsoft_docs_fetch"]
assert [item["method"] for item in calls] == ["initialize", "tools/list"]
@pytest.mark.asyncio
async def test_microsoft_learn_runtime_search_returns_external_evidence() -> None:
def _handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content.decode("utf-8"))
if payload["method"] == "initialize":
return _sse(
{
"jsonrpc": "2.0",
"id": payload["id"],
"result": {
"serverInfo": {"name": "Microsoft Learn MCP Server", "version": "1.0.0"},
},
}
)
if payload["method"] == "tools/call":
assert payload["params"]["name"] == "microsoft_docs_search"
assert payload["params"]["arguments"]["query"] == "azure openai quickstart"
return _sse(
{
"jsonrpc": "2.0",
"id": payload["id"],
"result": {
"content": [
{
"type": "text",
"text": json.dumps(
{
"results": [
{
"title": "Quickstart: Azure OpenAI",
"content": "Use Azure OpenAI with official quickstart guidance.",
"contentUrl": "https://learn.microsoft.com/azure/ai-foundry/openai/chatgpt-quickstart",
}
]
},
ensure_ascii=False,
),
}
]
},
}
)
raise AssertionError(f"unexpected method: {payload['method']}")
runtime = MicrosoftLearnMCPRuntime(
endpoint_url="https://learn.microsoft.com/api/mcp",
client_factory=lambda: httpx.AsyncClient(transport=httpx.MockTransport(_handler)),
)
evidence = await runtime.search(query="azure openai quickstart", result_limit=3)
assert len(evidence) == 1
assert evidence[0].source_type == "mcp_search"
assert evidence[0].title == "Quickstart: Azure OpenAI"
assert evidence[0].target_url == "https://learn.microsoft.com/azure/ai-foundry/openai/chatgpt-quickstart"
assert evidence[0].authority_score >= 0.9

104
backend/tests/test_model_routing_service.py
View File

@ -362,3 +362,107 @@ async def test_model_context_bundle_summary_uses_routing_names(db_env, monkeypat
first_row = summary["function_routing_template_matrix"][0]
assert "routing_snapshot" in first_row
assert "model_routing" not in first_row
@pytest.mark.asyncio
async def test_get_embedding_async_falls_back_to_local_ollama_embed_when_active_engine_is_not_embedding_capable(
db_env,
monkeypatch,
) -> None:
db_session, _maker = db_env
captured: dict[str, object] = {}
class _FakeResponse:
def raise_for_status(self) -> None:
return None
def json(self) -> dict[str, object]:
return {"embeddings": [[0.11, 0.22, 0.33]]}
class _FakeAsyncClient:
def __init__(self, *args, **kwargs) -> None:
_ = args
captured["init_kwargs"] = dict(kwargs)
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, tb) -> None:
_ = (exc_type, exc, tb)
return None
async def post(self, url: str, headers: dict[str, str], json: dict[str, object]):
captured["url"] = url
captured["headers"] = dict(headers)
captured["payload"] = dict(json)
return _FakeResponse()
async def _fake_resolve_active_engine(db, *, tenant_id: int):
_ = (db, tenant_id)
return {
"provider": "glm",
"model": "qwen-plus",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"api_key": "glm-key",
}
monkeypatch.setattr(llm_service, "_resolve_active_engine", _fake_resolve_active_engine)
monkeypatch.setattr("app.services.llm_service.mixin_task_helpers.httpx.AsyncClient", _FakeAsyncClient)
monkeypatch.setenv("OLLAMA_HOST", "http://127.0.0.1:11434")
monkeypatch.setenv("OLLAMA_EMBED_MODEL", "nomic-embed-text:latest")
embedding = await llm_service.get_embedding_async(
"广州今天天气如何",
db=db_session,
tenant_id=1,
)
assert embedding == [0.11, 0.22, 0.33]
assert captured["url"] == "http://127.0.0.1:11434/api/embed"
assert captured["payload"] == {"model": "nomic-embed-text:latest", "input": "广州今天天气如何"}
assert captured["init_kwargs"]["trust_env"] is False
@pytest.mark.asyncio
async def test_get_embedding_async_returns_empty_when_all_real_backends_fail_instead_of_pseudo_vector(
db_env,
monkeypatch,
) -> None:
db_session, _maker = db_env
class _FailingAsyncClient:
def __init__(self, *args, **kwargs) -> None:
_ = (args, kwargs)
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, tb) -> None:
_ = (exc_type, exc, tb)
return None
async def post(self, url: str, headers: dict[str, str], json: dict[str, object]):
_ = (url, headers, json)
raise RuntimeError("embedding backend unavailable")
async def _fake_resolve_active_engine(db, *, tenant_id: int):
_ = (db, tenant_id)
return {
"provider": "glm",
"model": "qwen-plus",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"api_key": "glm-key",
}
monkeypatch.setattr(llm_service, "_resolve_active_engine", _fake_resolve_active_engine)
monkeypatch.setattr("app.services.llm_service.mixin_task_helpers.httpx.AsyncClient", _FailingAsyncClient)
monkeypatch.setenv("OLLAMA_HOST", "http://127.0.0.1:11434")
monkeypatch.setenv("OLLAMA_EMBED_MODEL", "nomic-embed-text:latest")
embedding = await llm_service.get_embedding_async(
"今天荔湾区天气如何",
db=db_session,
tenant_id=1,
)
assert embedding == []

75
backend/tests/test_project_knowledge_graph_event_queue_contract.py
View File

@ -17,6 +17,8 @@ from app.models.kg_evidence import KGEvidence
from app.models.kg_relation import KGRelation
from app.models.kg_sync_run import KGSyncRun
from app.models.project import Project
from app.models.semantic_fact import SemanticFact
from app.models.semantic_fact_extraction_run import SemanticFactExtractionRun
from app.models.signal import Signal
from app.models.user import User
from app.models.work_item import WorkItem
@ -42,6 +44,8 @@ async def db_session():
IngestOffset.__table__,
WorkItem.__table__,
AIOrchestrationRun.__table__,
SemanticFact.__table__,
SemanticFactExtractionRun.__table__,
KGEntity.__table__,
KGRelation.__table__,
KGEvidence.__table__,
@ -191,6 +195,77 @@ async def test_enqueue_work_item_delta_events_reads_feishu_work_items_into_kg(db
assert relation is not None
@pytest.mark.asyncio
async def test_process_work_item_event_enqueues_semantic_fact_extract_for_project(db_session) -> None:
project = Project(
tenant_id=1,
name="alpha",
title="Alpha 项目",
status="active",
owner_id=None,
assigned_role="backend",
)
seed_ingest_event = IngestEvent(
tenant_id=1,
channel="seed",
event_key="seed-semantic-fact",
status="done",
payload={},
)
db_session.add_all([project, seed_ingest_event])
await db_session.flush()
target_work_item = WorkItem(
tenant_id=1,
ingest_event_id=int(seed_ingest_event.id),
source_system="feishu",
source_message_id="om_work_item_semantic_fact",
state="waiting_manual",
intent="project.create",
project_id=int(project.id),
execution_result={"status": "pending_manual"},
review_summary={"verified": False, "issues": ["pending_manual"]},
)
db_session.add(target_work_item)
await db_session.flush()
run = AIOrchestrationRun(
tenant_id=1,
trace_id="trace-semantic-fact-1",
run_type="ingress",
status="completed",
source_system="feishu",
source_message_id="om_work_item_semantic_fact",
sender="测试用户",
channel_ref="chat-alpha",
message_text="请创建项目 Alpha",
work_item_id=int(target_work_item.id),
project_id=int(project.id),
)
db_session.add(run)
await db_session.commit()
service = ProjectKnowledgeGraphService(db_session, tenant_id=1)
await service.enqueue_work_item_delta_events(limit=20)
process_result = await service.process_pending_events(max_batch=20)
semantic_event = (
await db_session.execute(
select(IngestEvent).where(
IngestEvent.tenant_id == 1,
IngestEvent.channel == "semantic_fact_extract",
IngestEvent.event_key == f"project:{int(project.id)}",
)
)
).scalar_one_or_none()
assert process_result["done"] >= 1
assert semantic_event is not None
assert semantic_event.payload["entity_type"] == "project"
assert semantic_event.payload["entity_id"] == int(project.id)
assert semantic_event.payload["trigger_type"] == "work_item_delta"
@pytest.mark.asyncio
async def test_enqueue_feishu_delta_events_is_compat_alias_for_work_item_delta(db_session) -> None:
project = Project(

178
backend/tests/test_semantic_fact_admin_api.py Normal file
View File

@ -0,0 +1,178 @@
from __future__ import annotations
from types import SimpleNamespace
import pytest
import pytest_asyncio
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
from sqlalchemy import JSON, select
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.api.semantic_facts_api import routes as semantic_fact_routes
from app.dependencies.auth import get_current_user, get_tenant_id
from app.core.database import get_db
from app.models.base import Base
from app.models.ingest_event import IngestEvent
from app.models.knowledge_base_doc import KnowledgeBaseDoc
from app.models.project import Project
from app.models.semantic_fact import SemanticFact
from app.models.semantic_fact_extraction_run import SemanticFactExtractionRun
from app.models.signal import Signal
from app.models.task import Task
from app.models.truth_node import TruthNode
from app.models.user import User
def _use_sqlite_json_types(*tables) -> None:
for table in tables:
for column in table.columns:
if column.type.__class__.__name__ in {"JSONB", "ARRAY", "Vector"}:
column.type = JSON()
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
tables = [
User.__table__,
Project.__table__,
Signal.__table__,
TruthNode.__table__,
Task.__table__,
KnowledgeBaseDoc.__table__,
SemanticFact.__table__,
SemanticFactExtractionRun.__table__,
IngestEvent.__table__,
]
_use_sqlite_json_types(*tables)
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: Base.metadata.create_all(sync_conn, tables=tables))
maker = async_sessionmaker(engine, expire_on_commit=False)
async with maker() as session:
yield session
await engine.dispose()
def _build_app(db_session, *, role: str) -> FastAPI:
app = FastAPI()
app.include_router(semantic_fact_routes.router)
async def _fake_get_db():
yield db_session
app.dependency_overrides[get_db] = _fake_get_db
app.dependency_overrides[get_tenant_id] = lambda: 1
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=7,
tenant_id=1,
role=role,
username="tester",
email="tester@example.com",
name="Tester",
)
return app
@pytest.mark.asyncio
async def test_semantic_fact_admin_api_requires_manager_role(db_session) -> None:
app = _build_app(db_session, role="AI")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/semantic-facts/admin/overview")
assert response.status_code == 403
@pytest.mark.asyncio
async def test_semantic_fact_admin_api_exposes_overview_objects_detail_and_actions(db_session) -> None:
project = Project(
tenant_id=1,
name="invoice-helper",
title="开票助手",
description="自动生成发票",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.flush()
db_session.add(
SemanticFact(
tenant_id=1,
entity_type="project",
entity_id=int(project.id),
fact_namespace="alias",
fact_key="aliases",
fact_value_text="开票系统",
fact_value_norm="invoice_system",
value_type="keyword",
confidence=0.95,
evidence_source_type="project",
evidence_source_id=int(project.id),
evidence_excerpt="自动生成发票",
extract_version="semantic-facts-v1",
freshness_at=None,
is_active=True,
)
)
db_session.add(
SemanticFactExtractionRun(
tenant_id=1,
entity_type="project",
entity_id=int(project.id),
trigger_type="manual_retry",
status="completed",
input_fingerprint="fp-1",
facts_written=1,
failure_reason=None,
extract_version="semantic-facts-v1",
started_at="2026-04-15T10:00:00+00:00",
finished_at="2026-04-15T10:00:02+00:00",
)
)
await db_session.commit()
app = _build_app(db_session, role="ADMIN")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
overview_response = await client.get("/semantic-facts/admin/overview")
objects_response = await client.get("/semantic-facts/admin/objects", params={"entity_type": "project", "q": "开票"})
detail_response = await client.get(f"/semantic-facts/admin/objects/project/{int(project.id)}")
retry_response = await client.post(f"/semantic-facts/admin/objects/project/{int(project.id)}/reextract")
retry_event = (
await db_session.execute(
select(IngestEvent).where(
IngestEvent.tenant_id == 1,
IngestEvent.channel == "semantic_fact_extract",
IngestEvent.event_key == f"project:{int(project.id)}",
)
)
).scalar_one()
retry_trigger_type = retry_event.payload["trigger_type"]
backfill_response = await client.post(
"/semantic-facts/admin/backfill",
json={"entity_types": ["project"], "only_missing": False},
)
assert overview_response.status_code == 200
assert overview_response.json()["code"] == 0
assert overview_response.json()["data"]["queue"]["pending"] >= 1
assert objects_response.status_code == 200
assert objects_response.json()["code"] == 0
assert objects_response.json()["data"]["items"][0]["display_name"] == "开票助手"
assert detail_response.status_code == 200
assert detail_response.json()["code"] == 0
assert detail_response.json()["data"]["facts"][0]["fact_value_text"] == "开票系统"
assert retry_response.status_code == 200
assert retry_response.json()["code"] == 0
assert retry_trigger_type == "manual_retry"
assert backfill_response.status_code == 200
assert backfill_response.json()["code"] == 0
assert backfill_response.json()["data"]["enqueued"] >= 1

250
backend/tests/test_semantic_fact_runtime_hooks.py Normal file
View File

@ -0,0 +1,250 @@
from __future__ import annotations
import pytest
import pytest_asyncio
from sqlalchemy import JSON, select
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.models.base import Base
from app.models.ingest_event import IngestEvent
from app.models.knowledge_base_doc import KnowledgeBaseDoc
from app.models.project import Project
from app.models.signal import Signal
from app.models.task import Task
from app.models.truth_node import TruthNode
from app.models.user import User
from app.services.semantic_fact_service.hooks import register_semantic_fact_hooks
def _use_sqlite_json_types(*tables) -> None:
for table in tables:
for column in table.columns:
if column.type.__class__.__name__ in {"JSONB", "ARRAY", "Vector"}:
column.type = JSON()
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
tables = [
User.__table__,
Project.__table__,
Signal.__table__,
TruthNode.__table__,
Task.__table__,
KnowledgeBaseDoc.__table__,
IngestEvent.__table__,
]
_use_sqlite_json_types(*tables)
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: Base.metadata.create_all(sync_conn, tables=tables))
maker = async_sessionmaker(engine, expire_on_commit=False)
async with maker() as session:
yield session
await engine.dispose()
@pytest.mark.asyncio
async def test_semantic_fact_hooks_enqueue_extract_events_for_supported_entities(db_session) -> None:
register_semantic_fact_hooks()
user = User(
tenant_id=1,
username="semantic_admin",
email="semantic_admin@example.com",
password_hash="hash",
name="管理员",
role="ADMIN",
status="active",
is_active=True,
)
db_session.add(user)
await db_session.flush()
project = Project(
tenant_id=1,
name="alpha",
title="Alpha 项目",
description="项目描述",
owner_id=int(user.id),
status="active",
assigned_role="backend",
)
signal = Signal(
tenant_id=1,
title="信号标题",
content="信号内容",
status="待处理",
processed=False,
priority="high",
)
doc = KnowledgeBaseDoc(
tenant_id=1,
title="知识文档",
file_type="md",
source_type="file",
status="complete",
uploaded_by=int(user.id),
)
db_session.add_all([project, signal, doc])
await db_session.flush()
charter = TruthNode(
tenant_id=1,
node_type="charter",
parent_id=None,
project_id=int(project.id),
signal_id=None,
created_by=int(user.id),
title="开票系统需求",
content={"summary": "建设开票系统"},
version=1,
status="draft",
)
release = TruthNode(
tenant_id=1,
node_type="release",
parent_id=None,
project_id=int(project.id),
signal_id=None,
created_by=int(user.id),
title="发布记录",
content={"summary": "上线版本"},
version=1,
status="pending",
)
review = TruthNode(
tenant_id=1,
node_type="review",
parent_id=None,
project_id=int(project.id),
signal_id=None,
created_by=int(user.id),
title="阶段复盘",
content={"summary": "复盘内容"},
version=1,
status="draft",
)
db_session.add_all([charter, release, review])
await db_session.flush()
task = Task(
tenant_id=1,
truth_node_id=int(charter.id),
project_id=int(project.id),
assignee_id=int(user.id),
ai_recommended_assignees=None,
ai_breakdown_payload=None,
manual_overrides=None,
title="实现开票接口",
description="任务描述",
required_skill_tag_ids=[],
status="todo",
priority="high",
estimated_hours=None,
actual_hours=None,
quality_score=None,
due_date=None,
completed_at=None,
)
db_session.add(task)
await db_session.commit()
rows = (
await db_session.execute(
select(IngestEvent).where(
IngestEvent.tenant_id == 1,
IngestEvent.channel == "semantic_fact_extract",
).order_by(IngestEvent.event_key.asc())
)
).scalars().all()
payload_by_key = {str(row.event_key): row.payload for row in rows}
assert payload_by_key["document:%s" % int(doc.id)]["operation"] == "extract"
assert payload_by_key["project:%s" % int(project.id)]["operation"] == "extract"
assert payload_by_key["release:%s" % int(release.id)]["entity_type"] == "release"
assert payload_by_key["requirement:%s" % int(charter.id)]["entity_type"] == "requirement"
assert payload_by_key["review:%s" % int(review.id)]["entity_type"] == "review"
assert payload_by_key["signal:%s" % int(signal.id)]["entity_type"] == "signal"
assert payload_by_key["task:%s" % int(task.id)]["entity_type"] == "task"
assert payload_by_key["user:%s" % int(user.id)]["entity_type"] == "user"
@pytest.mark.asyncio
async def test_semantic_fact_hooks_switch_to_deactivate_for_archive_disable_and_delete(db_session) -> None:
register_semantic_fact_hooks()
user = User(
tenant_id=1,
username="semantic_admin_2",
email="semantic_admin_2@example.com",
password_hash="hash",
name="管理员2",
role="ADMIN",
status="active",
is_active=True,
)
project = Project(
tenant_id=1,
name="alpha",
title="Alpha 项目",
description="项目描述",
owner_id=None,
status="active",
assigned_role="backend",
)
signal = Signal(
tenant_id=1,
title="信号标题",
content="信号内容",
status="待处理",
processed=False,
priority="high",
)
doc = KnowledgeBaseDoc(
tenant_id=1,
title="知识文档",
file_type="md",
source_type="file",
status="complete",
uploaded_by=None,
)
db_session.add_all([user, project, signal, doc])
await db_session.flush()
review = TruthNode(
tenant_id=1,
node_type="review",
parent_id=None,
project_id=int(project.id),
signal_id=None,
created_by=int(user.id),
title="阶段复盘",
content={"summary": "复盘内容"},
version=1,
status="draft",
)
db_session.add(review)
await db_session.commit()
user.is_active = False
user.status = "offline"
signal.status = "已关闭"
review.status = "archived"
await db_session.delete(doc)
await db_session.commit()
rows = (
await db_session.execute(
select(IngestEvent).where(
IngestEvent.tenant_id == 1,
IngestEvent.channel == "semantic_fact_extract",
)
)
).scalars().all()
payload_by_key = {str(row.event_key): row.payload for row in rows}
assert payload_by_key["document:%s" % int(doc.id)]["operation"] == "deactivate"
assert payload_by_key["review:%s" % int(review.id)]["operation"] == "deactivate"
assert payload_by_key["signal:%s" % int(signal.id)]["operation"] == "deactivate"
assert payload_by_key["user:%s" % int(user.id)]["operation"] == "deactivate"

575
backend/tests/test_semantic_fact_service.py Normal file
View File

@ -0,0 +1,575 @@
from __future__ import annotations
import pytest
import pytest_asyncio
from sqlalchemy import JSON, select
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.models.base import Base
from app.models.knowledge_base_chunk import KnowledgeBaseChunk
from app.models.knowledge_base_doc import KnowledgeBaseDoc
from app.models.project import Project
from app.models.semantic_fact import SemanticFact
from app.models.semantic_fact_extraction_run import SemanticFactExtractionRun
from app.models.signal import Signal
from app.models.task import Task
from app.models.truth_node import TruthNode
from app.models.user import User
from app.services.adaptive_rag_service.mixin_answer_payload import AdaptiveRAGAnswerPayloadMixin
from app.services.adaptive_rag_service.mixin_fast_answer_project_main import AdaptiveRAGFastAnswerProjectMainMixin
from app.services.adaptive_rag_service.mixin_fast_answer_project_utils import AdaptiveRAGFastAnswerProjectUtilsMixin
from app.services.adaptive_rag_service.mixin_retrieval_router_db import AdaptiveRAGRetrievalRouterDBMixin
from app.services.llm_service import StructuredExtractionResult
from app.services.semantic_fact_service.service import SemanticFactService
def _use_sqlite_json_types(*tables) -> None:
for table in tables:
for column in table.columns:
if column.type.__class__.__name__ in {"JSONB", "ARRAY", "Vector"}:
column.type = JSON()
@pytest_asyncio.fixture
async def db_session():
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
tables = [
User.__table__,
Project.__table__,
Signal.__table__,
TruthNode.__table__,
Task.__table__,
KnowledgeBaseDoc.__table__,
KnowledgeBaseChunk.__table__,
SemanticFact.__table__,
SemanticFactExtractionRun.__table__,
]
_use_sqlite_json_types(*tables)
async with engine.begin() as conn:
await conn.run_sync(lambda sync_conn: Base.metadata.create_all(sync_conn, tables=tables))
maker = async_sessionmaker(engine, expire_on_commit=False)
async with maker() as session:
yield session
await engine.dispose()
@pytest.mark.asyncio
async def test_semantic_fact_service_upserts_and_queries_project_fact(db_session) -> None:
project = Project(
tenant_id=1,
name="a-share-analyzer",
title="A股趋势打分工具",
description="面向证券投研团队，聚焦 A 股公告解析。",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.flush()
service = SemanticFactService(db_session, tenant_id=1)
await service.upsert_fact(
entity_type="project",
entity_id=int(project.id),
fact_namespace="theme",
fact_key="market_theme",
fact_value_text="A股",
fact_value_norm="a_share",
value_type="keyword",
confidence=0.93,
evidence_source_type="project",
evidence_source_id=int(project.id),
evidence_excerpt="聚焦 A 股公告解析",
extract_version="semantic-facts-v1",
freshness_at=None,
)
await db_session.commit()
rows = await service.query_facts(
entity_type="project",
query_type="existence",
filters=[{"fact_key": "market_theme", "value": "A股", "normalized_value": "a_share"}],
)
assert len(rows) == 1
assert rows[0].fact_key == "market_theme"
assert rows[0].fact_value_norm == "a_share"
@pytest.mark.asyncio
async def test_semantic_fact_service_extracts_project_facts_from_llm_payload(monkeypatch, db_session) -> None:
project = Project(
tenant_id=1,
name="invoice-helper",
title="开票助手",
description="面向财务运营，自动生成增值税发票草稿。",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.commit()
async def _fake_generate_structured_object(prompt: str, **kwargs):
_ = (prompt, kwargs)
return StructuredExtractionResult(
payload={
"facts": [
{
"fact_namespace": "domain",
"fact_key": "business_domain",
"fact_value_text": "开票系统",
"fact_value_norm": "invoicing",
"value_type": "keyword",
"confidence": 0.94,
"evidence_excerpt": "自动生成增值税发票草稿",
},
{
"fact_namespace": "alias",
"fact_key": "aliases",
"fact_value_text": "开票系统",
"fact_value_norm": "invoice_system",
"value_type": "keyword",
"confidence": 0.9,
"evidence_excerpt": "开票助手",
},
]
},
source_field="payload",
provider="test",
model="test-model",
validation="valid",
)
monkeypatch.setattr(
"app.services.llm_service.llm_service.generate_structured_object",
_fake_generate_structured_object,
)
service = SemanticFactService(db_session, tenant_id=1)
run = await service.extract_project_facts(project_id=int(project.id), trigger_type="manual_retry")
await db_session.commit()
facts = (
await db_session.execute(
select(SemanticFact).where(
SemanticFact.tenant_id == 1,
SemanticFact.entity_type == "project",
SemanticFact.entity_id == int(project.id),
)
)
).scalars().all()
run_row = (
await db_session.execute(
select(SemanticFactExtractionRun).where(
SemanticFactExtractionRun.tenant_id == 1,
SemanticFactExtractionRun.id == int(run.id),
)
)
).scalar_one()
assert len(facts) == 2
assert {row.fact_key for row in facts} == {"business_domain", "aliases"}
assert str(run_row.status) == "completed"
assert int(run_row.facts_written or 0) == 2
@pytest.mark.asyncio
async def test_semantic_fact_service_extract_entity_facts_replaces_active_rows_for_requirement(monkeypatch, db_session) -> None:
user = User(
tenant_id=1,
username="semantic_owner",
email="semantic_owner@example.com",
password_hash="hash",
name="语义负责人",
role="ADMIN",
status="active",
is_active=True,
)
db_session.add(user)
await db_session.flush()
charter = TruthNode(
tenant_id=1,
node_type="charter",
parent_id=None,
project_id=None,
signal_id=None,
created_by=int(user.id),
title="开票系统需求",
content={"summary": "为财务团队建设开票平台"},
version=1,
status="draft",
)
db_session.add(charter)
await db_session.commit()
payloads = [
{
"facts": [
{
"fact_namespace": "alias",
"fact_key": "aliases",
"fact_value_text": "开票系统",
"fact_value_norm": "invoice_system",
"value_type": "keyword",
"confidence": 0.93,
"evidence_excerpt": "建设开票平台",
}
]
},
{
"facts": [
{
"fact_namespace": "alias",
"fact_key": "aliases",
"fact_value_text": "财务开票平台",
"fact_value_norm": "finance_invoice_platform",
"value_type": "keyword",
"confidence": 0.95,
"evidence_excerpt": "建设开票平台",
}
]
},
]
async def _fake_generate_structured_object(prompt: str, **kwargs):
_ = (prompt, kwargs)
return StructuredExtractionResult(
payload=payloads.pop(0),
source_field="payload",
provider="test",
model="test-model",
validation="valid",
)
monkeypatch.setattr(
"app.services.llm_service.llm_service.generate_structured_object",
_fake_generate_structured_object,
)
service = SemanticFactService(db_session, tenant_id=1)
first_run = await service.extract_entity_facts(
entity_type="requirement",
entity_id=int(charter.id),
trigger_type="manual_retry",
)
second_run = await service.extract_entity_facts(
entity_type="requirement",
entity_id=int(charter.id),
trigger_type="manual_retry",
)
await db_session.commit()
all_rows = (
await db_session.execute(
select(SemanticFact).where(
SemanticFact.tenant_id == 1,
SemanticFact.entity_type == "requirement",
SemanticFact.entity_id == int(charter.id),
).order_by(SemanticFact.id.asc())
)
).scalars().all()
active_rows = [row for row in all_rows if row.is_active]
assert str(first_run.status) == "completed"
assert str(second_run.status) == "completed"
assert len(all_rows) == 2
assert len(active_rows) == 1
assert active_rows[0].fact_value_text == "财务开票平台"
assert all_rows[0].is_active is False
@pytest.mark.asyncio
async def test_semantic_fact_service_deactivate_entity_facts_marks_rows_inactive(db_session) -> None:
project = Project(
tenant_id=1,
name="market-tool",
title="趋势分析工具",
description="这是一个证券行业分析工具。",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.flush()
service = SemanticFactService(db_session, tenant_id=1)
await service.upsert_fact(
entity_type="project",
entity_id=int(project.id),
fact_namespace="theme",
fact_key="market_theme",
fact_value_text="A股",
fact_value_norm="a_share",
value_type="keyword",
confidence=0.95,
evidence_source_type="project",
evidence_source_id=int(project.id),
evidence_excerpt="A股",
extract_version="semantic-facts-v1",
freshness_at=None,
)
await db_session.commit()
deactivated = await service.deactivate_entity_facts(entity_type="project", entity_id=int(project.id))
await db_session.commit()
rows = (
await db_session.execute(
select(SemanticFact).where(
SemanticFact.tenant_id == 1,
SemanticFact.entity_type == "project",
SemanticFact.entity_id == int(project.id),
)
)
).scalars().all()
assert deactivated == 1
assert rows
assert all(row.is_active is False for row in rows)
@pytest.mark.asyncio
async def test_project_existence_answer_prefers_semantic_facts_before_text_fallback(db_session) -> None:
project = Project(
tenant_id=1,
name="market-tool",
title="趋势分析工具",
description="这是一个证券行业分析工具。",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.flush()
fact_service = SemanticFactService(db_session, tenant_id=1)
await fact_service.upsert_fact(
entity_type="project",
entity_id=int(project.id),
fact_namespace="theme",
fact_key="market_theme",
fact_value_text="A股",
fact_value_norm="a_share",
value_type="keyword",
confidence=0.95,
evidence_source_type="project",
evidence_source_id=int(project.id),
evidence_excerpt="A股",
extract_version="semantic-facts-v1",
freshness_at=None,
)
await db_session.commit()
class _FakeService(
AdaptiveRAGFastAnswerProjectMainMixin,
AdaptiveRAGFastAnswerProjectUtilsMixin,
):
def __init__(self) -> None:
self.db = db_session
self.tenant_id = 1
@staticmethod
def _extract_project_lookup_keyword(question: str) -> str: # pragma: no cover - should not be called
raise AssertionError(f"lexical keyword fallback should be skipped: {question}")
@staticmethod
def _extract_query_terms(question: str, *, preferred: str = "") -> list[str]: # pragma: no cover - should not be called
raise AssertionError(f"lexical term fallback should be skipped: {question}, {preferred}")
service = _FakeService()
result = await service._try_fast_project_existence_answer(
clean_q="系统里有没有A股项目？",
project_id=None,
query_semantics={
"subject_focus": "project",
"scope_hint": "global",
"query_type": "existence",
"operation_contract": {
"query_type": "existence",
"object_mentions": ["A股项目"],
},
},
)
assert result is not None
assert "趋势分析工具" in str(result["answer"])
assert str(result["sources"][0]["metadata"]["table"]) == "semantic_facts"
@pytest.mark.asyncio
async def test_structured_db_retrieval_prefers_semantic_fact_context_for_project_queries(db_session) -> None:
project = Project(
tenant_id=1,
name="market-tool",
title="趋势分析工具",
description="这是一个证券行业分析工具。",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add(project)
await db_session.flush()
fact_service = SemanticFactService(db_session, tenant_id=1)
await fact_service.upsert_fact(
entity_type="project",
entity_id=int(project.id),
fact_namespace="theme",
fact_key="market_theme",
fact_value_text="A股",
fact_value_norm="a_share",
value_type="keyword",
confidence=0.95,
evidence_source_type="project",
evidence_source_id=int(project.id),
evidence_excerpt="A股",
extract_version="semantic-facts-v1",
freshness_at=None,
)
await db_session.commit()
class _FakeService(
AdaptiveRAGRetrievalRouterDBMixin,
AdaptiveRAGFastAnswerProjectUtilsMixin,
AdaptiveRAGAnswerPayloadMixin,
):
def __init__(self) -> None:
self.db = db_session
self.tenant_id = 1
@staticmethod
def _extract_project_lookup_keyword(question: str) -> str: # pragma: no cover - should not be called
raise AssertionError(f"lexical keyword fallback should be skipped: {question}")
@staticmethod
def _extract_query_terms(question: str, *, preferred: str = "") -> list[str]: # pragma: no cover - should not be called
raise AssertionError(f"lexical term fallback should be skipped: {question}, {preferred}")
service = _FakeService()
rows = await service._retrieve_structured_db(
"系统里有没有A股项目？",
project_id=None,
query_semantics={
"subject_focus": "project",
"scope_hint": "global",
"query_type": "existence",
"operation_contract": {
"query_type": "existence",
"object_mentions": ["A股项目"],
},
},
)
assert rows
assert rows[0].metadata["table"] == "semantic_facts"
assert "趋势分析工具" in rows[0].content
@pytest.mark.asyncio
async def test_structured_db_retrieval_prefers_semantic_fact_context_for_task_queries(db_session) -> None:
user = User(
tenant_id=1,
username="task_owner",
email="task_owner@example.com",
password_hash="hash",
name="任务负责人",
role="ADMIN",
status="active",
is_active=True,
)
project = Project(
tenant_id=1,
name="invoice-project",
title="开票平台项目",
description="项目描述",
owner_id=None,
status="active",
assigned_role="backend",
)
db_session.add_all([user, project])
await db_session.flush()
charter = TruthNode(
tenant_id=1,
node_type="charter",
parent_id=None,
project_id=int(project.id),
signal_id=None,
created_by=int(user.id),
title="开票需求",
content={"summary": "建设开票系统"},
version=1,
status="draft",
)
db_session.add(charter)
await db_session.flush()
task = Task(
tenant_id=1,
truth_node_id=int(charter.id),
project_id=int(project.id),
assignee_id=int(user.id),
ai_recommended_assignees=None,
ai_breakdown_payload=None,
manual_overrides=None,
title="实现开票接口",
description="对接发票创建能力",
required_skill_tag_ids=[],
status="todo",
priority="high",
estimated_hours=None,
actual_hours=None,
quality_score=None,
due_date=None,
completed_at=None,
)
db_session.add(task)
await db_session.flush()
fact_service = SemanticFactService(db_session, tenant_id=1)
await fact_service.upsert_fact(
entity_type="task",
entity_id=int(task.id),
fact_namespace="alias",
fact_key="aliases",
fact_value_text="开票任务",
fact_value_norm="invoice_task",
value_type="keyword",
confidence=0.96,
evidence_source_type="task",
evidence_source_id=int(task.id),
evidence_excerpt="实现开票接口",
extract_version="semantic-facts-v1",
freshness_at=None,
)
await db_session.commit()
class _FakeService(
AdaptiveRAGRetrievalRouterDBMixin,
AdaptiveRAGFastAnswerProjectUtilsMixin,
AdaptiveRAGAnswerPayloadMixin,
):
def __init__(self) -> None:
self.db = db_session
self.tenant_id = 1
service = _FakeService()
rows = await service._retrieve_structured_db(
"有没有开票任务？",
project_id=None,
query_semantics={
"subject_focus": "task",
"scope_hint": "global",
"query_type": "existence",
"operation_contract": {
"query_type": "existence",
"object_mentions": ["开票任务"],
},
},
)
assert rows
assert rows[0].metadata["table"] == "semantic_facts"
assert "开票任务" in rows[0].content

58
backend/tests/test_semantic_lint.py
View File

@ -48,6 +48,64 @@ def test_semantic_lint_flags_workflow_semantic_planner_directory(tmp_path) -> No
assert problems[0].rule_id == "semantic_hint_constant"
def test_semantic_lint_flags_query_route_object_tables_in_adaptive_rag(tmp_path) -> None:
module = _load_semantic_lint_module()
target = tmp_path / "backend" / "app" / "services" / "adaptive_rag_service" / "route_gateway.py"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text('_INTERNAL_ROUTE_OBJECTS = {"system_config"}\n', encoding="utf-8")
problems = module.collect_problems(
root=tmp_path,
scan_roots=(Path("backend/app/services/adaptive_rag_service"),),
baseline_path=Path("scripts/semantic_lint_baseline.txt"),
)
assert len(problems) == 1
assert problems[0].rule_id == "semantic_route_object_table"
def test_semantic_lint_flags_query_route_probe_helpers_in_adaptive_rag(tmp_path) -> None:
module = _load_semantic_lint_module()
target = tmp_path / "backend" / "app" / "services" / "adaptive_rag_service" / "route_gateway.py"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text("def _external_probe_is_strong(snapshot):\n return bool(snapshot)\n", encoding="utf-8")
problems = module.collect_problems(
root=tmp_path,
scan_roots=(Path("backend/app/services/adaptive_rag_service"),),
baseline_path=Path("scripts/semantic_lint_baseline.txt"),
)
assert len(problems) == 1
assert problems[0].rule_id == "semantic_route_probe_helper"
def test_semantic_lint_flags_external_object_kind_mapping(tmp_path) -> None:
module = _load_semantic_lint_module()
target = tmp_path / "backend" / "app" / "services" / "external_query_service" / "planner.py"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(
'if "weather" in referenced_objects:\n query_kind = "weather"\n',
encoding="utf-8",
)
problems = module.collect_problems(
root=tmp_path,
scan_roots=(Path("backend/app/services/external_query_service"),),
baseline_path=Path("scripts/semantic_lint_baseline.txt"),
)
assert len(problems) == 1
assert problems[0].rule_id == "semantic_object_kind_mapping"
def test_semantic_lint_default_roots_cover_adaptive_rag_and_external_query() -> None:
module = _load_semantic_lint_module()
assert Path("backend/app/services/adaptive_rag_service") in module.DEFAULT_SCAN_ROOTS
assert Path("backend/app/services/external_query_service") in module.DEFAULT_SCAN_ROOTS
def test_semantic_lint_repo_has_no_compat_baseline_exceptions() -> None:
repo_root = Path(__file__).resolve().parents[2]
baseline_path = repo_root / "scripts" / "semantic_lint_baseline.txt"

2
backend/tmp_acceptance/consultant_dispatch_feishu_runs.jsonl
View File

@ -3,3 +3,5 @@
{"generated_at": "2026-03-28T08:52:16.466761", "case": "challenge", "source_message_id": "codex-feishu-challenge-1774659109", "question": "我要把整个系统一次性全部重构掉，这么做靠谱吗", "run_status": "missing", "processing": {"status": "success", "route": "ai_pmo.query.answer", "intent": "query.answer", "reply_source": "action_succeeded", "error_message": ""}, "outbox": [{"id": 655, "status": "sent", "attempts": 1, "sent_message_id": "om_x100b53462dfeb8a4b4a22d394e5186c", "text": "不靠谱。 \n依据：当前系统后端模块统计为全零（api/services/models/schemas均为0），但热点模块集中在`backend/app/api/sstc.py`等5个文件，说明核心逻辑实际集中、未模块化；前端仅6个路由，功能边界清晰；156个待处理信号中P0级3个、P1级46个，系统处于高负载运行态。 \n下一步：立即暂停重构决策，优先完成`sstc.py`模块拆解图谱与接口契约定义，并确认杨宇（唯一跨角色开发者，workload_ratio=796.67）是否参与重构设计。"}], "final_reply": "不靠谱。 \n依据：当前系统后端模块统计为全零（api/services/models/schemas均为0），但热点模块集中在`backend/app/api/sstc.py`等5个文件，说明核心逻辑实际集中、未模块化；前端仅6个路由，功能边界清晰；156个待处理信号中P0级3个、P1级46个，系统处于高负载运行态。 \n下一步：立即暂停重构决策，优先完成`sstc.py`模块拆解图谱与接口契约定义，并确认杨宇（唯一跨角色开发者，workload_ratio=796.67）是否参与重构设计。"}
{"generated_at": "2026-03-28T08:53:35.385412", "case": "brainstorm", "source_message_id": "codex-feishu-brainstorm-1774659145", "question": "我们一起想几个 AI 劝退助手的方向", "run_status": "missing", "processing": {"status": "success", "route": "ai_pmo.draft_pending_confirm", "intent": "project_signal.create", "reply_source": "draft_pending_confirm", "error_message": ""}, "outbox": [{"id": 656, "status": "sent", "attempts": 1, "sent_message_id": "om_x100b5346298338a4b2c4f90aaa6bc9f", "text": "📋 我先按产品经理视角起草一版评审建议，请确认：\n\n📌 需求草稿\n我们一起想几个 AI 劝退助手的方向\n\n🔎 同名 / 相似项目：\n· #51 AI劝退助手：现有项目已有一定描述，建议对比当前需求的目标、范围和交付边界。；若范围重叠，可能造成重复建设或维护分散。；名称或目标高度接近，优先确认是否复用或合并。\n\n🚀 MVP 建议：\n· 明确项目目标与边界\n· 完成同名/相似项目查重\n· 交付最小可验证闭环\n\n一、定位\n· 一句话描述：我们一起想几个 AI 劝退助手的方向\n· 目标客户：待补充\n\n二、核心痛点\n· 待补充核心痛点\n\n三、MVP 功能范围（8周交付）\nPhase 1：核心功能（Week 1-4）\n· 明确项目目标与边界\n· 完成同名/相似项目查重\n· 交付最小可验证闭环\nPhase 2：差异化功能（Week 5-8）\n· 待补充：确认差异化方向后再展开\n\n四、技术架构（精简版）\n· 待补充技术架构\n\n五、商业模式\n· 待补充商业模式\n\n六、验证指标\n验证指标：\n· 待补充\n\n七、快速启动建议\n快速启动建议：\n· 待补充\n\n八、团队配置\n· 待补充团队配置\n\n📈 前景判断：\n· 该项目具备落地价值，但建议先确认范围边界和现有项目复用情况，再决定是否推进\n\n🧩 需求清单：\n· 项目定位与目标\n· MVP 功能范围\n\n❓ 还缺的信息：\n· 暂无\n\n✅ 可行性：待讨论\n信息仍需补充，建议先确认边界。\n\n👤 推荐负责人：系统管理员 · 当前负载100%\n基于当前负载与任务匹配度给出的建议。\n\n📅 跟进计划：\n· 明天：确认触发时机与验收口径\n· 一周后：复核执行效果并决定是否扩展\n\n---\n回复 确认 → 入库\n回复 修改 [具体修改意见] → 重新起草\n回复 取消 → 放弃\n⏰ 24小时内未确认自动取消"}], "final_reply": "📋 我先按产品经理视角起草一版评审建议，请确认：\n\n📌 需求草稿\n我们一起想几个 AI 劝退助手的方向\n\n🔎 同名 / 相似项目：\n· #51 AI劝退助手：现有项目已有一定描述，建议对比当前需求的目标、范围和交付边界。；若范围重叠，可能造成重复建设或维护分散。；名称或目标高度接近，优先确认是否复用或合并。\n\n🚀 MVP 建议：\n· 明确项目目标与边界\n· 完成同名/相似项目查重\n· 交付最小可验证闭环\n\n一、定位\n· 一句话描述：我们一起想几个 AI 劝退助手的方向\n· 目标客户：待补充\n\n二、核心痛点\n· 待补充核心痛点\n\n三、MVP 功能范围（8周交付）\nPhase 1：核心功能（Week 1-4）\n· 明确项目目标与边界\n· 完成同名/相似项目查重\n· 交付最小可验证闭环\nPhase 2：差异化功能（Week 5-8）\n· 待补充：确认差异化方向后再展开\n\n四、技术架构（精简版）\n· 待补充技术架构\n\n五、商业模式\n· 待补充商业模式\n\n六、验证指标\n验证指标：\n· 待补充\n\n七、快速启动建议\n快速启动建议：\n· 待补充\n\n八、团队配置\n· 待补充团队配置\n\n📈 前景判断：\n· 该项目具备落地价值，但建议先确认范围边界和现有项目复用情况，再决定是否推进\n\n🧩 需求清单：\n· 项目定位与目标\n· MVP 功能范围\n\n❓ 还缺的信息：\n· 暂无\n\n✅ 可行性：待讨论\n信息仍需补充，建议先确认边界。\n\n👤 推荐负责人：系统管理员 · 当前负载100%\n基于当前负载与任务匹配度给出的建议。\n\n📅 跟进计划：\n· 明天：确认触发时机与验收口径\n· 一周后：复核执行效果并决定是否扩展\n\n---\n回复 确认 → 入库\n回复 修改 [具体修改意见] → 重新起草\n回复 取消 → 放弃\n⏰ 24小时内未确认自动取消"}
{"generated_at": "2026-04-07T00:07:35.291848", "idx": 9001, "source_message_id": "real-scene-p1-manual-9001-1775491344", "question": "跟你聊个新需求，关于换季系统自动推荐搭配衣服", "message": "跟你聊个新需求，关于换季系统自动推荐搭配衣服", "capture_status": "answered", "capture_direct_reply_text": "可以，我们先不创建项目，先把这件事收成一版需求。\n\n我先把结果主要给谁用定一下，这会直接影响后面的内容粒度和表达方式。\n\nA. 核心执行人 / 内部 PM\n更适合：先服务负责跟进、拆动作、推动落地的人\n\nB. 负责人 / 管理者\n更适合：先看结论、风险和待拍板项\n\nC. 整个团队 / 终端用户\n更适合：需要直接同步所有相关方，或结果就是给最终使用者看的\n\n你回 A / B / C，也可以直接说更准确的对象。", "consultant_dispatch": null, "run_status": "missing", "ack": "", "final_reply": "可以，我们先不创建项目，先把这件事收成一版需求。\n\n我先把结果主要给谁用定一下，这会直接影响后面的内容粒度和表达方式。\n\nA. 核心执行人 / 内部 PM\n更适合：先服务负责跟进、拆动作、推动落地的人\n\nB. 负责人 / 管理者\n更适合：先看结论、风险和待拍板项\n\nC. 整个团队 / 终端用户\n更适合：需要直接同步所有相关方，或结果就是给最终使用者看的\n\n你回 A / B / C，也可以直接说更准确的对象。", "prompt_scene": "unknown", "inferred_scene": "unknown", "actual_scene": "unknown", "thinking_leak": false, "has_capacity_warning": false, "processing": {"route": "ai_pmo.query.answer", "intent": "query.answer", "status": "success", "duration_ms": 9984, "error_message": "", "reply_source": "action_succeeded"}, "outbox": [{"id": 966, "status": "pending", "attempts": 0, "created_at": "2026-04-07 00:02:34.138267", "text": "可以，我们先不创建项目，先把这件事收成一版需求。\n\n我先把结果主要给谁用定一下，这会直接影响后面的内容粒度和表达方式。\n\nA. 核心执行人 / 内部 PM\n更适合：先服务负责跟进、拆动作、推动落地的人\n\nB. 负责人 / 管理者\n更适合：先看结论、风险和待拍板项\n\nC. 整个团队 / 终端用户\n更适合：需要直接同步所有相关方，或结果就是给最终使用者看的\n\n你回 A / B / C，也可以直接说更准确的对象。"}], "call_log_candidates": [], "pass": false, "pass_rule": "unknown"}
{"generated_at": "2026-04-16T00:59:02.932070", "idx": 99, "source_message_id": "feishu-weather-e2e-99-1776271963", "question": "湛江市未来五天的天气", "message": "湛江市未来五天的天气", "capture_status": "answered", "capture_direct_reply_text": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。", "consultant_dispatch": null, "run_status": "missing", "ack": "", "final_reply": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。", "prompt_scene": "unknown", "inferred_scene": "unknown", "actual_scene": "unknown", "thinking_leak": false, "has_capacity_warning": false, "processing": {"route": "ai_pmo.query.answer", "intent": "query.answer", "status": "success", "duration_ms": 41383, "error_message": "", "reply_source": "action_succeeded"}, "outbox": [{"id": 1146, "status": "pending", "attempts": 0, "created_at": "2026-04-16 00:54:01.540858", "text": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。"}], "call_log_candidates": [], "pass": false, "pass_rule": "unknown"}
{"generated_at": "2026-04-16T10:04:28.903335", "idx": 901, "source_message_id": "codex-feishu-weather-901-1776304695", "question": "湛江市未来五天的天气", "message": "湛江市未来五天的天气", "capture_status": "answered", "capture_direct_reply_text": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。", "consultant_dispatch": null, "run_status": "missing", "ack": "", "final_reply": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。", "prompt_scene": "unknown", "inferred_scene": "unknown", "actual_scene": "unknown", "thinking_leak": false, "has_capacity_warning": false, "processing": {"route": "ai_pmo.query.answer", "intent": "query.answer", "status": "success", "duration_ms": 56505, "error_message": "", "reply_source": "action_succeeded"}, "outbox": [{"id": 1151, "status": "pending", "attempts": 0, "created_at": "2026-04-16 09:59:27.962651", "text": "当前正在需求模式，这条线程先只处理“当前这条需求”的收敛。你可以继续回 A / B / C / D、补一句、出草稿、重出这轮；如果要聊别的，请先回复“退出需求模式”。"}], "call_log_candidates": [{"id": 6075, "scene": "unknown", "function": "adaptive_rag_answer_normal", "created_at": "2026-04-16 10:02:19.531104"}], "pass": false, "pass_rule": "unknown"}

78
backend/tmp_acceptance/feishu_weather2_run_20260416_1000.json Normal file
View File

File diff suppressed because one or more lines are too long

23
docs/architecture/ARCHITECTURE_INVENTORY.md
View File

@ -25,6 +25,7 @@
- `docs/architecture/DATAFLOW.md`：主链与数据流说明
- `docs/architecture/TASK_BRIEF_TEMPLATE.md`：任务包模板
- `docs/superpowers/plans/`：实现计划
- `docs/superpowers/specs/`：需求/方案规格
- `docs/task_briefs/`：任务包产物与验收截图
## 4. 后端库存
@ -37,6 +38,7 @@
- `backend/app/api/sstc_api/`
- `backend/app/api/signals_api/`
- `backend/app/api/signal_sources_api/`
- `backend/app/api/semantic_facts_api/`
- `backend/app/api/skill_tags_api/`
- `backend/app/api/ai_logs_api/`
@ -51,6 +53,9 @@
### 4.3 问答、模型与生成
- `backend/app/services/adaptive_rag_service/`
- `backend/app/services/adaptive_rag_service/mixin_query_route_gateway.py`
- `backend/app/services/semantic_fact_service/`
- `backend/app/services/semantic_fact_service/hooks.py`
- `backend/app/services/llm_service/`
- `backend/app/services/ai_steward_service/`
- `backend/app/services/feedback_center/`
@ -80,6 +85,9 @@
- `backend/app/models/reply_iteration.py`
- `backend/app/models/training_pipeline.py`
- `backend/app/models/mounted_ai_runtime.py`
- `backend/app/models/query_route_memory.py`
- `backend/app/models/semantic_fact.py`
- `backend/app/models/semantic_fact_extraction_run.py`
- `backend/app/api/ai_steward_api/routes_reply_iteration.py`
- `backend/app/api/ai_steward_api/routes_training_pipeline.py`
- `backend/app/api/ai_steward_api/routes_mounted_ai.py`
@ -88,7 +96,16 @@
- `backend/app/services/` 根目录已目录化收敛，新增服务优先进入对应子包。
- API canonical 入口优先使用 `*_api/` 子包，不再优先引用顶层兼容壳。
- 运行时业务动作归因优先走“对象/上下文绑定 -> `SemanticContextPacket` -> `SemanticDecision` -> gatekeeper”。
- non-plan `query.answer` 现已在 `AdaptiveRAG` 内增加 `Query Route Gateway`：只消费上游单次结构化 `query_profile/source_policy/effective_source_policy` 与显式 hint，归一化产出 `route_mode / decision_origin / reply_contract / probe_snapshot(contract audit)`，不再在运行时执行 internal/external probe 或 route memory 强拍板，再进入检索与回答主链。
- internal `query.answer` 的下一阶段基线已经确认：`Query Route Gateway` 之后不再直接靠快答分支拍板，而是统一进入 `Operation Contract -> Target Resolver -> Query DSL Plan -> 白名单受控执行器 -> semantic facts 优先 -> verifier` 主链；相关设计规格见 `docs/superpowers/specs/2026-04-15-internal-query-dsl-plan-design.md` 与 `docs/superpowers/specs/2026-04-15-semantic-facts-read-model-design.md`。
- external `query.answer` 已完成 first-pass 收口：`Query Route Gateway` 之后统一进入 `backend/app/services/external_query_service/`，按 `External Query Planner -> External Query Gateway -> External Provider Registry -> MCP adapter 优先 -> bounded fallback live search -> external answer verifier` 执行；AdaptiveRAG external probe、freshness fetch 与 external route retrieval 已接到这条链上，并对 external sources 执行 fail-closed 过滤；相关设计规格见 `docs/superpowers/specs/2026-04-15-external-query-gateway-design.md`。
- `query.answer` 的客户可见回答展示也已确认独立基线：运行时证据必须拆成 `customer_safe` 与 `admin_debug` 双视图，客户侧统一消费中文化 `display_label`，禁止直接暴露 `raw_source_label`、机器来源或内部术语；相关设计规格见 `docs/superpowers/specs/2026-04-15-answer-presentation-policy-design.md`。
- `semantic_fact_service` 已升级为统一基线：`semantic_facts / semantic_fact_extraction_runs` 读模型、ORM 提交钩子自动收集对象变更并 upsert `ingest_events(channel=semantic_fact_extract)`、统一 worker 抽取/失活，以及 internal query 的 facts-first 主路径。
- semantic facts 自动抽取覆盖 `project / task / signal / document / user / requirement(charter) / release / review` 八类业务对象；manual retry 与 backfill 走同一队列、同一 worker，不再开旁路。
- MCP 模板接入中心已接入 `backend/app/api/mcp_api/` + `backend/app/services/mcp_config/`：租户内统一保存 MCP 模板 catalog、server 配置 bundle、命令预览、validate 结果、runtime probe 与 `pending_local_apply / user_confirmed / not_required` 状态；远程 `streamable_http` 模板可由 PMFlow 后端直连，本地 `stdio` 模板只输出 Local Bridge/兼容 host 接入说明，服务端不直写客户机器 `~/.codex/config.toml`。
- `query_route_memories` 作为独立路由记忆表，承接 `QARecord` 的 route audit、feedback 回灌和离线评估样本，不复用原 answer-quality 资产表。
- `backend/tests/test_runtime_structured_semantics_contract.py` 与 `backend/tests/test_feishu_pipeline_queue_contract.py` 共同守门 runtime 结构化语义收口，锁定飞书入口、signal capture、reply/runtime routing 不得回流旧词法拍板 helper。
- `backend/tests/test_external_query_planner.py` 现已补进 external fallback 回归，锁定 planner fallback 只能产出 generic external plan，不得再根据 `weather/news/public_model/...` 之类对象映射在运行时硬分 query_kind。
## 5. 前端库存
@ -103,6 +120,7 @@
- `frontend/src/pages/feishu-config/`
- `frontend/src/pages/ai-config/`
- `frontend/src/pages/model-routing-config/`
- `frontend/src/pages/ai-config/mcp-config/`
- `frontend/src/pages/task-brief-center/`
- `frontend/src/pages/task-brief-runs/`
- `frontend/src/pages/task-brief-template/`
@ -110,6 +128,7 @@
- `frontend/src/pages/contract-sync-board/`
- `frontend/src/pages/dashboard/`
- `frontend/src/pages/knowledge-graph-context-board/`
- `frontend/src/pages/semantic-facts-board/`
- `frontend/src/pages/login/`
- `frontend/src/pages/skill-tasks/`
- `frontend/src/pages/skills/`
@ -123,7 +142,9 @@
- `frontend/src/api/aiSteward.ts`
- `frontend/src/api/taskBriefs.ts`
- `frontend/src/api/modelRouting.ts`
- `frontend/src/api/mcp.ts`
- `frontend/src/api/knowledgeGraph.ts`
- `frontend/src/api/semanticFacts.ts`
- `frontend/src/api/mountedAi.ts`
- `frontend/src/api/feedbackCenter.ts`
- `frontend/src/api/sstc.ts`
@ -137,6 +158,7 @@
- 页面级编排逻辑优先收敛到 `pages/*/` 子目录。
- API 调用统一走 `frontend/src/api/`，不在页面或组件内直接 `fetch`。
- `frontend/src/components/sstc/` 继续作为 SSTC 共享组件目录维护。
- `AI Config -> Governance Tools` 现新增 `/ai-config/mcp-config` 与 `/ai-config/semantic-facts-board` 入口；两类治理页都只对 `BOSS/ADMIN` 开放。MCP 页负责模板化配置、后端直连/Local Bridge 状态展示、runtime probe、命令预览与本机确认；semantic facts 页负责 KPI、队列、对象表与详情抽屉。
## 6. 工具与运行库存
- `backend/scripts/repair_requirement_plan_history.py`
@ -145,6 +167,7 @@
- `scripts/gen_task_brief.sh`
- `scripts/check_contract_sync.sh`
- `frontend/playwright.config.ts`
- `frontend/vitest.setup.ts`
- `frontend/e2e/feishu-project-full-flow.spec.ts`
## 7. 维护建议

17
docs/architecture/CODE_REFERENCE_INDEX.md
View File

@ -30,8 +30,19 @@
### 2.3 RAG / AI PMO / Knowledge Graph
- RAG 问答主入口：参考 `backend/app/api/rag_api/` 与 `backend/app/api/rag_ask_contract.py`
- Query 路由网关与显式来源回复合同：参考 `backend/app/services/adaptive_rag_service/mixin_query_route_gateway.py`
- External Query Service 主链：参考 `backend/app/services/external_query_service/`（`planner.py`、`gateway.py`、`registry.py`、`adapters/`）
- 运行时语义守门与回归：参考 `scripts/semantic_lint.py`、`backend/tests/test_semantic_lint.py`、`backend/tests/test_external_query_planner.py`
- MCP backend-direct runtime / Microsoft Learn bridge：参考 `backend/app/services/mcp_config/runtime.py` 与 `backend/app/services/mcp_config/service.py`
- 路由记忆样本模型：参考 `backend/app/models/query_route_memory.py`
- Internal Query `Operation Contract / Target Resolver / DSL / verifier` 主链：参考 `backend/app/services/adaptive_rag_service/mixin_target_resolution.py`、`mixin_answer_mainline.py`、`mixin_deterministic_query_core.py`、`mixin_answer_payload.py`
- Semantic facts 读模型与抽取服务：参考 `backend/app/models/semantic_fact.py`、`backend/app/models/semantic_fact_extraction_run.py`、`backend/app/services/semantic_fact_service/`
- Semantic facts 自动入队钩子：参考 `backend/app/services/semantic_fact_service/hooks.py`
- Semantic facts 异步事件消费：参考 `backend/app/services/project_knowledge_graph_service/mixin_event_queue.py`
- Semantic facts 治理后台接口：参考 `backend/app/api/semantic_facts_api/routes.py`
- KG 上下文闭环接口：参考 `backend/app/api/knowledge_graph.py`
- 任务包中心接口：参考 `backend/app/api/task_briefs.py`
- MCP 模板接入中心接口：参考 `backend/app/api/mcp_api/routes.py`
- AI PMO 恢复接口：参考 `POST /ai-pmo/work-items/{work_item_id}/resume`
- AI PMO 草稿决策接口：参考 `POST /ai-pmo/work-items/{work_item_id}/draft-decision`
- AI PMO 审批接口：参考 `POST /ai-pmo/work-items/{work_item_id}/decision`
@ -60,6 +71,11 @@
- Scheduler Center / Mounted AI Console：参考 `frontend/src/pages/scheduler-center/`
- SSTC 工作流组件：参考 `frontend/src/components/sstc/`
- Model Routing 页面：参考 `frontend/src/pages/model-routing-config/`
- MCP 模板接入页：参考 `frontend/src/pages/ai-config/mcp-config/`
- Semantic facts 治理页：参考 `frontend/src/pages/semantic-facts-board/`
- MCP 前端 API：参考 `frontend/src/api/mcp.ts`
- Semantic facts 前端 API：参考 `frontend/src/api/semanticFacts.ts`
- AI Config 治理工具入口：参考 `frontend/src/pages/ai-config/constants.ts` 与 `frontend/src/router/index.tsx`
## 4. 模型与配置真源
- 模型配置唯一来源：参考 `backend/app/services/model/model_config.py`
@ -67,6 +83,7 @@
- Prompt 聚合：`backend/app/config/ai_prompts.py`
- Prompt 场景化分片：`backend/app/config/ai_prompts_catalog/`
- Prompt registry：`backend/app/config/ai_prompt_registry.py`
- External Query Planner Prompt：`backend/app/config/ai_prompts.py` 中的 `PROMPT_EXTERNAL_QUERY_PLAN`
## 5. 维护建议
- 新增接口时：先更新本文档，再同步 `API_CONTRACTS.md`

Some files were not shown because too many files have changed in this diff Show More