300 lines
13 KiB
Markdown
300 lines
13 KiB
Markdown
# PMFlow Full Adaptive RAG
|
||
|
||
**Goal:** Questions asked in Feishu or PMFlow get answered directly by AI. Questions never enter the decision queue.
|
||
**Knowledge coverage:** Project data + document library + Feishu history + PMFlow internal records + external URLs
|
||
**Answer entry points:** Feishu direct reply + PMFlow AI Steward chat
|
||
|
||
## Progress Log
|
||
|
||
- 2026-03-02: 已完成文档页码来源引用（PDF 上传保留 page marker 并写入 chunk `page_number`，检索标签显示 `p.xx`）。
|
||
- 2026-03-02: 已完成 Feishu/PMFlow 向量检索项目组隔离（按 `project_id` 严格过滤，避免跨项目数据泄漏），并在 Feishu 问答链路接入群上下文项目推断。
|
||
- 2026-03-02: 已完成 Feishu 直答深链输出（命中 PMFlow 记录时追加 `View details → /projects/...`）。
|
||
- 2026-03-02: 已完成检索阶段时延预算控制（DB 查询超时 5s、向量检索超时 10s，超时自动降级且返回检索指标）。
|
||
- 2026-03-02: 已完成 QA Records 统计聚合接入（`/rag/qa-records/stats` 前端展示）、时间范围筛选（本周/近30天/全部）与 `corrected` 过滤。
|
||
- 2026-03-02: 已完成 Alembic 回填 `add_adaptive_rag_tables`，新增 `knowledge_base_docs / knowledge_base_chunks / qa_records / qa_feedback` 及核心索引。
|
||
- 2026-03-02: 已完成 Feishu `question/mixed` 分流直答与 AI Steward RAG 问答入口，前端已接 QA Records tab。
|
||
- 2026-03-02: 已完成 PMFlow 写时向量化（charter/task/review create/update）异步回填 `knowledge_base_chunks(source_type=pmflow)`。
|
||
- 2026-03-02: 已完成知识库管理首版（文件上传解析、URL新增与手动刷新、processing状态3秒轮询、删除级联向量）。
|
||
- 2026-03-02: 已完成 URL 周期回抓调度（默认每168小时）并提供手动触发接口。
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
User asks question (Feishu or PMFlow)
|
||
↓
|
||
Intent classification (question / statement)
|
||
↓
|
||
├─ statement → existing signal processing flow (unchanged)
|
||
│
|
||
└─ question → Router selects knowledge domain(s)
|
||
├─ Project structured data → direct DB query
|
||
├─ Document library → vector search
|
||
├─ Feishu history → vector search
|
||
├─ PMFlow records → vector search
|
||
└─ External URLs → fetch + vector search
|
||
↓
|
||
GLM-4 generates answer
|
||
↓
|
||
Feishu reply / AI Steward chat display
|
||
```
|
||
|
||
---
|
||
|
||
## Part 1 — Intent Classification
|
||
|
||
Add as the first step in the existing AI pipeline, before any other processing.
|
||
|
||
- [x] Classify every incoming Feishu message:
|
||
- `question` — interrogative, query, "does X exist", "how to", "why", "who is"
|
||
- `statement` — requirement description, project intent, signal content
|
||
- `mixed` — contains both question and statement, split and handle separately
|
||
- [x] Classify every incoming Feishu message:
|
||
- `question` — interrogative, query, "does X exist", "how to", "why", "who is"
|
||
- `statement` — requirement description, project intent, signal content
|
||
- `mixed` — contains both question and statement, split and handle separately
|
||
- [x] `question` → RAG flow, reply directly in Feishu, **never enters decision queue**
|
||
- [x] `statement` → existing signal processing flow, unchanged
|
||
- [x] `mixed` → RAG answers the question part, signal queue handles the statement part. Feishu reply includes: "I've answered your question and logged the requirement as a signal."
|
||
|
||
---
|
||
|
||
## Part 2 — Knowledge Domains
|
||
|
||
### Domain 1 — Project structured data (direct DB query, no vectors)
|
||
|
||
Real-time queries. Not vectorized — ensures data is always current.
|
||
|
||
| Queryable content | Example questions |
|
||
|------------------|------------------|
|
||
| Project list / detail / status | "Is there a customer service project?" "What is project #5?" |
|
||
| Signal source | "What signals are pending?" "How many P0 signals are there?" |
|
||
| Charter content / versions | "What is the goal in the invoicing system charter?" |
|
||
| Task progress | "How many tasks are unfinished?" "Who is working on the API?" |
|
||
| Team workload | "What's Zhang San's current workload?" |
|
||
|
||
- [x] Build `db_query_tool`: AI automatically determines which tables to query, generates SQL or calls existing APIs
|
||
- [x] Results fed directly to GLM-4 to generate natural language answer
|
||
|
||
### Domain 2 — Document library (vector search)
|
||
|
||
User-uploaded documents, specs, templates, static knowledge:
|
||
|
||
- [x] Supported formats: PDF / Word / Markdown / TXT
|
||
- [x] On upload: auto-chunk (chunk size 500 tokens, overlap 50), auto-vectorize
|
||
- [x] Store vectors in pgvector (reuse existing PostgreSQL, no new DB)
|
||
- [x] Search returns top-5 relevant chunks + source filename + page number
|
||
|
||
**New DB tables:**
|
||
```
|
||
knowledge_base_docs: id / title / file_type / uploaded_by / created_at
|
||
knowledge_base_chunks: id / doc_id / content / embedding / chunk_index / metadata
|
||
```
|
||
|
||
### Domain 3 — Feishu message history (vector search)
|
||
|
||
All received Feishu messages including filtered and archived ones:
|
||
|
||
- [x] Every Feishu message vectorized asynchronously on receipt (does not block main pipeline)
|
||
- [x] Search filterable by project group to prevent cross-project data leakage
|
||
|
||
### Domain 4 — PMFlow internal records (vector search)
|
||
|
||
Charter content, retrospective records, task descriptions:
|
||
|
||
- [x] Auto-vectorize on charter create/update
|
||
- [x] Auto-vectorize on retrospective record write
|
||
- [x] Task description vectorization (optional, enable when volume is high)
|
||
|
||
### Domain 5 — External URLs (fetch + vector search)
|
||
|
||
- [x] User can submit external URLs in knowledge base management page
|
||
- [x] On submission: `web_fetch` retrieves content → auto-chunk → vectorize → stored in `knowledge_base_chunks` with `source_type = url`
|
||
- [x] Re-fetch scheduled weekly to keep content fresh (or manual refresh button)
|
||
- [x] Supported: public documentation pages, Notion links, Google Docs (public), GitHub READMEs（当前按公开 URL 通用抓取）
|
||
|
||
---
|
||
|
||
## Part 3 — RAG Query Flow
|
||
|
||
### Router (domain selection)
|
||
|
||
- [x] AI determines which domain(s) apply to the question (multiple allowed):
|
||
- Contains project / signal / task / charter / progress / workload keywords → `db_query_tool`
|
||
- Contains document / spec / how-to / process keywords → document library search
|
||
- Contains "previously said" / "last time" / "history" → Feishu history search
|
||
- Contains retrospective / charter content / task description → PMFlow records search
|
||
- Contains URL or "according to [site]" → external URL search
|
||
- Ambiguous → parallel search across all vector domains, merge results
|
||
|
||
### Retrieve → Generate flow
|
||
|
||
```
|
||
Question
|
||
↓
|
||
Router (domain selection)
|
||
↓
|
||
Parallel retrieval (DB query + vector searches across selected domains)
|
||
↓
|
||
Context assembly:
|
||
- Original question
|
||
- Retrieved results (with source labels)
|
||
- Current project context (if question asked inside a project group)
|
||
- Last 10 turns of conversation (for follow-up questions)
|
||
↓
|
||
GLM-4 generates answer
|
||
↓
|
||
Answer + source citations
|
||
```
|
||
|
||
- [x] If retrieval relevance score < 0.75: AI responds "I couldn't find relevant information, please provide more context"
|
||
- [x] Never fabricate answers. If uncertain, explicitly state uncertainty.
|
||
- [x] Max context window per query: 8000 tokens (trim oldest retrieved chunks if exceeded)
|
||
|
||
---
|
||
|
||
## Part 4 — Answer Entry Points
|
||
|
||
### Entry point 1 — Feishu direct reply
|
||
|
||
- [x] Feishu bot receives question → RAG processing → reply directly under original message
|
||
- [x] Reply format:
|
||
```
|
||
[AI answer]
|
||
|
||
Source: Project database / Document [filename] / Feishu history / [URL]
|
||
```
|
||
- [x] If answer references a specific PMFlow page, append deep link "View details →"
|
||
- [x] Response time target: < 5s for DB queries, < 10s for vector searches
|
||
|
||
### Entry point 2 — PMFlow AI Steward chat
|
||
|
||
- [x] Reuse existing AI Steward page, add RAG capability
|
||
- [x] Multi-turn conversation, retain last 10 turns of context
|
||
- [x] Each answer shows source citation inline
|
||
- [x] Follow-up questions work: "What's the charter for that project you just mentioned?" → AI resolves reference from conversation history
|
||
- [x] Show "Searching knowledge base..." indicator while RAG is running
|
||
|
||
---
|
||
|
||
## Part 5 — Knowledge Base Management Page
|
||
|
||
Add "知识库 / Knowledge Base" tab to existing 智能配置 (AI Config) page:
|
||
|
||
- [x] File upload zone (drag & drop or click to browse)
|
||
- [x] Uploaded files list: filename / type / upload time / vectorization status (processing / complete / failed)
|
||
- [x] Delete file → also deletes all associated chunks and vectors
|
||
- [x] URL submission: input field + "Add URL" button, shows URL list with last-fetched time + manual refresh button
|
||
- [x] Vectorization status shown in real time (poll every 3s while processing)
|
||
|
||
---
|
||
|
||
## Implementation Order
|
||
|
||
| Week | Deliverables |
|
||
|------|-------------|
|
||
| 1 | Intent classification (question/statement split). `db_query_tool`. Structured project data queries. Feishu direct reply. |
|
||
| 2 | pgvector setup. Document library: upload / chunk / vectorize / search. Knowledge base management page. |
|
||
| 3 | Feishu history + PMFlow records vectorization. External URL fetch + vectorize. Multi-domain Router. Parallel search + result merge. Source citations. |
|
||
| 4 | AI Steward multi-turn RAG integration. Follow-up context linking. End-to-end test across all 5 domains. |
|
||
|
||
---
|
||
|
||
## Out of scope (this version)
|
||
|
||
- Voice-to-text input
|
||
- Image / attachment content understanding (Feishu image messages)
|
||
- Per-user knowledge base access control
|
||
- Self-hosted embedding model (use API)
|
||
- Real-time web search (not the same as indexed external URLs)
|
||
|
||
---
|
||
|
||
## Part 6 — QA Recording & Feedback (Training Data Collection)
|
||
|
||
**Placement:** New "QA Records" tab in AI Steward page, between "Daily Brief" and "Engine Config".
|
||
|
||
### Backend
|
||
|
||
**New DB tables:**
|
||
```
|
||
qa_records:
|
||
id / question / answer / sources (JSON array) /
|
||
intent / domains_used (JSON array) /
|
||
response_time_ms / asked_by / asked_via (feishu/pmflow) /
|
||
project_id (nullable) / created_at
|
||
|
||
qa_feedback:
|
||
id / qa_id /
|
||
rating (positive/negative) /
|
||
helpfulness (fully/partially/could_be_better) — for positive ratings /
|
||
error_types (JSON array: wrong_answer/wrong_source/incomplete/off_topic/other) — for negative /
|
||
correct_answer (text, optional) /
|
||
submitted_by / created_at
|
||
```
|
||
|
||
- [x] Auto-write `qa_records` on every RAG answer (both Feishu reply and AI Steward chat)
|
||
- [x] `POST /api/qa-records/{id}/feedback` — submit rating + structured feedback
|
||
- [x] `GET /api/qa-records?project_id=&date_from=&date_to=&rating=` — paginated list with filters
|
||
- [x] `GET /api/qa-records/export` — export as JSONL for model fine-tuning (admin only)
|
||
|
||
**JSONL export format:**
|
||
```json
|
||
{"prompt": "有客服工具的项目吗？", "response": "有，一个AI客服工具项目，进行中", "rating": "positive", "helpfulness": "fully"}
|
||
{"prompt": "开票系统宪章目标是什么？", "response": "...", "rating": "negative", "correct_answer": "...", "error_type": "incomplete"}
|
||
```
|
||
|
||
- [x] Export only includes records with feedback (unrated excluded by default, optional flag to include)
|
||
- [x] Admin can filter export by date range, rating, domain, project
|
||
|
||
### Frontend — QA Records Tab
|
||
|
||
**Top bar:**
|
||
```
|
||
[ Search questions... ] [ Export JSONL ] [ All ratings ▾ ] [ This week ▾ ]
|
||
```
|
||
|
||
**Each record card:**
|
||
```
|
||
┌─────────────────────────────────────────────────┐
|
||
│ 14:23 有客服工具的项目吗？ 飞书 · #5项目 │
|
||
│ AI：有，一个AI客服工具项目，进行中 │
|
||
│ 来源：项目数据库 耗时 2.3s │
|
||
│ │
|
||
│ 👍 有用 👎 有问题 ✏️ 修正 │
|
||
└─────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**On 👍 click — helpfulness picker (non-blocking, slides in below card):**
|
||
```
|
||
这条回答对你有多大帮助？
|
||
● 完全解决了问题 ○ 部分有用 ○ 有用但可以更好
|
||
[跳过] [提交]
|
||
```
|
||
|
||
**On 👎 click — structured feedback form (slides in below card):**
|
||
```
|
||
这条回答哪里有问题？（可多选）
|
||
☐ 答案错误 ☐ 来源错误 ☐ 回答不完整
|
||
☐ 答非所问 ☐ 其他
|
||
|
||
正确答案应该是：（选填，填了训练权重更高）
|
||
┌──────────────────────────────┐
|
||
│ │
|
||
└──────────────────────────────┘
|
||
[提交]
|
||
```
|
||
|
||
- [x] After submitting feedback: card shows "✅ 已评价" badge, feedback form collapses
|
||
- [x] Cards with correct_answer filled: highlighted with green left border (highest value training data)
|
||
- [x] Filter by: all / positive / negative / corrected / unrated
|
||
|
||
### Stats summary (top of QA Records tab)
|
||
|
||
```
|
||
总问答 342 👍 已评价 89% 平均帮助度 4.2/5 可导出训练对 156
|
||
```
|
||
|
||
- [x] `可导出训练对` = records with feedback (both positive with helpfulness + negative with or without correction)
|