第10节：设计高效混合检索架构，提升召回精度

RAG与Agent性能调优第8节打造可配置可扩展的自动化预处理流水线Gitee地址https://gitee.com/agiforgagaplus/OptiRAGAgent文章详情目录RAG与Agent性能调优上一节第9节FAISSHNSW还是BM25如何选择最适合业务的向量检索引擎如何选择最适合业务的向量检索引擎下一节待更新混合检索的作用混合检索结合关键词匹配和语义搜索的优势提供更准确与上下文更有关的内容进而让检索更加全面混合检索的架构混合检索架构通常包括关键词匹配、语义搜索和结果重排序等组件关键词匹配使用BM25等传统的关键词匹配算法快速筛选与查询关键词相关的文档语义搜索利用向量空间模型将文档和查询转换为向量表示通过云弦相似度计算文档与查询的相似性结果重排序根据关键词匹配和语义搜索的结果通过模型学习调整结果的排名顺序提升与查询相关较高的文档排名靠前混合检索的效果混合检索的效果通常取决于架构设计、数据质量和模型训练较高的召回率通过关键词匹配和语义搜索的组合能够更加全面的筛选相关文档提升召回率较高的准确性通过结果重排序模型提更与查询相关的文档排名靠前的准确率提升用户体验Dify中混合检索的设置MilvusLlamaIndex构建混合检索%pip install llama-index-vector-stores-milvus %pip install llama-index-embeddings-openai %pip install llama-index-llms-openai启动Milvus服务器Docker 部署version: 3.5 services: etcd: container_name: milvus-etcd image: quay.io/coreos/etcd:v3.5.18 environment: - ETCD_AUTO_COMPACTION_MODErevision - ETCD_AUTO_COMPACTION_RETENTION1000 - ETCD_QUOTA_BACKEND_BYTES4294967296 - ETCD_SNAPSHOT_COUNT50000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd command: etcd -advertise-client-urlshttp://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd healthcheck: test: [CMD, etcdctl, endpoint, health] interval: 30s timeout: 20s retries: 3 minio: container_name: milvus-minio image: minio/minio:RELEASE.2023-03-20T20-16-18Z environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin ports: - 9001:9001 - 9000:9000 volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data command: minio server /minio_data --console-address :9001 healthcheck: test: [CMD, curl, -f, http://localhost:9000/minio/health/live] interval: 30s timeout: 20s retries: 3 standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.6.0-rc1 command: [milvus, run, standalone] security_opt: - seccomp:unconfined environment: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 MQ_TYPE: woodpecker volumes: - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus healthcheck: test: [CMD, curl, -f, http://localhost:9091/healthz] interval: 30s start_period: 90s timeout: 20s retries: 3 ports: - 19530:19530 - 9091:9091 depends_on: - etcd - minio networks: default: name: milvus使用docker compose up -d注意Milvus Standalone、Milvus Distributed 和 Zilliz Cloud 目前支持全文搜索但 Milvus Lite 尚不支持全文搜索。# 加载文档 from llama_index.core import SimpleDirectoryReader documents SimpleDirectoryReader(./10/docs/).load_data() print(Example document:\n, documents[0])使用BM25执行混合检索from llama_index.vector_stores.milvus import MilvusVectorStore from llama_index.core import StorageContext, VectorStoreIndex URI http://localhost:19530 # Milvus URI vector_store MilvusVectorStore( uriURI, # tokenTOKEN, dim1536, # vector dimension depends on the embedding model enable_sparseTrue, # enable the default full-text search using BM25 overwriteTrue, # drop the collection if it already exists ) storage_context StorageContext.from_defaults(vector_storevector_store) index VectorStoreIndex.from_documents(documents, storage_contextstorage_context)参数说明dim (int, optional):Collections 的嵌入向量维度。enable_sparse (bool):用于启用或禁用稀疏嵌入。默认为假。启动混合检索在查询阶段启用混合搜索将vector_store_query_mode 设置为 hybrid。开启后将对语义搜索和全文搜索的搜索结果进行合并和 Rerankers。import textwrap query_engine index.as_query_engine( vector_store_query_modehybrid, similarity_top_k10 ) response query_engine.query(孙悟空名字的由来) print(textwrap.fill(str(response), 100)) for idx, node in enumerate(response.source_nodes, 1): print(f结果 {idx}: ) print(textwrap.fill(str(node.node.text), 100)) print(\n)# 导入必要的库 from llama_index.core import SimpleDirectoryReader, VectorStoreIndex from llama_index.core.retrievers import VectorIndexRetriever from llama_index.core.query_engine import RetrieverQueryEngine import textwrap # 1. 加载文档 print(Loading documents...) documents SimpleDirectoryReader(./10/docs/).load_data() # 替换为你的文档路径 # 2. 构建索引 print(Building index...) index VectorStoreIndex.from_documents(documents) # 3. 设置混合检索参数 alpha 0.1 # 权重参数越大越偏向语义检索越小偏向关键词检索 top_k 10 # 返回前 top_k 个结果 retriever index.as_retriever( retriever_modehybrid, similarity_top_ktop_k, alphaalpha ) # 4. 创建查询引擎 query_engine RetrieverQueryEngine(retrieverretriever) # 5. 执行查询 query 孙悟空名字的由来 print(f\nQuery: {query}) response query_engine.query(query) print(\nResponse:) print(textwrap.fill(str(response), 100)) # 6. 输出来源文档片段 print(\nSource Nodes:) for idx, node in enumerate(response.source_nodes, 1): print(f结果 {idx}: ) print(textwrap.fill(str(node.node.text), 100)) print(\n)# 导入必要的库 from llama_index.core import SimpleDirectoryReader, VectorStoreIndex from llama_index.core.retrievers import VectorIndexRetriever from llama_index.core.query_engine import RetrieverQueryEngine import textwrap # 1. 加载文档 print(Loading documents...) documents SimpleDirectoryReader(./10/docs/).load_data() # 替换为你的文档路径 # 2. 构建索引 print(Building index...) index VectorStoreIndex.from_documents(documents) # 3. 设置混合检索参数 alpha 0.9 # 权重参数越大越偏向语义检索越小偏向关键词检索 top_k 10 # 返回前 top_k 个结果 retriever index.as_retriever( retriever_modehybrid, similarity_top_ktop_k, alphaalpha ) # 4. 创建查询引擎 query_engine RetrieverQueryEngine(retrieverretriever) # 5. 执行查询 query 孙悟空名字的由来 print(f\nQuery: {query}) response query_engine.query(query) print(\nResponse:) print(textwrap.fill(str(response), 100)) # 6. 输出来源文档片段 print(\nSource Nodes:) for idx, node in enumerate(response.source_nodes, 1): print(f结果 {idx}: ) print(textwrap.fill(str(node.node.text), 100)) print(\n)加权平均LlamaIndex支持通过alpha参数对语意和关键词检索结果进行线性加权融合alpha1.0 表达完全依赖语气检索alpha0.0表达完全依赖关键词检索中间值则表示两者加权融合扩展支持混合检索的平台工具及其稀疏嵌入方法