智能搜索技术介绍
相关主题
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
Navigation
User: Switch to navigation. System: Ok, switched to Navigation system. User: Take me to Little Garden Restaurant, but stop by Bosch Page Mill Office. System: Ok, I will plan a route to Little Garden Restaurant that goes by Bosch Page Mill Office. Would you like anything else? User: Get me the fastest route.
• The dream: a vast repository of knowledge we can “talk to”
• We’re a long way from there…
Q&A Thank you !
– 从片段中抽取符合答案类型的短语作为候选 答案
– 候选答案评分
• 和问题用词的相似程度 • 和问题结构的相似程度 • 和问题语义的相似程度
问题回答: 示例
When did Hawaii become a state?
问问题题分分析析
when
句句法法分分析析
状语(When) 主语(Hawaii) 谓 语(become) 补语(a state)
命命名C名Ch实h实uu体n体nkk识in识ing别g别 Hawaii became a state on
August 21,1959
答答案案生生成成
......
分分句句
August 21,1959
6. 问题回答技术应用举例
• 电子商务领域的应用 • 车载对话系统 • Call Center导购系统
3. “Hawaii” “ became a state” + in (…. Hawaii, the last state, became a state in
1959 …..) …….. “Hawaii” “became” “state”
模块介绍：答案生成
• 输入: 问题/答案类型/片段 • 输出: 可能的答案(按相关程度排序) • 方法
Conclusion
• Question answering is an exciting research area!
– Lies at the intersection of information retrieval and natural language processing
– A real-world application of NLP technologies
Subject: Hawaii Predicate: became a state
Verb: became Complement: a state
Adverbial modifier: when
• 特征分类
When => Answer type is Date “in…” or “on…”
模块介绍：查询生成
Flat list
Question Answering
1. 任务描述 2. 和传统搜索引擎的区别 3. 基本框架 4. 关键技术 5. 基本方法 6. 应用举例
1. 任务描述
• 来源： 真实文本 • 输入： 问题 • 输出： 简洁的答案
“When did Hawaii become
a state?”
Broad/surface crawling
Customizable schedules Fixed schedule
From seconds to months Days on average
Structured, Navigational
–Taxonomy drill-down –Sorting & grouping –Clustering & collapsing
答答案案类类型型分分类类
WWoordrdNNeett
查查询询生生成成
日期 “Hawaii became a state in”
GGoooogglele
搜搜索索引引擎擎
“Hawaii became a state” + in 词词性性标标记记 “Hawaii” “ became a state” + in
模块介绍：问题分析
• 输入: 问题 Ex: When did Hawaii become a state?
• 输出 答案类型 (Date) 查询 “Hawaii became a state in” “Hawaii became a state” + in ……..
模块介来自百度文库：答案类型分类
• 词性标记/句法分析
电子商务领域的应用
车载对话系统
Fuliang Weng, et al,. CHAT to Your Destination. Proc. of the 8th SIGDial
workshop on Discourse and Dialogue, September 2007: 59-62.
Dialog Example
Fuliang Weng, et al,. CHAT to Your Destination. Proc. of the 8th SIGDial
workshop on Discourse and Dialogue, September 2007: 59-62.
Call Center
Ranjit Bose. Natural Language Processing: Current state and future directions . International Journal of the Computer, the Internet and Management Vol. 12#1 (January-April, 2004) pp 1-11
答案类型
答案排序 答案验证
多文档 自动文摘
精确定位
反馈信息
反馈信息
辨伪
去冗 答案生成
搜索引擎
相关文本
Web
4. 关键技术
• Part-of-speech tagging • NE tagging • Semantic tagging • BNP Chunking • Syntactic parsing
–Constrained context –Structured data –Domain Taxonomy
Focused/deep crawling
Web Search
Global and general Typically less structured Fixed algorithm Popularity-based
智能搜索技术介绍
-- Question Answering
广告和搜索技术中心 林锋 2010-1-28
内容
• 前言：电子商务搜索 VS Web搜索 • 问题回答技术简介
– 任务描述 – 和传统搜索引擎的区别 – 基本框架 – 关键技术 – 基本方法 – 应用举例
什么是电子商务搜索
Search Engines of Online Marketplaces
电子商务搜索量增长迅速
电子商务搜索-Web搜索
Index Size Document Type Relevance
Comprehensive ness Freshness Presentation
Vertical Search
Smaller and specialized
Typically more structured Highly customizable Relevance enhanced by
Restaurant Finder
User: Switch to restaurant. System: Ok. Switched to the restaurant finder. User: Can you find me a Chinese restaurant in Palo Alto with good service? System: There are 11 Chinese restaurants in Palo Alto that have good service. Here are the first few: Little Garden Restaurant China Delight Mandarin Gourmet Restaurant Should I continue?
“1959”
QA SYSTEM
Text Corpora & RDBMS
2. 和传统搜索引擎的区别
• 更自然的人机界面
– 自然语言问题 – 关键字
• 更加简洁的答案
– 抽取答案和必要的证据 – 排序的文档列表
3. 基本框架
问题
答案
问题分类体系 答案分类体系
问题分类 查询生成
答案分类 问题分析
查询
• 关键技术: paraphrase(重述)
1. “Hawaii became a state in” (….Hawaii became a state in 1959…..)
2. “Hawaii became a state” + in (…. in 1959 Hawaii became a state …..)
5. 基本方法
问题
问问题题分分析析
疑问词
句法成分
查查询询生生成成 查询
搜搜索索引引擎擎 片段
答答案案生生成成 答案
句句法法分分析析
答答案案类类型型分分类类 答案类型
WWoordrdNNeett GGoooogglele
词词性性标标记记 CChhuunnkkiningg 命命名名实实体体识识别别
分分句句
User: Switch to navigation. System: Ok, switched to Navigation system. User: Take me to Little Garden Restaurant, but stop by Bosch Page Mill Office. System: Ok, I will plan a route to Little Garden Restaurant that goes by Bosch Page Mill Office. Would you like anything else? User: Get me the fastest route.
• The dream: a vast repository of knowledge we can “talk to”
• We’re a long way from there…
Q&A Thank you !
– 从片段中抽取符合答案类型的短语作为候选 答案
– 候选答案评分
• 和问题用词的相似程度 • 和问题结构的相似程度 • 和问题语义的相似程度
问题回答: 示例
When did Hawaii become a state?
问问题题分分析析
when
句句法法分分析析
状语(When) 主语(Hawaii) 谓 语(become) 补语(a state)
命命名C名Ch实h实uu体n体nkk识in识ing别g别 Hawaii became a state on
August 21,1959
答答案案生生成成
......
分分句句
August 21,1959
6. 问题回答技术应用举例
• 电子商务领域的应用 • 车载对话系统 • Call Center导购系统
3. “Hawaii” “ became a state” + in (…. Hawaii, the last state, became a state in
1959 …..) …….. “Hawaii” “became” “state”
模块介绍：答案生成
• 输入: 问题/答案类型/片段 • 输出: 可能的答案(按相关程度排序) • 方法
Conclusion
• Question answering is an exciting research area!
– Lies at the intersection of information retrieval and natural language processing
– A real-world application of NLP technologies
Subject: Hawaii Predicate: became a state
Verb: became Complement: a state
Adverbial modifier: when
• 特征分类
When => Answer type is Date “in…” or “on…”
模块介绍：查询生成
Flat list
Question Answering
1. 任务描述 2. 和传统搜索引擎的区别 3. 基本框架 4. 关键技术 5. 基本方法 6. 应用举例
1. 任务描述
• 来源： 真实文本 • 输入： 问题 • 输出： 简洁的答案
“When did Hawaii become
a state?”
Broad/surface crawling
Customizable schedules Fixed schedule
From seconds to months Days on average
Structured, Navigational
–Taxonomy drill-down –Sorting & grouping –Clustering & collapsing
答答案案类类型型分分类类
WWoordrdNNeett
查查询询生生成成
日期 “Hawaii became a state in”
GGoooogglele
搜搜索索引引擎擎
“Hawaii became a state” + in 词词性性标标记记 “Hawaii” “ became a state” + in
模块介绍：问题分析
• 输入: 问题 Ex: When did Hawaii become a state?
• 输出 答案类型 (Date) 查询 “Hawaii became a state in” “Hawaii became a state” + in ……..
模块介来自百度文库：答案类型分类
• 词性标记/句法分析
电子商务领域的应用
车载对话系统
Fuliang Weng, et al,. CHAT to Your Destination. Proc. of the 8th SIGDial
workshop on Discourse and Dialogue, September 2007: 59-62.
Dialog Example
Fuliang Weng, et al,. CHAT to Your Destination. Proc. of the 8th SIGDial
workshop on Discourse and Dialogue, September 2007: 59-62.
Call Center
Ranjit Bose. Natural Language Processing: Current state and future directions . International Journal of the Computer, the Internet and Management Vol. 12#1 (January-April, 2004) pp 1-11
答案类型
答案排序 答案验证
多文档 自动文摘
精确定位
反馈信息
反馈信息
辨伪
去冗 答案生成
搜索引擎
相关文本
Web
4. 关键技术
• Part-of-speech tagging • NE tagging • Semantic tagging • BNP Chunking • Syntactic parsing
–Constrained context –Structured data –Domain Taxonomy
Focused/deep crawling
Web Search
Global and general Typically less structured Fixed algorithm Popularity-based
智能搜索技术介绍
-- Question Answering
广告和搜索技术中心 林锋 2010-1-28
内容
• 前言：电子商务搜索 VS Web搜索 • 问题回答技术简介
– 任务描述 – 和传统搜索引擎的区别 – 基本框架 – 关键技术 – 基本方法 – 应用举例
什么是电子商务搜索
Search Engines of Online Marketplaces
电子商务搜索量增长迅速
电子商务搜索-Web搜索
Index Size Document Type Relevance
Comprehensive ness Freshness Presentation
Vertical Search
Smaller and specialized
Typically more structured Highly customizable Relevance enhanced by
Restaurant Finder
User: Switch to restaurant. System: Ok. Switched to the restaurant finder. User: Can you find me a Chinese restaurant in Palo Alto with good service? System: There are 11 Chinese restaurants in Palo Alto that have good service. Here are the first few: Little Garden Restaurant China Delight Mandarin Gourmet Restaurant Should I continue?
“1959”
QA SYSTEM
Text Corpora & RDBMS
2. 和传统搜索引擎的区别
• 更自然的人机界面
– 自然语言问题 – 关键字
• 更加简洁的答案
– 抽取答案和必要的证据 – 排序的文档列表
3. 基本框架
问题
答案
问题分类体系 答案分类体系
问题分类 查询生成
答案分类 问题分析
查询
• 关键技术: paraphrase(重述)
1. “Hawaii became a state in” (….Hawaii became a state in 1959…..)
2. “Hawaii became a state” + in (…. in 1959 Hawaii became a state …..)
5. 基本方法
问题
问问题题分分析析
疑问词
句法成分
查查询询生生成成 查询
搜搜索索引引擎擎 片段
答答案案生生成成 答案
句句法法分分析析
答答案案类类型型分分类类 答案类型
WWoordrdNNeett GGoooogglele
词词性性标标记记 CChhuunnkkiningg 命命名名实实体体识识别别
分分句句