数据挖掘课件第1章

数据挖掘:引言
Data Mining 主讲教师：骆嘉伟
Office number: 软件院309 E-mail: luojiawei@
2014年10月22日星期三
1
教学目的
本课程主要介绍数据挖掘的基本概念、算法、 技术和系统，使学生理解数据挖掘在处理海量数 据过程中的作用和意义；了解现有数据挖掘的基 本理论和基本方法，能够根据实际数据挖掘问题 分析建模，开发并使用数据挖掘系统来分析大量 数据。
Provide better, customized services for an edge (e.g. in Customer Relationship Management)
Why Mine Data? Scientific Viewpoint
Data collected and stored at enormous speeds (GB/hour)
Taxable Income Cheat 75K 50K 150K ? ? ? ? ? ?
Yes No No Yes No No Yes No No No
Single Married Single Married
Divorced 90K Single Married 40K 80K
Divorced 95K Married 60K
[Predictive] [Predictive]
Deviation Detection
Classification: Definition
Given a collection of records (training set )
Each record contains a set of attributes, one of the attributes is the class.
Enormity of data High dimensionality of data Heterogeneous, distributed nature of data
Statistics/ AI Machine Learning/ Pattern Recognition
Data Mining
Database systems
Divorced 220K Single Married Single 85K 75K 90K
No Yes No Yes
Test Set
Training Set
Learn Classifier
Model
Classification: Application 1
Direct Marketing
Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product. Approach: Use the data for a similar product introduced before. We know which customers decided to buy and which decided otherwise. This {buy, don’t buy} decision forms the class attribute. Collect various demographic, lifestyle, and companyinteraction related information about all such customers.
When does a customer buy, what does he buy, how often he pays on time, etc
Label past transactions as fraud or fair transactions. This forms the class attribute. Learn a model for the class of the transactions. Use this model to detect fraud by observing credit card transactions on an account.
Traditional techniques infeasible for raw data Data mining may help scientists
in classifying and segmenting data in Hypothesis Formation
Mining Large Data Sets - Motivation
Type of business, where they stay, how much they earn, etc.
Use this information as input attributes to learn a classifier model.
Classification: Application 2
Find a model for class attribute as a function of the values of other attributes. Goal: previously unseen records should be assigned a class as accurately as possible.
A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.
5
数据挖掘研究的促发因素
位置签到服务
实时发布给好友，使朋友间可以更快速地了解相 互间的状态。 商家了解哪些顾客是经常光顾自己的，以便给予 常客更多地优惠等。
带有地理标注的图片
即将图片与某些地理位置联系起来，若在查询某 个景点时能有一些相关的图片。 使用搜索和数据挖掘的技术给图片相关性 （relevance）打分，将得分高的结果返回给用 户。
2014年10月22日星期三
12
数据挖掘广义定义
对储存在数据库、数据仓库和其他各种信息 源的海量数据信息中隐含的有趣信息的发现 过程。 数据挖掘是知识发现过程的一个步骤
2014年10月22日星期三
13
Origins of Data Mining
Draws ideas from machine learning/AI, pattern recognition, statistics, and database systems Traditional Techniques may be unsuitable due to
Classification Example
Tid Refund Marital Status 1 2 3 4 5 6 7 8 9 10
10
Taxable Income Cheat 125K 100K 70K 120K No No No No Yes No
10
Refund Marital Status No Yes No Yes No No Single Married Married
There is often information “hidden” in the data that is not readily evident Human analysts may take weeks to discover useful information Much of the data is never analyzed at all 数据富裕，知识贫乏 大量数据背后隐藏着重要的知识 需要有效的数据挖掘工具支持
基于出租车数据的线路推荐系统
当用户查询路线时，系统根据目的地、目前时间等 信息搜索以往出租车的行驶路线，即最佳路线。
2014年10月22日星期三
6
Why Mine Data? Commercial Viewpoint
Lots of data is being collected and warehoused
Data Mining Tasks
Prediction Methods
Use some variables to predict unknown or future values of other variables.
Description Methods
Find human-interpretable patterns that describe the data.
2014年10月22日星期三
3
“小数据”时代
数据样本小，数据珍贵 对数据准确性要求很高，容错性差 基于归纳，发现大概率事件（一般性规律） 依赖逻辑，注重因果性
2014年10月22日星期三
4
“大数据”时代
数据样本大 对数据准确性要求低，宽容性好 发现小概率事件（特殊性规律） 依赖计算，注重关联性
2014年10月22日星期三
2014年10月22日星期三
2
参考书目
Jiawei Han and Micheline Kamber，数据挖掘— 概念与技术(原书第三版)，范明、孟小峰等译， 机械工业出版社 PangNing Tan,Michael Steinbach,Vipin Kumar, 数据挖掘导论，范明、范宏建等译，人 民邮电出版社 Ian H. Witten and Eibe Frank，数据挖掘—实用 机器学习技术(原书第二版)，董琳、邱泉等译， 机械工业出版社
remote sensors on a satellite telescopes scanning the skies microarrays generating gene expression data scientific simulations generating terabytes of data