基于Chrome浏览器插件的爬虫系统

学校代码：10255

学号：2131569 基于Chrome浏览器插件的爬虫系统

WEB CRAWLER SYSTEM BASED ON CHROME

EXTENSION

学科专业：软件工程

作者：魏少鹏

指导教师：夏小玲

答辩日期：2016年1月

2015年12月

东华大学学位论文原创性声明

本人郑重声明：我恪守学术道德，崇尚严谨学风。所呈交的学位论文，是本人在导师的指导下，独立进行研究工作所取得的成果。除文中已明确注明和引用的内容外，本论文不包含任何其他个人或集体已经发表或撰写过的作品及成果的内容。论文为本人亲自撰写，我对所写的内容负责，并完全意识到本声明的法律结果由本人承担。

学位论文作者签名：

日期：年月日

东华大学学位论文版权使用授权书

学位论文作者完全了解学校有关保留、使用学位论文的规定，同意学校保留并向国家有关部门或机构送交论文的复印件和电子版，允许论文被查阅或借阅。本人授权东华大学可以将本学位论文的全部或部分内容编入有关数据库进行检索，可以采用影印、缩印或扫描等复制手段保存和汇编本学位论文。

保密□，在年解密后适用本版权书。

本学位论文属于

不保密□。

学位论文作者签名：指导教师签名：

日期：年月日日期：年月日

基于Chrome浏览器插件的爬虫系统

摘要

随着大数据时代的来临，网络信息以“爆炸”的方式增长，以新浪微博为例，其日均微博更新多达1.2亿条，但在信息空前丰富的前提下，用户获取自己需要数据的难度也日趋增加。传统搜索引擎如百度、Google等提供的零散结果已经不能满足用户的需求，人们不论在专业数据分析还是在日常生活方面更需要的是经过有效整合的数据，整合互联网数据要使用网络爬虫技术。然而目前常用的网络爬虫系统开发难度大，稳定性差并且使用不友好，已经不能满足当今用户的需求，因此研究设计一种开发扩展简单，稳定性高，适用范围广以及使用友好的网络爬虫系统具有非常重要的价值。

本文先研究了国内外现有网络爬虫技术、网络爬虫系统和反爬虫策略的现状，分析了现有网络爬虫系统实现过程复杂、稳定性差和使用不友好的原因，并且基于此提出了基于Chrome扩展的网络爬虫系统。其次，为了满足不同的需求和发挥互联网的优势，提出了在基于Chrome扩展的网络爬虫系统中实现两种信息抓取模块，分别是个人版信息抓取模块和服务器版信息抓取模块。最后，为了支持个人版信息抓取模块对中央服务器模块的高并发性需求，提出了基于Netty框架实现的中央服务器模块和采用主从库配置的数据库模块，并且为了在有更多新需求的情况下中央服务器模块能够更好的进行扩展，本文采用了面向接口编程并且引入了Spring框架来管理中央服务器模块

的类之间的依赖。

本文设计和实现的网络爬虫系统，不仅开发过程容易，扩展方便，支持网页类型多——包括普通静态网页、异步加载网页和动态生成信息，而且个人版信息抓取模块还可以最大化的发挥互联网的优势，利用每个网络爬虫用户来抓取信息。系统测试结果表明预期结果得以实现，并且在使用友好度、高可用性方面有很大的提高。

关键字：网络爬虫Chrome扩展JavaScript Netty 主从库

WEB CRAWLER SYSTEM BASED ON CHROME

EXTENSION

ABSTRACT

With the advent of the era of big data, network information has been growing in a explosive speed, for instance, the daily blogs posted on Sina Weibo reach to 120 million. However, the difficulty for users to get the data they need has been increasing in the context of unprecedentedly rich information. Scattered result presented by traditional search engine, like Baidu and Google, has no longer fulfilled user s’requirements, what they need more is the data of valid integration in professional data analysis and daily life. Crawler is one of technologies used in the process of internet data integration. However, common crawler technology used currently has difficult exploitation and poor stability, and is not user friendly, which can not meet the needs of user. Therefore, it is valuable to develop a new crawler system with the feature of simple expanding development, high stability, wide application and user friendly.

This article firstly analyzed current crawler system, crawler technology and anti-crawler strategies used both in China and abroad, as well as the reason leading to the complex implementation process of web crawler system, poor stability and user unfriendly, as a result, a new crawler system based on Chrome was created. Furthermore, in order to fulfill different user needs and give play to advantages of internet, two kinds of information capture modules in web crawler system based on Chrome expansion were proposed, which were personal version information capture module extension and server version information capture module extension. Finally, in order to support the high concurrency requirements of central server module for personal version information capture module, central server module based on Netty framework and database module using Master-slave Database configuration, and in order to make central server module extend better as more requests are made, this article used program to interface and introduced spring framework to manage the dependencies between the central server module and category.

The crawler system designed and developed in the article has the features that easy development, extended to facilitate and supporting many webpage types, including static webpage, asynchronous loading webpage and dynamic webpage, and personal version information capture module can also maximize the advantage of internet to grab information using each web crawler user. As the result presented in the test environment, all the features presented above has been successfully