Glossary Spark专业术语解读

Term Meaning
Application 应用程序User program built on Spark. Consists of a driver program and executors on the cluster.
Spark上建立的用户程序：在集群中由一个驱动程序和执行者组成
Application jar 应用程序Jar包A jar containing the user's Spark application. In some cases users will want to create an "uber jar" containing their application along with its
dependencies. The user's jar should never include Hadoop or Spark libraries, however, these will be added at runtime.
一个jar包包含用户的Spark应用。

在某些情况下，用户想要创建出一个“uber jar”包，这个jar包包含了他们的应用及其依赖项。

用户jar包不能包含Hadoop或者Spark的类库，然而，这些类库将会在运行时进行添加。

Driver program 驱动程序The process running the main() function of the application and creating the SparkContext
这个程序运行在应用的主方法，并且创建出SparkContext
Cluster manager 集群管理器An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)
在集群中获取资源的外部服务
Deploy mode 部署模式Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.
区别驱动程序运行在什么地方。

在cluster模式中，框架在集群内部启动驱动。

在client模式中，提交者在集群外启动驱动。

Worker node 工作节点Any node that can run application code in the cluster 在集群中，任何一台可以运行应用程序代码的节点。

Executor 执行者A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.
一个程序在一台工作节点上启动一个应用程序，运行任务并且通过他们将数据保存到内存或者进行磁盘存储。

每一个应用程序都有他自己的执行者。

Task 任务A unit of work that will be sent to one executor 一个被发送给一个执行器的工作单元
Job 工作A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect); you'll see this term used in the driver's logs.
包含多个任务的并行计算，经过了Spark action的相应，在驱动日志中会看到这些信息。

Stage 阶段Each job gets divided into smaller sets of tasks called stages that depend on each other (similar to the map and reduce stages in MapReduce); you'll see this term used in the driver's logs.
每一个工作被分成较小的阶段的集合，这些阶段相互依赖（类似于MapReduce的map与reduce的阶段），在驱动日志中也会看到这些信息。