The Spark has 3 main components:

• The Master manages Workers.
• The Worker watting for the command from Master.
• The SparkSubmit provide the main gateway of launching a Spark application.

In the Spark Standalong Mode,

the master and worker communicate through akka.

The component of a Spark Worker looks like this:

In the begining, the worker register with master:

15/02/10 12:29:57 INFO Worker: Registered signal handlers for [TERM, HUP, INT]
15/02/10 12:29:57 WARN Utils: Your hostname, zhou resolves to a loopback address: 127.0.1.1; using 10.173.214.158 instead (on interface eth0)
15/02/10 12:29:57 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/02/10 12:29:57 INFO SecurityManager: Changing view acls to: jan
15/02/10 12:29:57 INFO SecurityManager: Changing modify acls to: jan
15/02/10 12:29:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jan); users with modify permissions: Set(jan)
15/02/10 12:29:58 INFO Slf4jLogger: Slf4jLogger started
15/02/10 12:29:58 INFO Remoting: Starting remoting
15/02/10 12:29:58 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@10.173.214.158:55939]
15/02/10 12:29:58 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkWorker@10.173.214.158:55939]
15/02/10 12:29:58 INFO Utils: Successfully started service 'sparkWorker' on port 55939.
15/02/10 12:29:58 INFO Worker: Starting Spark worker 10.173.214.158:55939 with 8 cores, 14.7 GB RAM
15/02/10 12:29:58 INFO Worker: Running Spark version 1.3.0-SNAPSHOT
15/02/10 12:29:58 INFO Worker: Spark home: /home/jan/projects/spark
15/02/10 12:29:58 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
15/02/10 12:29:58 INFO WorkerWebUI: Started WorkerWebUI at http://10.173.214.158:8081
15/02/10 12:29:58 INFO Worker: Connecting to master akka.tcp://sparkMaster@zhou:7077/user/Master...
15/02/10 12:29:58 INFO Worker: Successfully registered with master spark://zhou:7077

When a job launchs, the worker launch the executer and the coresponding driver¹:

15/02/10 12:30:05 INFO ExecutorRunner: Launch command: "java" "-cp" ":/home/jan/projects/spark/sbin/../conf:/home/jan/projects/spark/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar" "-XX:MaxPermSize=128m" "-Dspark.driver.port=35922" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://sparkDriver@10.173.214.158:35922/user/CoarseGrainedScheduler" "0" "10.173.214.158" "8" "app-20150210123005-0000" "akka.tcp://sparkWorker@10.173.214.158:55939/user/Worker"

The complete communication between worker and master are located in function

receiveWithLogging.

CommandUtils.scala

is the common code used by ‘DriverRunner’ and ‘ExecutorRunniner’,

including transfer the Driver/Executor jar(s) and start up the processes.

Each Driver/Executor run as an independent process.

Appendix: Latex Used To Generate the Image

documentclass[border=10pt,varwidth,convert]{standalone}
usepackage{tikz}
usetikzlibrary{calc, shapes, backgrounds}
usetikzlibrary{decorations.pathmorphing} % for snake lines
usetikzlibrary{matrix} % for block alignment
usetikzlibrary{arrows} % for arrow heads
usetikzlibrary{calc} % for manimula vtion of coordinates
usetikzlibrary{positioning,fit,calc}
usetikzlibrary{decorations.markings} % for vecArrow
usetikzlibrary{patterns}
pagecolor{olive!50!yellow!50!white}
begin{document}

begin{tikzpicture}[scale=0.75, every node/.style={transform shape}]
node[rectangle,align=center,minimum width=8cm,draw,rounded corners,text centered] (worker) {Worker};
node[below left=0.5cm and -3.5cm of worker,rectangle,align=center,minimum width=3.5cm,draw,rounded corners,text centered] (driverrunner) {DriverRunner};
node[right=1cm of driverrunner,rectangle,align=center,minimum width=3.5cm,draw,rounded corners,text centered] (executorrunner) {ExecutorRunner};
end{tikzpicture}