Spark(4)Deal with Mesos
5. Running Spark with Mesos
I am an old developer, I already have Java, Scala, and Spark on my local machine.
>java -version
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-456-11M4508)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-456, mixed mode)
>scala -version
Scala code runner version 2.10.1 -- Copyright 2002-2013, LAMP/EPFL
spark is built based on the branch scala-2.10
>git status
# On branch scala-2.10
>git remote -v
origin https://github.com/mesos/spark.git (fetch)
origin https://github.com/mesos/spark.git (push)
5.1 Mesos
The version for Mesos right now is 0.12.0. But the version which in spark website is based on 0.9.0. I decide to use Mesos with the latest version following the document with 0.9.0. Haha.
Download mesos-0.12.0-incubating.tar.gz
Unzip this file and place in my working directory. And follow the README file.
>./configure
Error Message:
conftest.cpp:7: warning: 'JNI_CreateJavaVM' is deprecated (declared at /System/Library/Frameworks/JavaVM.framework/Headers/jni.h:1937)
configure: error: failed to build with JNI
Solution:
I do not quite understand this, but I use this command instead
>./configure --disable-java
>make
>make check
>sudo make install
Both make check failed, but it seems that make install success. But I saw nothing under /usr/local/mesos. Let do it again.
>mkdir build
>cd build
>../configure --with-python-headers=/usr/include/python2.6 --with-java-home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home --with-java-headers=/System/Library/Frameworks/JavaVM.framework/Headers --with-webui --with-included-zookeeper --prefix=/User/carl/tool/mesos-0.12.0
Headers is here
/System/Library/Frameworks/JavaVM.framework/Headers
JAVA_HOME is here
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>../configure --with-python-headers=/usr/include/python2.6 --with-java-home=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home --with-java-headers=/System/Library/Frameworks/JavaVM.framework/Headers --with-webui --with-included-zookeeper --prefix=/Users/carl/tool/mesos-0.12.0 --disable-java
>make
>make install
It seems there /User/carl/tool/mesos-0.12.0.
Start mesos to verify my Installation.
Start the Master
>sbin/mesos-master.sh
Start the Slave
>sbin/mesos-slave.sh --master=127.0.0.1:5050
Watching it from the webUI
http://localhost:5050/
5.2 Start Mesos in Cluster mode
On the master node
List all the masters and slaves in the configuration files
>cd /Users/carl/tool/mesos-0.12.0/var/mesos/deploy
>cat masters
localhost
>cat slaves
localhost
On the slave node
>cd /Users/carl/tool/mesos-0.12.0/var/mesos/conf
>cp mesos.conf.template mesos.conf
>cat mesos.conf
master=localhost:5050
Start the server
>sbin/mesos-start-cluster.sh
It works well.
Stop the server
>sbin/mesos-stop-cluster.sh
5.3 Spark On Mesos
Changes in the spark configuration
>cd /opt/spark/conf
MESOS_NATIVE_LIBRARY=/Users/carl/tool/mesos-0.12.0/lib/libmesos.dylib
SCALA_HOME=/opt/scala2.10.0
Change the spark Context as follow:
var sparkMaster = "mesos://localhost:5050"
Run the spark job. But I get error Message as follow:
Error Message:
Failed to load native Mesos library from .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
[error] (run-main) java.lang.UnsatisfiedLinkError: no mesos in java.library.path
java.lang.UnsatisfiedLinkError: no mesos in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
Solution:
Haha, it is easy to understand that I am using spark ----> scala ---> Java, but I --disable-java during the configure step. That is why I get pain here.
Take a look at these documents, It figure out a way to update the JDK to 1.7.
https://github.com/airbnb/chronos/blob/master/docs/FAQ.md
http://qnalist.com/questions/2020253/problem-building-on-mac-os
http://docs.oracle.com/javase/7/docs/webnotes/install/mac/mac-jdk.html
I download the jdk from oracle jdk-7u25-macosx-x64.dmg
>java -version
java version "1.7.0_25"
>export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)
And here is the JAVA_HOME
>echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home
Configure the mesos like this
>../configure --with-python-headers=/usr/include/python2.6 --with-java-home=/System/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home --with-java-headers=/System/Library/Frameworks/JavaVM.framework/Headers --with-webui --with-included-zookeeper --prefix=/Users/carl/tool/mesos-0.12.0
Good, it is working.
It seems that use Mesos latest version is a bad idea. I will try to use 0.9.0-inclubating instead.
Get the file from here: http://download.nextag.com/apache/incubator/mesos/mesos-0.9.0-incubating/
>../configure --with-python-headers=/usr/include/python2.6 --with-java-home=/System/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home --with-java-headers=/System/Library/Frameworks/JavaVM.framework/Headers --with-webui --with-included-zookeeper --prefix=/Users/carl/tool/mesos-0.9.0
>make
>make install
>sbin/mesos-master
Then the webui is from URL http://localhost:8080
>sbin/mesos-slave --master=127.0.0.1:5050
Then the webui is from URL http://localhost:8081
Still I got the Error Message
Failed to load native Mesos library from /Users/carl/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
[error] (run-main) java.lang.UnsatisfiedLinkError: no mesos in java.library.path
java.lang.UnsatisfiedLinkError: no mesos in java.library.path
Solution:
/Library/Java/Extensions
/System/Library/Java/Extensions
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home
Try to do the soft link to Mesos.
>sudo ln -s /Users/carl/tool/mesos-0.9.0/lib/libmesos.dylib /Library/Java/Extensions/libmesos.dylib
That may not be caused by the version of the Mesos, but I should soft link the dylib to java system.
>sudo ln -s /Users/carl/tool/mesos-0.12.0/lib/libmesos.dylib /Library/Java/Extensions/libmesos.dylib
It is working, and we can see the logs from the webui. But I got some error message like this>
Error Message:
13/07/19 16:38:43 ERROR executor.Executor: Exception in task ID 1 java.io.OptionalDataException at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1368) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at scala.collection.immutable.$colon$colon.readObject(List.scala:366) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:96) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:100) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:135)
Solution:
Information about the Mesos version, we have Mesos in
SPARK_HOME/project/SparkBuild.scala
SPARK_HOME/pom.xml
But I found this is working
>./run spark.examples.SparkPi mesos://localhost:5050
I did not rebuild the spark based on Mesos 0.12.0. The root reason to cause the first Error Message is that, the scala version of my command line is 2.10.1. I switch to 2.10.0. It is ok now.
But I notice that there is still second Error Message there:
13/07/19 17:21:58 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts akka.pattern.AskTimeoutException: Timed out at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:312) at akka.actor.DefaultScheduler$$anon$8.run(Scheduler.scala:191) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:137) at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:506) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) 13/07/19 17:22:11 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 2 attempts java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:96) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:100) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
Solution:
I just try this way, update the Mesos version in pom.xml and project/SparkBuild.scala and Update the Spark source codes and build spark again.
Error Message
Unexpected NOT to have spark.hostPort set
Solution:
Try to rollback to 0.9.0 version of Mesos and recompile spark. The error message still exists, it should come from the spark source codes. So I may be build and recompile it several days later.
6. Configure Mesos working with Zookeeper
come soon...
References:
Running Configuration
http://spark-project.org/docs/latest/running-on-yarn.html
http://spark-project.org/docs/latest/running-on-mesos.html
http://spark-project.org/docs/latest/spark-standalone.html
mesos
http://jm.taobao.org/2012/06/13/spark%E7%B3%BB%E5%88%97-%E5%AE%89%E8%A3%85/
http://spark-project.org/docs/latest/running-on-mesos.html
spark
http://sillycat.iteye.com/blog/1871204
mesos
http://mesos.apache.org/
https://github.com/apache/incubator-mesos/blob/trunk/docs/Home.md
https://github.com/apache/incubator-mesos/blob/79cbe52160cf58d58332e0e0ec609fd080bea151/docs/Configuration.textile
https://github.com/apache/incubator-mesos/blob/79cbe52160cf58d58332e0e0ec609fd080bea151/docs/Deploy-Scripts.textile
https://github.com/apache/incubator-mesos/blob/trunk/docs/Home.md
http://comments.gmane.org/gmane.comp.lang.scala.spark.user/1728
- 浏览: 2486808 次
- 性别:
- 来自: 成都
文章分类
最新评论
-
nation:
你好,在部署Mesos+Spark的运行环境时,出现一个现象, ...
Spark(4)Deal with Mesos -
sillycat:
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat:
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat:
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat:
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
评论
1 楼
nation
2017-07-11
你好,在部署Mesos+Spark的运行环境时,出现一个现象,如果集群中防火墙关闭了,则任务正常执行,如果打开防火墙,任务就会卡死,但是网上查找资料并没有看到要设置具体的防火墙端口,请问你有没有相关的配置经验,可否告知具体要开发的防火墙端口,或可以查到的文档地址,谢谢
发表评论
-
Update Site will come soon
2021-06-02 04:10 1609I am still keep notes my tech n ... -
Hadoop Docker 2019 Version 3.2.1
2019-12-10 07:39 256Hadoop Docker 2019 Version 3.2. ... -
Nginx and Proxy 2019(1)Nginx Enable Lua and Parse JSON
2019-12-03 04:17 376Nginx and Proxy 2019(1)Nginx En ... -
Data Solution 2019(13)Docker Zeppelin Notebook and Memory Configuration
2019-11-09 07:15 239Data Solution 2019(13)Docker Ze ... -
Data Solution 2019(10)Spark Cluster Solution with Zeppelin
2019-10-29 08:37 206Data Solution 2019(10)Spark Clu ... -
AMAZON Kinesis Firehose 2019(1)Firehose Buffer to S3
2019-10-01 10:15 286AMAZON Kinesis Firehose 2019(1) ... -
Rancher and k8s 2019(3)Clean Installation on CentOS7
2019-09-19 23:25 262Rancher and k8s 2019(3)Clean In ... -
Pacemaker 2019(1)Introduction and Installation on CentOS7
2019-09-11 05:48 299Pacemaker 2019(1)Introduction a ... -
Crontab-UI installation and Introduction
2019-08-30 05:54 409Crontab-UI installation and Int ... -
Spiderkeeper 2019(1)Installation and Introduction
2019-08-29 06:49 456Spiderkeeper 2019(1)Installatio ... -
Supervisor 2019(2)Ubuntu and Multiple Services
2019-08-19 10:53 321Supervisor 2019(2)Ubuntu and Mu ... -
Supervisor 2019(1)CentOS 7
2019-08-19 09:33 286Supervisor 2019(1)CentOS 7 Ins ... -
Redis Cluster 2019(3)Redis Cluster on CentOS
2019-08-17 04:07 334Redis Cluster 2019(3)Redis Clus ... -
Amazon Lambda and Version Limit
2019-08-02 01:42 388Amazon Lambda and Version Limit ... -
MySQL HA Solution 2019(1)Master Slave on MySQL 5.7
2019-07-27 22:26 413MySQL HA Solution 2019(1)Master ... -
RabbitMQ Cluster 2019(2)Cluster HA and Proxy
2019-07-11 12:41 418RabbitMQ Cluster 2019(2)Cluster ... -
Running Zeppelin with Nginx Authentication
2019-05-25 21:35 289Running Zeppelin with Nginx Aut ... -
Running Zeppelin with Nginx Authentication
2019-05-25 21:34 297Running Zeppelin with Nginx Aut ... -
ElasticSearch(3)Version Upgrade and Cluster
2019-05-20 05:00 293ElasticSearch(3)Version Upgrade ... -
Jetty Server and Cookie Domain Name
2019-04-28 23:59 362Jetty Server and Cookie Domain ...
相关推荐
金币火花 金币的火花。 用法 Docker 镜像由 spark 版本标记。 Mesos 的主 URL 格式为 mesos://host:5050(对于单主 Mesos 集群)或 mesos://zk://host:... --master <mesos> --conf spark.executor.uri=<spark d
Deploy Apache Mesos to concurrently run cutting edge data processing frameworks like Spark, Hadoop and Storm in parallel Share resources between various cluster computing applications and web ...
During 2014, 2015, and 2016, surveys show that among all software developers, those with higher wages are the data engineers, the data scientists, and the data architects. This is because there is a ...
We will show a demo for how to launch a Mesos cluster and then deploy Spark. Then we’ll show a sample app based on PMML for importing predictive models from analytics frameworks such as R, SAS, SPSS,...
Big Data SMACK: A Guide to Apache Spark, Mesos, Akka, Cassandra, and Kafka by Raul Estrada, Isaac Ruiz English | ISBN: 1484221745 | 2016 | EPUB | 264 pages | 2.35 MB This book is about how to ...
Deploy Spark jobs to various clusters such as Mesos, EC2, Chef, YARN, EMR, and so on Use Shark's SQL query-like syntax with Spark In Detail Spark is a framework for writing fast, distributed programs...
Integrate Mesos with Spark and other big data frameworks Use networking features in Mesos for effective communication between containers Configure Mesos for high availability using Zookeeper Secure ...
With Mesos, you have the power to manage a wide range of resources in a multi-tenant environment. Starting with the basics, this book will give you an insight into all the features that Mesos has to...
mesos-spark-源码.rar
2.5Spark-on-Mesos模式配置 13 2.6Hive-on-Spark配置 13 第三章 Spark计算模型 15 1 RDD编程 15 1.1弹性分布式数据集RDD 15 1.2构建RDD对象 15 2RDD操作 15 2.1将函数传递给Spark 16 2.2了解闭包 16 2.3Pair RDD模型...
摘要:在今天的文章中,我们将着重探讨如何利用SMACK(即Spark、Mesos、Akka、Cassandra以及Kafka)堆栈构建可扩展数据处理平台。虽然这套堆栈仅由数个简单部分组成,但其能够实现大量不同系统设计。除了纯粹的批量...
Mesos Spark Shell 用于在Mesos集群上创建Spark 1.4.1 shell的docker镜像。 目前,通过.deb安装程序支持Mesos 0.23.0。跑步要查看如何运行Docker映像,请查看run_spark_shell.sh文件。 基本上,您只需要用Mesos ...
Deploy Spark jobs to various clusters such as Mesos, EC2, Chef, YARN, EMR, and so on This is a step-by-step tutorial that unleashes the power of Spark and its latest features Who This Book Is For Fast...
根据图识,项目将分为三个部分。...项目系统主要包括前端+后端+机器学习,前端采用React Native,Native,后端采用Dubbo+Spring+java,机器学习采用Spark进行实现,本项目机器学习-spark代码运行在mesos上。
1. Spark是特性 高可伸缩性 高容错 于内存计算 2. Spark的生态体系(BDAS,中文:伯利克分析栈) MapReduce属于Hadoop生态体系之一,Spark则属于BDAS生态体系之一 ... Spark on Mesos (Spark在Mesos)
Spark的单机版便于测试,同时本章也会提到通过SSH用Spark的内置部署脚本搭建Spark集群,使用Mesos、Yarn或者Chef来部署Spark。对于Spark在云环境中的部署,本章将介绍在EC2(基本环境和EC2MR)上的部署。如果你的...
Coverage includes Spark SQL, Tachyon, Kerberos, ML Lib, YARN, and Mesos, with clear, actionable guidance on resource scheduling, db connectors, streaming, security, and much more. Spark has become ...
Realize how to deploy Spark with YARN, MESOS or a Stand-alone cluster manager. Learn the concepts of Spark SQL, SchemaRDD, Caching, Spark UDFs and working with Hive and Parquet file formats Understand...