Spark(6)Upgrade to 1.0.2 Version again with YARN
Download the prebuilt version
>wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2-bin-hadoop2.tgz
Setup and Make hadoop2 running on my VMs
http://sillycat.iteye.com/blog/2090186
Prepare the file on hadoop
>hdfs dfs -mkdir /user/sillycat
>hdfs dfs -put /opt/spark/log.txt /user/sillycat/
Login on the shell.
>MASTER=spark://ubuntu-master1:7077 bin/spark-shell
>val file = sc.textFile("hdfs://ubuntu-master1:9000/user/sillycat/log.txt")
>file.first()
Error Message:
Server IPC version 9 cannot communicate with client version 4
Solution:
Version error, I am using spark-hadoop1 to connect to hadoop 2.4.1
It works in the shell.
Go on and configure the YARN.
>sbin/start-dfs.sh
>sbin/start-yarn.sh
>sbin/mr-jobhistory-daemon.sh start historyserver
YARN is running now. Then we can get info from these URL
http://ubuntu-master1:50070/dfshealth.html#tab-overview
http://ubuntu-master1:8088/cluster/nodes
http://ubuntu-master1:19888/jobhistory
Running Spark Shell on YARN
Change the configuration file of spark
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
>MASTER=yarn-client bin/spark-shell
Submit the task as follow
>bin/spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master yarn /Users/carl/work/current/simplesparkapp/target/sparkwordcount-0.0.1-SNAPSHOT.jar hdfs://ubuntu-master1:9000/user/sillycat/log.txt 2
Spark works great on YARN
>bin/spark-submit --class com.sillycat.spark.app.ClusterComplexJob --master yarn /Users/carl/work/sillycat/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar book1
The sample project is in sillycat-spark
Even the standalone cluster is working
>bin/spark-submit --class com.sillycat.spark.app.FindWordJob --master spark://ubuntu-master1:7077 /Users/carl/work/sillycat/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar book1
The command to start the master and slave
>sbin/start-master.sh
>bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu-master1:7077
Configuration on master1
>cat conf/spark-env.sh
#!/usr/bin/env bash
export SPARK_LOCAL_IP=ubuntu-master1
#export SPARK_EXECUTOR_MEMORY=1G
export SPARK_MASTER_IP=ubuntu-master1
export SPARK_WORKER_MEMORY=1024M
Configuration on slave1
>cat conf/spark-env.sh
#!/usr/bin/env bash
export SPARK_LOCAL_IP=ubuntu-slave1
#export SPARK_EXECUTOR_MEMORY=1G
export SPARK_MASTER_IP=ubuntu-master1
export SPARK_WORKER_MEMORY=1024M
Tips
Spark Job runs
>bin/spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master local /Users/carl/work/current/simplesparkapp/target/sparkwordcount-0.0.1-SNAPSHOT.jar /opt/spark/README.md 2
Error Message
java.lang.OutOfMemoryError: Java heap space
Solution:
Change the Memory configuration from here
>vi bin/spark-class
References:
http://spark.apache.org/docs/latest/running-on-yarn.html
https://github.com/snowplow/spark-example-project
http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/
http://parambirs.wordpress.com/2014/05/20/running-spark-1-0-0-snapshot-on-hadoopyarn-2-4-0/
http://parambirs.wordpress.com/2014/05/20/install-hadoopyarn-2-4-0-on-ubuntu-virtualbox/
http://parambirs.wordpress.com/2014/05/20/building-and-running-spark-1-0-0-snapshot-on-ubuntu/
相关推荐
the firmware version begin with H1 can't upgrade to the firmware verson begin with H3 or V3.2.4.02 the firmware version begin with H3 can't upgrade to the firmware verson begin with H1 or V3.2.4.02 ...
46 upgrade to 60046 upgrade to 60046 upgrade to 60046 upgrade to 600
46upgrade to 60 0二46upgrade to 60 0二46upgrade to 60 0二
Using vihostupdate to upgrade ESXi 4.0 U1 to ESXi 4.1 . . 34 Step 6. Using Update Manager to upgrade ESX 4.0 U1 Host to ESX 4.1 .. . 41 Step 7. Importing Offline Patches . . . . . . . . . . . . . . ...
Please upgrade to the latest version. For documentation please go to: http://sparkjava.com/documentation For usage questions, please use stack overflow with the “spark-java” tag Javadoc: ...
why upgrade to project2013
This White Paper is dedicated to have a single source of information on general aspects related to upgrading from older V6 releases to 3DEXPERIENCE platform
NULL 博文链接:https://sap.iteye.com/blog/574564
Upgrade to SAP NetWeaver Process Integration 7.1.pdf
Upgrade_-_ECC_5.0_to_ECC_6.0
This short early adopter book details both how you'll upgrade existing web and other applications from Rails 4.2 to 5 and how to create new applications in Rails 5. The headline features, Rails API ...
Apache Hadoop YARN is the modern distributed operating system for big data applications....And you’ll leave with all the knowledge of how to upgrade painlessly from 2.x to 3.x to get all the benefits.
For example, we haven't really tested upgrade to this version from the latest stable. If you want to help out, please play with it, and report your findings in mantis , marking the issue for version ...
-On ST-Link V2, upgrade firmware to V2.J27.S6, -需要其他版本请关注我
The classA and classC sections are unchanged compared to LoRaWAN1.0.2 with the exception of a new MAC command "DeviceTimeRequest" used to synchronize the real time clock of a device. For devices ...
VB6 to VB.Net Upgrade Assessment Tool Setup
Upgrade, Migrate & Consolidate to Oracle Database 12c: Strategies, General Preparation Steps, Upgrade & Migration Cases; Fallback Strategies; New Features in Oracle 12c; Performance Management.
Upgrade to Oracle Database 12c 1Z0-060 exam认证题库
You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details....
upgrade SMS 2003 to SCCM 2007