Spark(5)Upgrade the Spark to 1.0.2 Version
1. Upgrade the Version to 1.0.2
If plan to build from the source
>git clone https://github.com/apache/spark.git
check out the tag version
>git tag -l
It will list the tags
>git checkout v1.0.2
>git pull origin v1.0.2
>sbt/sbt -Dhadoop.version=2.2.0 -Pyarn assembly
>sbt/sbt -Dhadoop.version=2.2.0 -Pyarn publish-local
Or build the normal version
>sbt/sbt update
>sbt/sbt compile
>sbt/sbt assembly
But I download the binary version from official website and go on with my example.
Error Message
14/08/08 17:33:03 WARN scheduler.TaskSetManager: Loss was due to java.lang.NoClassDefFoundError java.lang.NoClassDefFoundError: Could not initialize class scala.Predef$
Bad type in putfield/putstatic
14/08/08 22:07:07 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,run-main-group-0] java.lang.VerifyError: Bad type on operand stack Exception Details: Location: scala/collection/IndexedSeq$.ReusableCBF$lzycompute()Lscala/collection/generic/GenTraversableFactory$GenericCanBuildFrom; @19: putfield Reason: Type 'scala/collection/IndexedSeq$$anon$1' (current frame, stack[1]) is not assignable to 'scala/collection/generic/GenTraversableFactory$GenericCanBuildFrom' Current Frame: bci: @19 flags: { } locals: { 'scala/collection/IndexedSeq$', 'scala/collection/IndexedSeq$' } stack: { 'scala/collection/IndexedSeq$', 'scala/collection/IndexedSeq$$anon$1' }
Solution:
https://spark.apache.org/docs/latest/tuning.html#data-serialization
joda time problem.
-Duser.timezone=UTC
new DateTime(DateTimeZone.forID("UTC"))
Object serializer
https://github.com/EsotericSoftware/kryo
Update the scala version to 2.10.4
2. Deployment
Standalone
Start Master
>vi conf/spark-env.sh
SPARK_MASTER_IP=localhost
SPARK_LOCAL_IP=localhost
>./sbin/start-master.sh
The main class org.apache.spark.deploy.master.Master
Web UI
http://localhost:8080/
Start Worker
>./bin/spark-class org.apache.spark.deploy.worker.Worker spark://localhost:7077
Error Message
java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolMessageEnum
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
Solution:
>wget http://central.maven.org/maven2/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar
>wget http://central.maven.org/maven2/org/spark-project/protobuf/protobuf-java/2.4.1-shaded/protobuf-java-2.4.1-shaded.jar
Check the log file, Error Message
>tail -f spark-root-org.apache.spark.deploy.master.Master-1-carl-macbook.local.out
Error Message:
14/08/09 03:48:45 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@192.168.11.11:7077] -> [akka.tcp://spark@192.168.11.11:62531]: Error [Association failed with [akka.tcp://spark@192.168.11.11:62531]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@192.168.11.11:62531] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.11.11:62531 ]
Solution:
After a while I switch to try to build it myself.
I directly go to System Conferences to add one user spark.
>ssh-keygen -t rsa
if needed
Find the public key
>cat /Users/carl/.ssh/id_rsa.pub
Seems not working. So I plan to do that on ubuntu VM machine.
>sudo adduser sparkWorker --force-badname
Checkout my host
>host ubuntu-master
Host ubuntu-master not found: 3(NXDOMAIN)
Check my running of spark
>netstat -at | grep 7077
tcp6 0 0 ubuntu-master:7077 [::]:* LISTEN
>bin/spark-submit --class com.sillycat.spark.app.ClusterComplexJob --master spark://192.168.11.12:7077 --total-executor-cores 1 /Users/carl/work/sillycat/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar
Turn off ipv6 on MAC
networksetup -listallnetworkservices | sed 1d | xargs -I {} networksetup -setv6off {}
All of these tries are not working.
Try the latest version from github, 1.1.0-SNAPSHOT.
The standalone cluster is still not working.
References:
http://spark.apache.org/docs/latest/spark-standalone.html
http://spark.apache.org/docs/latest/building-with-maven.html
https://github.com/mesos/spark.git
http://www.iteblog.com/archives/1038
http://www.iteblog.com/archives/1016
My Spark Blogs
http://sillycat.iteye.com/blog/1871204
http://sillycat.iteye.com/blog/1872478
http://sillycat.iteye.com/blog/2083193
http://sillycat.iteye.com/blog/2083194
ubuntu add/remove user
https://www.digitalocean.com/community/tutorials/how-to-add-and-delete-users-on-ubuntu-12-04-and-centos-6
https://spark.apache.org/docs/latest/submitting-applications.html
https://spark.apache.org/docs/latest/spark-standalone.html
disable ipv6
http://askubuntu.com/questions/309461/how-to-disable-ipv6-permanently
spark source code
http://www.cnblogs.com/hseagle/p/3673147.html
- 浏览: 2489403 次
- 性别:
- 来自: 成都
文章分类
最新评论
-
nation:
你好,在部署Mesos+Spark的运行环境时,出现一个现象, ...
Spark(4)Deal with Mesos -
sillycat:
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat:
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat:
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat:
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
发表评论
-
Update Site will come soon
2021-06-02 04:10 1613I am still keep notes my tech n ... -
NodeJS12 and Zlib
2020-04-01 07:44 433NodeJS12 and Zlib It works as ... -
Traefik 2020(1)Introduction and Installation
2020-03-29 13:52 295Traefik 2020(1)Introduction and ... -
Private Registry 2020(1)No auth in registry Nginx AUTH for UI
2020-03-18 00:56 378Private Registry 2020(1)No auth ... -
Buffer in NodeJS 12 and NodeJS 8
2020-02-25 06:43 337Buffer in NodeJS 12 and NodeJS ... -
NodeJS ENV Similar to JENV and PyENV
2020-02-25 05:14 419NodeJS ENV Similar to JENV and ... -
Prometheus HA 2020(3)AlertManager Cluster
2020-02-24 01:47 365Prometheus HA 2020(3)AlertManag ... -
Serverless with NodeJS and TencentCloud 2020(5)CRON and Settings
2020-02-24 01:46 294Serverless with NodeJS and Tenc ... -
GraphQL 2019(3)Connect to MySQL
2020-02-24 01:48 211GraphQL 2019(3)Connect to MySQL ... -
GraphQL 2019(2)GraphQL and Deploy to Tencent Cloud
2020-02-24 01:48 394GraphQL 2019(2)GraphQL and Depl ... -
GraphQL 2019(1)Apollo Basic
2020-02-19 01:36 278GraphQL 2019(1)Apollo Basic Cl ... -
Serverless with NodeJS and TencentCloud 2020(4)Multiple Handlers and Running wit
2020-02-19 01:19 266Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(3)Build Tree and Traverse Tree
2020-02-19 01:19 267Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(2)Trigger SCF in SCF
2020-02-19 01:18 255Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(1)Running with Component
2020-02-19 01:17 240Serverless with NodeJS and Tenc ... -
NodeJS MySQL Library and npmjs
2020-02-07 06:21 238NodeJS MySQL Library and npmjs ... -
Python Library 2019(1)requests and aiohttp
2019-12-18 01:12 228Python Library 2019(1)requests ... -
Hadoop Docker 2019 Version 3.2.1
2019-12-10 07:39 259Hadoop Docker 2019 Version 3.2. ... -
Nginx and Proxy 2019(1)Nginx Enable Lua and Parse JSON
2019-12-03 04:17 378Nginx and Proxy 2019(1)Nginx En ... -
Data Solution 2019(13)Docker Zeppelin Notebook and Memory Configuration
2019-11-09 07:15 243Data Solution 2019(13)Docker Ze ...
相关推荐
Please upgrade to the latest version. For documentation please go to: http://sparkjava.com/documentation For usage questions, please use stack overflow with the “spark-java” tag Javadoc: ...
Running the Pre-Upgrade Checker . . . . 6 Step 3. Migration to vCenter Server 4.1 . . . . . . 13 Step 4. Upgrading hosts to ESX(i) 4.1 . . . . . . . . 33 Step 5. Using vihostupdate to upgrade ESXi ...
46 upgrade to 60046 upgrade to 60046 upgrade to 60046 upgrade to 600
What are the new changes to Rails 5 and how to upgrade from Rails 4.2 How to use the new Ruby Persistence tool/framework, ActiveRecord How to handle test framework changes What are the latest Rails ...
the firmware version begin with H1 can't upgrade to the firmware verson begin with H3 or V3.2.4.02 the firmware version begin with H3 can't upgrade to the firmware verson begin with H1 or V3.2.4.02 ...
Upgrade, Migrate & Consolidate to Oracle Database 12c: Strategies, General Preparation Steps, Upgrade & Migration Cases; Fallback Strategies; New Features in Oracle 12c; Performance Management.
The next step is to consider your reasons for an upgrade. Different industries will have different motivations for migrating to the next version of SQL Server. These reasons can be grouped into the ...
Apache Hadoop YARN is the modern distributed operating system for big data applications....And you’ll leave with all the knowledge of how to upgrade painlessly from 2.x to 3.x to get all the benefits.
西门子_Checking Used System Calls (SFC) in STEP 7 Projects for the Upgrade to the new SIMATIC S7-300 CPpdf,
抓包神器, charles-proxy4.2 非常好用Charles Web Debugging Proxy - Official Site... The latest version of Charles is 4.2. Read the Release Notes. Paid Upgrade for Charles 3 to Charles 4 C...
upgrade SMS 2003 to SCCM 2007
**FLASH:** Upload the firmware file to the device and begin the upgrade/downgrade. **BACKUP:** Will make a flashable firmware backup of the currently connected device if applicable. It will save the ...
Then we talk about how to write tests for our app and how we can upgrade our Angular 1 apps to Angular 4+. Finally, we close with a chapter on writing native mobile apps with Angular using Native...
Upgrade_-_ECC_5.0_to_ECC_6.0
46upgrade to 60 0二46upgrade to 60 0二46upgrade to 60 0二
The classA and classC sections are unchanged compared to LoRaWAN1.0.2 with the exception of a new MAC command "DeviceTimeRequest" used to synchronize the real time clock of a device. For devices ...
DS-01276-CN_5_Reasons_to_Upgrade_to_SEP12.pdf
as the professional version, but you have to upgrade to the professional version after you graduate (but you get a discount on the upgrade also). There is an electronic copy of the book that is ...
Then, to upgrade your existing eLicense registration code, simply click the Upgrade button that appears on the 30-trial screen that you see when you first run this new version of zMUD. This new ...
Running BlogEngine.NET using SQL Server 2008 and up: If you wish to use SQL Server to store all your blog data, ...- Use scripts in Archive folder to bring your DB to version 3.1, then run Upgrade.sql