DevOps(5)Spark Deployment on VM
1. Old Environment
1.1 Jdk
java version "1.6.0_45"
Switch version on ubuntu system.
>sudo update-alternatives --config java
Set up ubuntu JAVA_HOME
>vi ~/.profile
export JAVA_HOME="/usr/lib/jvm/java-6-oracle"
Java Compile Version Problem
[warn] Error reading API from class file : java.lang.UnsupportedClassVersionError: com/digby/localpoint/auth/util/Base64$OutputStream : Unsupported major.minor version 51.0
>sudo update-alternatives --config java
>sudo update-alternatives --config javac
1.2 Cassandra
cassandra 1.2.13 version
> sudo mkdir -p /var/log/cassandra
> sudo chown -R carl /var/log/cassandra
carl is my username
> sudo mkdir -p /var/lib/cassandra
> sudo chown -R carl /var/lib/cassandra
Change the config if needed, start the cassandra single mode
> cassandra -f conf/cassandra.yaml
Test that from client
> cassandra-cli -host ubuntu-dev1 -port 9160
Setup the multiple nodes, Config changes
listen_address: ubuntu-dev1
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "ubuntu-dev1,ubuntu-dev2"
Change that on both nodes on ubuntu-dev1, ubuntu-dev2.
Start the 2 nodes in backend
> nohup cassandra -f conf/cassandra.yaml &
Verify that the cluster is working
> nodetool -h ubuntu-dev1 ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
7068820527558753619
10.190.191.195 rack1 Up Normal 132.34 KB 36.12% -4714763636920163240
==========
Address Rack Status State Load Owns Token
7068820527558753619
10.190.191.195 rack1 Up Normal 132.34 KB 36.12% -4714763636920163240
10.190.190.190 rack1 Up Normal 65.18 KB 63.88% 7068820527558753619
1.3 Spark
I am choosing this old version.
spark-0.9.0-incubating-bin-hadoop1.tgz
Place that in the right place.
Set up the access across among the masters and slaves.
On Master
> ssh-keygen -t rsa
> cat ~/.ssh/id_rsa.pub
On slave
> mkdir ~/.ssh
> vi ~/.ssh/authorized_keys
Put the public key from rsa.pub
Config the Spark file here /opt/spark/conf/spark-env.sh
SCALA_HOME=/opt/scala/scala-2.10.3
SPARK_WORKER_MEMORY=512m
#SPARK_CLASSPATH='/opt/localpoint-profiles-spark/*jar'
#SPARK_JAVA_OPTS="-Dbuild.env=lmm.sdprod"
SPARK_WORKER_MEMORY=512m
#SPARK_CLASSPATH='/opt/localpoint-profiles-spark/*jar'
#SPARK_JAVA_OPTS="-Dbuild.env=lmm.sdprod"
USER=carl
/opt/spark/conf/slaves
ubuntu-dev1
ubuntu-dev2
Command to start the Spark Server
>sbin/start-all.sh
Spark single mode Command
>java -Dbuild.env=sillycat.dev cp /opt/YOU_PROJECT/lib/*.jar com.sillycat.YOUR_CLASS
>java -Dbuild.env=sillycat.dev -Dsparkcontext.Master=“spark://YOURSERVER:7070” cp /opt/YOU_PROJECT/lib/*.jar com.sillycat.YOUR_CLASS
Visit the homepage for Spark Master
3. Prepare Mysql
>sudo apt-get install software-properties-common
>sudo add-apt-repository ppa:ondrej/mysql-5.6
>sudo apt-get update
>sudo apt-get update
>sudo apt-get install mysql-server
Command to create the database and set up the password
>use mysql;
>grant all privileges on test.* to root@"%" identified by 'kaishi';
>flush privileges;
on the client, maybe only install mysql client
>sudo apt-get install mysql-client-core-5.6
Change the bind address in sudo vi /etc/mysql/my.cnf
bind-address = 127.0.0.1
>sudo service mysql stop
>sudo service mysql start
4. Install Grails
Download from here, I am using an old version.
>wget
5. Install tomcat on Master
>wget
Config the database in this file, TOMCAT_HOME/conf/context.xml
<Resource name="jdbc/lmm" auth="Container" type="javax.sql.DataSource"
maxIdle="30" maxWait="-1" maxActive="100"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testOnBorrow="true"
validationQuery="select 1"
logAbandoned="true"
username="root"
password="kaishi"
driverClassName="com.mysql.jdbc.Driver"
maxIdle="30" maxWait="-1" maxActive="100"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testOnBorrow="true"
validationQuery="select 1"
logAbandoned="true"
username="root"
password="kaishi"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/lmm?autoReconnect=true&useServerPrepStmts=false&rewriteBatchedStatements=true"/>
Download and place the right mysql driver
> ls -l lib | grep mysql
-rw-r--r-- 1 carl carl 786484 Dec 10 09:30 mysql-connector-java-5.1.16.jar
Change the config to avoid OutOfMemoryError
> vi bin/catalina.sh
JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=512m"
6. Running Assembly Jar File
build the assembly jar and place in the lib directory, create a shell file in the bin directory
> cat bin/startup.sh
#!/bin/bash
#!/bin/bash
java -Xms512m -Xmx1024m -Dbuild.env=lmm.sparkvm -Dspray.can.server.request-timeout=300s -Dspray.can.server.idle-timeout=360s -cp /opt/YOUR_MODULE/lib/*.jar com.sillycat,YOUPACKAGE.YOUMAINLCASS
Setup the Bouncy Castle Jar
>cd /usr/lib/jvm/java-6-oracle/jre/lib/ext
>cd /usr/lib/jvm/java-6-oracle/jre/lib/security
>sudo vi java.security
security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider
7. JCE Problem
download file jce_policy-6.zip from http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html
Unzip the file and place the jar into this directory.
8. Command to Check data in cqlsh
Connect to cassandra
> cqlsh localhost 9160
Check the key space
cqlsh> select * from system.schema_keyspaces;
Check the version
cqlsh> show version
[cqlsh 3.1.8 | Cassandra 1.2.13 | CQL spec 3.0.0 | Thrift protocol 19.36.2]
Use the key space, something like database;
cqlsh> use device_lookup;
check the table
cqlsh:device_lookup> select count(*) from profile_devices limit 300000;
During testing, if need to clear the data
delete from profile_devices where deviceid = 'ios1009528' and brandcode = 'spark' and profileid = 5;
delete from profile_devices where brandcode = 'spark' and profileid = 5;
Deployment Option One
1 Put a serialize class there.
package com.sillycat.easyspark.profile
import com.sillycat.easyspark.model.Attributes import org.apache.spark.serializer.KryoRegistrator
import com.esotericsoftware.kryo.Kryo
import com.sillycat.easyspark.model.Profile
class ProfileKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) { kryo.register(classOf[Attributes])
kryo.register(classOf[Profile])
}
}
Change the configuration and start SparkContent part as follow:
val config = ConfigFactory.load() val conf = new SparkConf() conf.setMaster(config.getString("sparkcontext.Master")) conf.setAppName("Profile Device Update") conf.setSparkHome(config.getString("sparkcontext.Home")) if (config.hasPath("jobJar")) { conf.setJars(List(config.getString("jobJar"))) } else { conf.setJars(SparkContext.jarOfClass(this.getClass).toSeq)
}
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.kryo.registrator", “com.sillycat.easyspark.profile.ProfileKryoRegistrator")
val sc = new SparkContext(conf)
It works.
Tips
1. Command to Unzip the jar file
>jar xf jar-file
References:
cassandra
spark
ubuntu server
grails
bouncy castle
tomcat out of memory
Tips
Spark Trouble Shooting
相关推荐
Spark Summit West 2015的演讲PPT,databricks专家介绍Spark DevOps进阶课程。
This book tells you everything you need to know to help your organization implement DevOps on the Microsoft platform. You will learn how to use Visual Studio, Visual Studio Team Services, and Azure to...
Work through practical examples and gain DevOps best practices to successfully deploy applications on AWS Successfully provision and operate distributed application systems and your AWS infrastructure...
This book tells you everything you need to know to help your organization implement DevOps on the Microsoft platform. You will learn how to use Visual Studio, Visual Studio Team Services, and Azure to...
The first guideline is that all the examples will be tested on all major Kubernetes platforms. Well, that might be a bit far-fetched. I'm aware that any sentence that mentions "all" together with ...
DevOps 五大理念及其落地实践 研发运维一体化(DevOps)成熟度模型 中国DevOps现状调查报告及解读 构建企业DevOps的度量体系 DevOps实践指南精要 分布式敏捷和DevOps实践案例 AWS DevOps 助力爱乐奇大规模业务扩展 ...
DevOps is a practical field that focuses on delivering business value as efficiently as possible. DevOps encompasses all code workflows from testing environments to production environments. It ...
Title: Hands-On DevOps with Vagrant Author: Alex Braunton Length: 232 pages Edition: 1 Language: English Publisher: Packt Publishing Publication Date: 2018-10-17 ISBN-10: 1789138051 ISBN-13: ...
Design an application that is suitable for continuous deployment systems with Devops in mind Store and manage your code effectively using different options such as Git, Gerrit, and Gitlab Configure a ...
简述DEVOPS.ppt
Learn how DBAs in a DevOps environment manage data platforms and change requests to support and optimize continuous integration, delivery, testing, and deployment in the application development life ...
An exploration of continuous deployment to a Kubernetes cluster, using a wide range of Kubernetes platforms with instructions on how to develop a pipeline on a few of the most commonly used CI/CD ...
Setup continuous integration, delivery, and deployment pipelines for network functions Create test environments for network changes Understand how load balancing is becoming more software defined with...
devops主题系列演讲一共有14个部分,这是第5部分,每个部分里有10个大佬演讲的PPT,每个PPT都是独立的。 鞠炜刚-DevOps场景下云测试服务的架构及应用.pdf 刘文俊-大规模检索系统的DevOps实践分.pdf 刘亚丹-基于...
DevOps on the Microsoft Stack (The Expert Voice in .NET) by Wouter de Kort
非常实用的Devops学习资料,看了受益匪浅。值得一看!
Kantsev V. - Implementing DevOps on AWS - 2017 Kantsev V. - Implementing DevOps on AWS - 2017
高效DevOps
Chapter 5 DevOps Plays for Driving Innovation Chapter 6 Scaling DevOps for the Enterprise Chapter 7 Leading DevOps Adoption in the Enterprise Appendix Case Study: Example DevOps Adoption Roadmap
DevOps with Kubernetes and Helm DevOps with Kubernetes and Helm DevOps with Kubernetes and Helm