Prediction(5)Cluster Trouble Shooting
Face to some issue on local zeppelin with spark 1.5.0, zeppelin 0.6.0, hadoop 2.7.1.
It may be memory issue. But it does not help I configure the zeppelin as follow:
export MASTER="yarn-client"
export HADOOP_CONF_DIR="/opt/hadoop/etc/hadoop/"
export SPARK_HOME="/opt/spark"
. ${SPARK_HOME}/conf/spark-env.sh
export ZEPPELIN_CLASSPATH="${SPARK_CLASSPATH}"
export ZEPPELIN_JAVA_OPTS="-Dspark.yarn.driver.memoryOverhead=512 -Dspark.yarn.executor.memoryOverhead=512 -Dspark.akka.frameSize=100 -Dspark.executor.instances=2 -Dspark.driver.memory=3g -Dspark.storage.memoryFraction=0.7 -Dspark.core.connection.ack.wait.timeout=800 -Dspark.rdd.compress=true -Dspark.default.parallelism=18 -Dspark.executor.memory=3g"
Spark as follow:
export SPARK_DAEMON_JAVA_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70"
export HADOOP_CONF_DIR="/opt/hadoop/etc/hadoop"
#export SPARK_WORKER_MEMORY=1024m
#export SPARK_JAVA_OPTS="-Dbuild.env=lmm.sparkvm"
export USER=carl
Install phantomjs
> sudo apt-get install phantomjs
Build Zeppelin Again
> mvn clean package -Pspark-1.5 -Dspark.version=1.5.0 -Dhadoop.version=2.7.1 -Phadoop-2.6 -Pyarn -DskipTests
I just set up ubuntu-pilot to run the zeppelin and spark. ubuntu-master and ubuntu-dev1, ubuntu-dev2 will be the yarn cluster. Everything works fine now.
References:
http://machinelearningmastery.com/non-linear-classification-in-r-with-decision-trees/
分享到:
相关推荐
链路预测程序,Python源代码(link prediction)
Many compilers rely on branch prediction to improve program performance by identifying frequently executed regions and by aiding in scheduling instructions. Profile-based predictors require a time-...
Prioritization to Prediction Vol5.pdf
指标 EMA Prediction。
about prediction model
This thesis presents a scalable, generic methodology for microbial phenotype prediction based on supervised machine learning, several models for biological and ecological traits of high relevance, and...
ad-click-prediction
Linear prediction coding
The Trajectory Prediction Challenge of Apolloscapes Dataset: trajectory prediction_train.zip
CNN-Prediction-Based-Reversible-Data-Hiding-main CNN-Prediction-Based-Reversible-Data-Hiding-main CNN-Prediction-Based-Reversible-Data-Hiding-main CNN-Prediction-Based-Reversible-Data-Hiding-main
这款 Exp_EMA_Prediction EA 基于 EMA_Prediction 信号量指标生成的信号。
cnn-bilstm-attention-time-series-prediction_keras-mastercnn-bilstm-attention-time-series-prediction_keras-mastercnn-bilstm-attention-time-series-prediction_keras-mastercnn-bilstm-attention-time-series...
Prediction_RBF_nan.zip
Time-Series Prediction and Applications.
Application of improved Grey prediction model for settlement prediction of roadbed,谢正文,胡汉华,An improved Grey-based settlement predictor is promoted. Adopt Grey prediction as a forecasting ...
Epilepsy Prediction_Epilepsy_prediction
and prediction. Its ability to extract features from a large set of raw data without relying on prior knowledge of predictors makes deep learning potentially attractive for stock market prediction at ...
EMA_Prediction_HTF_Signal 指标基于 EMA_Prediction 指标通过彩色趋势指示的图形对象显示趋势的方向, 并可以通过声音信号提供提醒并给您的智能手机推送通知.
Personalized Click Prediction in Sponsored Search 计算广告