sillycat
- 浏览: 2527356 次
- 性别:
- 来自: 成都
-
最新评论
-
nation：
你好，在部署Mesos+Spark的运行环境时，出现一个现象， ...
Spark(4)Deal with Mesos -
sillycat：
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat：
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat：
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat：
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
文章列表
Zeppelin and Spark Library Dependency
I am using the third way to add the dependency as follow in spark interpreter
%spark.dep
z.load("com.sun.mail:javax.mail:1.6.2")
But I found that is always used, and every time it will say that
“Must be used before SparkInterpreter (%spark) initializ ...
Kafka Cluster 2019(5)WRK Performance Testing
A very simple script to support the POST using LUA
> cat post_message.lua
method="POST"
path = "/kafka/topic1"
batch = 10
request = function()
headers = {}
headers["Content-Type"] = "application/json"
...
Kafka Cluster 2019(4)Spring Boot Kafka Reactive
Found some articles and one samples here https://github.com/vanseverk/paymentprocessor-kafka-intro/tree/master/paymentprocessor-gateway
to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS
Add dependency supports to POM, pom.xml
< ...
Kafka Cluster 2019(3)Spring Boot Kafka
Support more complex object other than String
https://codenotfound.com/spring-kafka-apache-avro-serializer-deserializer-example.html
https://codenotfound.com/spring-kafka-json-serializer-deserializer-example.html
More Option to support in the YAML configurati ...
Kafka Cluster 2019(2)Kafka Cluster 2.12-2.2.0
Last time, I get the source codes and compile and package it myself. This time, I will just download the binary directly.
> wget http://mirror.cc.columbia.edu/pub/software/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz
Unzip and place it in the right direct ...
Zeppelin 2019(1)Alarm System monitor DynamoDB
First of all, I need shell interpreter there
%sh
cd ~/data
/opt/hadoop/bin/hdfs dfs -ls hdfs://localhost:9000/
CURRENT_DATE=`/bin/date +%Y-%m-%d`
DATA_PATH="monitor-$CURRENT_DATE"
echo $CURRENT_DATE
echo $DATA_PATH
Then I need to enable cron ...
Kafka Cluster 2019(1)ZooKeeper Cluster 3.5.5
Set up 2*N+1 Cluster for ZooKeeper first.
Check JAVA Version
> java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
Find the latest ver ...
Redis Cluster 2019(2)Reactive Connector Template in Spring Boot
Start 3 masters and 3 replicas on 2 Node Machines.
Start monitor on one machines
> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003 --server=7000 --daemon
Visit ...
Redis Cluster 2019(1)
Install the latest version 5.0.5 currently
> wget http://download.redis.io/releases/redis-5.0.5.tar.gz
Unzip and go to that directory
> make distclean
> make
Prepare the installation directory
> mkdir ~/tool/redis-5.0.5
> mkdir ~/tool/redis-5.0.5/bin
> cp s ...
Apache Cassandra 2019 and WebFlux and WebMVC
Create the Table in Cassandra
> CREATE KEYSPACE jobsmonitor WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
> use jobsmonitor;
> CREATE TABLE jobcounthistory(
... source_id text,
... ...
Apache Cassandra 2019 and Zeppelin
Open the interpreter and configure the key configurations.
cassandra.cluster=SillycatCluster
cassandra.hosts=ubuntu-master
cassandra.keyspace=patient
cassandra.native.port=9042
After that, in my notebook, I can do
%cassandra
use patient;
select * from exam where ...
Spark Zeppelin and Elasticsearch
Open Zeppelin and select ‘Interpreter’
Visit this URL to get more information about my Elastisearch cluster
http://ubuntu-master:9200/
Configure these informations
elasticsearch.client.type = transport
elasticsearch.cluster.name = sillycatcluster
elasticsearch.host ...
Spark Zeppelin and Elasticsearch
Open Zeppelin and select ‘Interpreter’
Visit this URL to get more information about my Elastisearch cluster
http://ubuntu-master:9200/
Configure these informations
elasticsearch.client.type = transport
elasticsearch.cluster.name = sillycatcluster
elasticsearch.host ...
Apache Cassandra 2019 Cluster
Haha, it looks like continues hash to form the cluster and replica the data in nodes.
Try to Set Up Cassandra Manually
> wget http://mirrors.ibiblio.org/apache/cassandra/3.11.4/apache-cassandra-3.11.4-bin.tar.gz
Unzip and place in the working directory
> sudo ln ...
Nginx and Cookie Access Log
Add the following configure in Nginx.conf
log_format cookie_debug '[$time_local] - $remote_addr:$remote_port - $remote_user - $upstream_addr $upstream_status $upstream_response_time - '
'"$request" $status $bytes_sent $request_time '
'"$http_referer" ...