- 浏览: 2486282 次
- 性别:
- 来自: 成都
文章分类
最新评论
-
nation:
你好,在部署Mesos+Spark的运行环境时,出现一个现象, ...
Spark(4)Deal with Mesos -
sillycat:
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat:
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat:
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat:
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
AMAZON ECS(2)Disk Size on EC2
I have a project which have a large file there, it is a known issue, we will fix that later.
I want to docker that API, I did that on EC2 instance. I create that instance long time ago, I was thinking that I only need 30G, but today. I met this problem saying not enough space for build the docker image.
Error Message:
Error response from daemon: Cannot destroy container clever_bartik: Driver devicemapper failed to remove root filesystem ef2f1e5b38fddaac78966c5e828d5a0aabd92600c3d87346f9e371bfeb5401d2: Error saving transaction metadata: Error writing metadata to /var/lib/docker/devicemapper/metadata/.tmp258670910: write /var/lib/docker/devicemapper/metadata/.tmp258670910: no space left on device
Solution:
First of all, clean the file first.
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 30G 30G 0 100% /
devtmpfs 7.5G 100K 7.5G 1% /dev
tmpfs 7.5G 0 7.5G 0% /dev/shm
This command will show me that all the space of disk is used.
sudo du -hsx * | sort -rh | head -10
12G home
1.2G usr
526M var
139M lib
62M boot
43M opt
23M lib64
13M sbin
7.6M etc
7.0M bin
This command will show me where did the space goes. I will then delete the files. Restart the instance.
And then I will do 2 things.
1 Add Disk Space to my Small Lonely EC2
click —> [ELASTIC BLOCK STORE] —>[Volumes] —>”Create Volume” Attach this Volume to My EC2
After create and attach the disk to that machine, that does not mean we can directly use it on linux.
List the Disk
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
└─xvda1 202:1 0 30G 0 part /
xvdf 202:80 0 100G 0 disk
loop0 7:0 0 100G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm
I have not file system on that.
> sudo file -s /dev/xvdf
/dev/xvdf: data
Create the file system
> sudo mkfs -t ext4 /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: baca4bba-b1ea-419c-b3ba-39b0a117f6bb
Create the Directory
> sudo mkdir data
Mount the disk to the directory
> sudo mount /dev/xvdf /data
I did not try auto mount.
2 Set up and Configure Docker to use the ext storage
Check the current docker info
> sudo docker info
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
The problem is here. They are using up 30GB.
Stop Docker
> sudo service docker stop
Clean the old file
> sudo rm -fr /var/lib/docker/
Config the storage
> sudo cat /etc/sysconfig/docker-storage
# This file may be automatically generated by an installation program.
# By default, Docker uses a loopback-mounted sparse file in
# /var/lib/docker. The loopback makes it slower, and there are some
# restrictive defaults, such as 100GB max storage.
# If your installation did not set a custom storage for Docker, you
# may do it below.
# Example: Use a custom pair of raw logical volumes (one for metadata,
# one for data).
# DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/dev/myvg/my-docker-metadata --storage-opt dm.datadev=/dev/myvg/my-docker-data"
DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/data/my-docker-metadata --storage-opt dm.datadev=/data/my-docker-data"
Start docker there
> sudo service docker start
This does not work.
Try this
> sudo vi /etc/sysconfig/docker-storage-setup
DEVS="/dev/xvdf"
STORAGE_DRIVER=devicemapper
> /usr/bin
> sudo wget https://raw.githubusercontent.com/projectatomic/docker-storage-setup/master/docker-storage-setup.sh
> sudo chmod a+x docker-storage-setup.sh
> ./docker-storage-setup.sh
It seems not working. I do not want to spend more time on this. I will do like this.
Soft the link the file to the configuration
> sudo ln -s /data/docker /var/lib/docker
> sudo mkdir -p devicemapper/devicemapper/data
> sudo mkdir -p devicemapper/devicemapper/metadata
> sudo mkdir /var/lib/docker/graph
> sudo mkdir /var/lib/docker/tmp
> sudo mkdir /var/lib/docker/devicemapper/metadata
>sudo mkdir /var/lib/docker/containers
>sudo mkdir /var/lib/docker/init
>sudo mkdir /var/lib/docker/trust
>sudo mkdir /var/lib/docker/volumes
Error Message:
open /var/lib/docker/repositories-devicemapper: no such file or directory
Solution:
> sudo vi /var/lib/docker/repositories-devicemapper
{"Repositories":{}}
It does not work. I guess it relates to the manual deleting docker.
Remote the docker
> sudo yum remove docker
Install the docker again
> wget -qO- https://get.docker.com/ | sh
Let me try this:
> sudo mkdir /var/lib/docker
> sudo mount /dev/xvdf /var/lib/docker
It works perfect. Here is the Dockerfile
#Run a Simple REST API based on playframework
#Prepre the OS
FROM centos:7
MAINTAINER Carl Luo <luohuazju@gmail.com>
ENV DEBIAN_FRONTEND noninteractive
ENV JVM_HEAP_MAX 12g
ENV JVM_HEAP_MIN 8g
ENV HTTP_PORT 8003
#Install Java
RUN yum install -y java-1.8.0-openjdk-headless-1.8.0.60-2.b27.el7_1 lapack atlas
RUN yum install -y unzip
RUN yum install -y wget
#Install the Application
RUN mkdir /share/
WORKDIR /share/
#ADD data/classifier_models.tar.gz /share/
RUN wget https://xxxxx/classifier_models.tar.gz
RUN tar zxvf classifier_models.tar.gz
RUN rm -fr classifier_models.tar.gz
ADD target/universal/classifier-play-1.0.zip /share/
RUN unzip classifier-play-1.0.zip
RUN rm -fr classifier-play-1.0.zip
#Start the Application
EXPOSE 8003
RUN mkdir -p /app/
ADD start.sh /app/
WORKDIR /app
CMD [ "./start.sh" ]
Here is the start.sh
#!/bin/sh -ex
APPLICATION_SECRET="nosessionshere"
cd /share/classifier-play-1.0
bin/classifier-play \
-Dconfig.file=conf/application-stage.conf \
-Dhttp.port=${HTTP_PORT} -Dhttp.address=0.0.0.0 \
-J-Xms${JVM_HEAP_MIN} -J-Xmx${JVM_HEAP_MAX} -J-server \
-J-XX:+UseG1GC -J-XX:+UseStringDeduplication
Reference:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
http://sillycat.iteye.com/blog/2256757
http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
http://www.projectatomic.io/docs/docker-storage-recommendation/
https://coreos.com/os/docs/latest/mounting-storage.html
I have a project which have a large file there, it is a known issue, we will fix that later.
I want to docker that API, I did that on EC2 instance. I create that instance long time ago, I was thinking that I only need 30G, but today. I met this problem saying not enough space for build the docker image.
Error Message:
Error response from daemon: Cannot destroy container clever_bartik: Driver devicemapper failed to remove root filesystem ef2f1e5b38fddaac78966c5e828d5a0aabd92600c3d87346f9e371bfeb5401d2: Error saving transaction metadata: Error writing metadata to /var/lib/docker/devicemapper/metadata/.tmp258670910: write /var/lib/docker/devicemapper/metadata/.tmp258670910: no space left on device
Solution:
First of all, clean the file first.
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 30G 30G 0 100% /
devtmpfs 7.5G 100K 7.5G 1% /dev
tmpfs 7.5G 0 7.5G 0% /dev/shm
This command will show me that all the space of disk is used.
sudo du -hsx * | sort -rh | head -10
12G home
1.2G usr
526M var
139M lib
62M boot
43M opt
23M lib64
13M sbin
7.6M etc
7.0M bin
This command will show me where did the space goes. I will then delete the files. Restart the instance.
And then I will do 2 things.
1 Add Disk Space to my Small Lonely EC2
click —> [ELASTIC BLOCK STORE] —>[Volumes] —>”Create Volume” Attach this Volume to My EC2
After create and attach the disk to that machine, that does not mean we can directly use it on linux.
List the Disk
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
└─xvda1 202:1 0 30G 0 part /
xvdf 202:80 0 100G 0 disk
loop0 7:0 0 100G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:1-270348-pool 253:0 0 100G 0 dm
I have not file system on that.
> sudo file -s /dev/xvdf
/dev/xvdf: data
Create the file system
> sudo mkfs -t ext4 /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: baca4bba-b1ea-419c-b3ba-39b0a117f6bb
Create the Directory
> sudo mkdir data
Mount the disk to the directory
> sudo mount /dev/xvdf /data
I did not try auto mount.
2 Set up and Configure Docker to use the ext storage
Check the current docker info
> sudo docker info
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
The problem is here. They are using up 30GB.
Stop Docker
> sudo service docker stop
Clean the old file
> sudo rm -fr /var/lib/docker/
Config the storage
> sudo cat /etc/sysconfig/docker-storage
# This file may be automatically generated by an installation program.
# By default, Docker uses a loopback-mounted sparse file in
# /var/lib/docker. The loopback makes it slower, and there are some
# restrictive defaults, such as 100GB max storage.
# If your installation did not set a custom storage for Docker, you
# may do it below.
# Example: Use a custom pair of raw logical volumes (one for metadata,
# one for data).
# DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/dev/myvg/my-docker-metadata --storage-opt dm.datadev=/dev/myvg/my-docker-data"
DOCKER_STORAGE_OPTIONS="--storage-opt dm.metadatadev=/data/my-docker-metadata --storage-opt dm.datadev=/data/my-docker-data"
Start docker there
> sudo service docker start
This does not work.
Try this
> sudo vi /etc/sysconfig/docker-storage-setup
DEVS="/dev/xvdf"
STORAGE_DRIVER=devicemapper
> /usr/bin
> sudo wget https://raw.githubusercontent.com/projectatomic/docker-storage-setup/master/docker-storage-setup.sh
> sudo chmod a+x docker-storage-setup.sh
> ./docker-storage-setup.sh
It seems not working. I do not want to spend more time on this. I will do like this.
Soft the link the file to the configuration
> sudo ln -s /data/docker /var/lib/docker
> sudo mkdir -p devicemapper/devicemapper/data
> sudo mkdir -p devicemapper/devicemapper/metadata
> sudo mkdir /var/lib/docker/graph
> sudo mkdir /var/lib/docker/tmp
> sudo mkdir /var/lib/docker/devicemapper/metadata
>sudo mkdir /var/lib/docker/containers
>sudo mkdir /var/lib/docker/init
>sudo mkdir /var/lib/docker/trust
>sudo mkdir /var/lib/docker/volumes
Error Message:
open /var/lib/docker/repositories-devicemapper: no such file or directory
Solution:
> sudo vi /var/lib/docker/repositories-devicemapper
{"Repositories":{}}
It does not work. I guess it relates to the manual deleting docker.
Remote the docker
> sudo yum remove docker
Install the docker again
> wget -qO- https://get.docker.com/ | sh
Let me try this:
> sudo mkdir /var/lib/docker
> sudo mount /dev/xvdf /var/lib/docker
It works perfect. Here is the Dockerfile
#Run a Simple REST API based on playframework
#Prepre the OS
FROM centos:7
MAINTAINER Carl Luo <luohuazju@gmail.com>
ENV DEBIAN_FRONTEND noninteractive
ENV JVM_HEAP_MAX 12g
ENV JVM_HEAP_MIN 8g
ENV HTTP_PORT 8003
#Install Java
RUN yum install -y java-1.8.0-openjdk-headless-1.8.0.60-2.b27.el7_1 lapack atlas
RUN yum install -y unzip
RUN yum install -y wget
#Install the Application
RUN mkdir /share/
WORKDIR /share/
#ADD data/classifier_models.tar.gz /share/
RUN wget https://xxxxx/classifier_models.tar.gz
RUN tar zxvf classifier_models.tar.gz
RUN rm -fr classifier_models.tar.gz
ADD target/universal/classifier-play-1.0.zip /share/
RUN unzip classifier-play-1.0.zip
RUN rm -fr classifier-play-1.0.zip
#Start the Application
EXPOSE 8003
RUN mkdir -p /app/
ADD start.sh /app/
WORKDIR /app
CMD [ "./start.sh" ]
Here is the start.sh
#!/bin/sh -ex
APPLICATION_SECRET="nosessionshere"
cd /share/classifier-play-1.0
bin/classifier-play \
-Dconfig.file=conf/application-stage.conf \
-Dhttp.port=${HTTP_PORT} -Dhttp.address=0.0.0.0 \
-J-Xms${JVM_HEAP_MIN} -J-Xmx${JVM_HEAP_MAX} -J-server \
-J-XX:+UseG1GC -J-XX:+UseStringDeduplication
Reference:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
http://sillycat.iteye.com/blog/2256757
http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
http://www.projectatomic.io/docs/docker-storage-recommendation/
https://coreos.com/os/docs/latest/mounting-storage.html
发表评论
-
Stop Update Here
2020-04-28 09:00 260I will stop update here, and mo ... -
NodeJS12 and Zlib
2020-04-01 07:44 429NodeJS12 and Zlib It works as ... -
Docker Swarm 2020(2)Docker Swarm and Portainer
2020-03-31 23:18 310Docker Swarm 2020(2)Docker Swar ... -
Docker Swarm 2020(1)Simply Install and Use Swarm
2020-03-31 07:58 321Docker Swarm 2020(1)Simply Inst ... -
Traefik 2020(1)Introduction and Installation
2020-03-29 13:52 291Traefik 2020(1)Introduction and ... -
Portainer 2020(4)Deploy Nginx and Others
2020-03-20 12:06 378Portainer 2020(4)Deploy Nginx a ... -
Private Registry 2020(1)No auth in registry Nginx AUTH for UI
2020-03-18 00:56 373Private Registry 2020(1)No auth ... -
Docker Compose 2020(1)Installation and Basic
2020-03-15 08:10 325Docker Compose 2020(1)Installat ... -
VPN Server 2020(2)Docker on CentOS in Ubuntu
2020-03-02 08:04 397VPN Server 2020(2)Docker on Cen ... -
Buffer in NodeJS 12 and NodeJS 8
2020-02-25 06:43 334Buffer in NodeJS 12 and NodeJS ... -
NodeJS ENV Similar to JENV and PyENV
2020-02-25 05:14 415NodeJS ENV Similar to JENV and ... -
Prometheus HA 2020(3)AlertManager Cluster
2020-02-24 01:47 359Prometheus HA 2020(3)AlertManag ... -
Serverless with NodeJS and TencentCloud 2020(5)CRON and Settings
2020-02-24 01:46 290Serverless with NodeJS and Tenc ... -
GraphQL 2019(3)Connect to MySQL
2020-02-24 01:48 207GraphQL 2019(3)Connect to MySQL ... -
GraphQL 2019(2)GraphQL and Deploy to Tencent Cloud
2020-02-24 01:48 390GraphQL 2019(2)GraphQL and Depl ... -
GraphQL 2019(1)Apollo Basic
2020-02-19 01:36 274GraphQL 2019(1)Apollo Basic Cl ... -
Serverless with NodeJS and TencentCloud 2020(4)Multiple Handlers and Running wit
2020-02-19 01:19 263Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(3)Build Tree and Traverse Tree
2020-02-19 01:19 259Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(2)Trigger SCF in SCF
2020-02-19 01:18 250Serverless with NodeJS and Tenc ... -
Serverless with NodeJS and TencentCloud 2020(1)Running with Component
2020-02-19 01:17 233Serverless with NodeJS and Tenc ...
相关推荐
Amazon EC2功能全面介绍,需要学习了解amazon ec2的同学最好看看....
亚马逊弹性计算云(Amazon EC2)是一个Web服务,提供可调整的云计算能力。它旨在为开发人员提供简便的使用网络规模计算。 Amazon EC2简单的Web服务界面,让您可以以最小的代价获取和配置计算能力。它让您完全控制您...
亚马逊ec2云亚马逊ec2云亚马逊ec2云亚马逊ec2云
To use Amazon EC2, you need an AWS account. If you don't already have one, you'll be prompted to create one when you sign up for Amazon EC2. Signing up for Amazon EC2 also automatically signs you up ...
开源工具用于在亚马逊ECS上构建定制的调度器。Open source tools for building custom schedulers on Amazon ECS。
amazon-ecs-agent, Amazon弹性容器服务代理 Amazon ECS容器代理 Amazon ECS容器代理是为Amazon弹性容器服务( 。亚马逊 ) 开发的软件。它运行在容器实例上,并代表 Amazon ECS启动容器。用法运行这里软件的最佳信息...
Amazon EC2 API Reference
您将学习如何使用 Amazon 系统映像启动 Amazon EC2 实例、创建适合 SSH 身份验证的密钥对、通过安全组保障 Amazon EC2 实例网络访问的安全、通过引导启动脚本自动配置 Amazon EC2 实例以及将弹性 IP 附 加到 Amazon ...
Over 40 hands-on recipes to develop and deploy real-world applications using Amazon EC2 AWS Cloud from Amazon offers a cutting-edge platform to architecture, build, and deploy web-scale cloud ...
使用竞价型实例的一个节点ECS集群这将在us-east-2(Ohio)中部署一个带有t3a.micro的单节点ECS,这将为您带来2vcpu / 1Gib约2美元的收益。 ECS Tasks将使用AWS Cloud Map进行服务发现,而Ha Proxy提供负载平衡。安装...
对于学习云计算,利用亚马逊EC2的同学会很有帮助的
Amazon EC2 预留实例和其他预留模型
2017年最新的AWS实践类图书,理解云计算有帮助,英文版
ec2instances.info, Amazon EC2实例比较站点 ec2instances.info我在亚马逊站点上比较EC2实例度量和定价的问题,所以我做了这个。 欢迎使用 !项目状态 我正积极地在贡献者的帮助下维护网站,但我不打算使用大型的新...
ECS使您的应用程序可以灵活地结合使用Amazon EC2和AWS Fargate以及Spot和On-Demand定价选项。 目的 了解Amazon ECS的基本概念和新功能。 练习用于Amazon ECS管理的有用工具。 您将使用的AWS服务 亚马逊弹性容器
programming amazon ec2
要求Jenkins至少具有2.176.1版AWS账户插件安装使用Jenkins插件管理器安装组态例子当前有以下示例设置(也在此仓库中): - 临时主机和代理 -由AWS设计的Jenkins安装Amazon ECS集群作为前提条件,您必须已创建具有...
Docker on Amazon Web Services starts with the basics of containers, Docker, and AWS, before teaching you how to install Docker on your local machine and establish access to your AWS account....