Customize Docker storage

I have been playing around with Docker lately. It is a very interesting piece of technology, which let you build relatively small and lightweight containers with a complete server environment with its own file system. Instead of running as a virtualized platform, it runs on top of your host operating systems kernel. You can pull some pre-made images from the public Docker repository, and customize them to your likings.

When learning how to use Docker, I have used CentOS 7 running in VirtualBox 5 as the host operating system. During my first tries with Docker, I often found various parts of my disks running out of space. Either the / partition or the Docker specific LVM logical volume docker-pool ran out of space. This is because Docker by default will store it's metadata and temporary data in /var/lib/docker on the root partition, and store the Docker images in the docker-pool logical volume.

When playing with Docker now, I always start out with adding new storage before installing and starting Docker.

Disclaimer

Some of the commands in this article can be destructive to your system if running on the wrong disk device. Don't run any of the commands unless you know exactly what the command will do to your system.

Adding a new disk device

Through the VirtualBox console I have added a new 100G VDI file to contain my new disk device. After starting CentOS 7 up, I can find this disk by running the lvmdiskscan command:
[root@localhost ~]# lvmdiskscan
  /dev/centos/root [       9,78 GiB]
  /dev/sda1        [     500,00 MiB]
  /dev/centos/swap [       1,20 GiB]
  /dev/sda2        [      11,51 GiB] LVM physical volume
  /dev/sdb         [     100,00 GiB]
  3 disks
  1 partition
  0 LVM physical volume whole disks
  1 LVM physical volume

From the above output we can see that the new 100GB disk is the /dev/sdb decive. On your environment it might be a different device. The rest of this post assumes the device is called /dev/sdb. 

# Ensure no disk partitions on /dev/sdb:

[root@localhost ~]# parted /dev/sdb print
Error: /dev/sdb: unrecognised disk label
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Please notice that it reported the disk to be 107GB large. We will need that information later when we create the partition.

# Create a disk partition using the entire 107GB:

[root@localhost ~]# parted /dev/sdb mklabel msdos
Information: You may need to update /etc/fstab.

[root@localhost ~]# parted /dev/sdb mkpart primary 1 107g
Information: You may need to update /etc/fstab.

[root@localhost ~]# parted /dev/sdb print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  107GB  107GB  primary

The lvmdiskscan command will verify that your new partition is called /dev/sdb1:
 [root@localhost ~]# lvmdiskscan
  /dev/centos/root [       9,78 GiB]
  /dev/sda1        [     500,00 MiB]
  /dev/centos/swap [       1,20 GiB]
  /dev/sda2        [      11,51 GiB] LVM physical volume
  /dev/sdb1        [     100,00 GiB]
  2 disks
  2 partitions
  0 LVM physical volume whole disks
  1 LVM physical volume

# Initialize the new disk:

[root@localhost ~]# pvcreate -v /dev/sdb1
    Set up physical volume for "/dev/sdb1" with 209713152 available sectors
    Zeroing start of device /dev/sdb1
    Writing physical volume data to disk "/dev/sdb1"
Physical volume "/dev/sdb1" successfully created

# Create a LVM volume group:

[root@localhost ~]# vgcreate docker-vg00 /dev/sdb1 -v
    Adding physical volume '/dev/sdb1' to volume group 'docker-vg00'
    Archiving volume group "docker-vg00" metadata (seqno 0).
    Creating volume group backup "/etc/lvm/backup/docker-vg00" (seqno 1).
Volume group "docker-vg00" successfully created

# Create a logical volume to contain the Docker metadata and temporary files as well as any storage you will share between your host and the Docker containers:

[root@localhost ~]# lvcreate -L 25g -n docker-storage docker-vg00
  Logical volume "docker-storage" created.

# Create and mount file system

[root@localhost ~]# mkfs.xfs /dev/docker-vg00/docker-storage
[root@localhost ~]# mkdir /docker
[root@localhost ~]# echo "/dev/docker-vg00/docker-storage /docker  xfs     defaults        0 0" >> /etc/fstab
[root@localhost ~]# mount /dev/docker-vg00/docker-storage /docker
[root@localhost ~]# mkdir -p /docker/data
[root@localhost ~]# mkdir -p /docker/sharedfs
[root@localhost ~]# chmod 777 /docker/sharedfs

# Verify that the /docker mount point was mounted:

[root@localhost ~]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
/dev/mapper/centos-root                    11G  3,7G  6,6G  36% /
devtmpfs                                  911M     0  911M   0% /dev
tmpfs                                     921M     0  921M   0% /dev/shm
tmpfs                                     921M  8,4M  912M   1% /run
tmpfs                                     921M     0  921M   0% /sys/fs/cgroup
/dev/sda1                                 497M  244M  254M  50% /boot
/dev/mapper/docker--vg00-docker--storage   25G   33M   25G   1% /docker

# Docker installation

[root@localhost ~]# yum install -y docker
[root@localhost ~]# yum update -y

# Take a backup of the Docker configuration file and the Docker storage creation file

[root@localhost ~]# NOW=$(date +"%Y-%m-%d_%H-%M")
[root@localhost ~]# cp /etc/sysconfig/docker /etc/sysconfig/docker.BACKUP.${NOW}
[root@localhost ~]# cp /bin/docker-storage-setup /bin/docker-storage-setup.BACKUP.${NOW}}

# Configure Docker to use the mount point /docker for metadata and temporary data

[root@localhost ~]# sed -i "s#OPTIONS\='--selinux-enabled'#\#OPTIONS\='--selinux-enabled'\nOPTIONS\='--selinux-enabled -g /docker/data -p /var/run/docker.pid'#" /etc/sysconfig/docker

# Configure the docker-pool to use the new LVM volume group we created, which should have 82GB left:

[root@localhost ~]# sed -i "s#\/bin\/bash#\/bin\/bash\n\nVG\=docker-vg00#" /bin/docker-storage-setup

# Start and enable the Docker service

The docker.service relies on docker-storage-setup.service, which will create the docker-pool logical volume based on the information in /bin/docker-storage-setup, which we have already customized to use the new volume group docker-vg00.
[root@localhost ~]# systemctl start docker
[root@localhost ~]# systemctl enable docker

# Verify that docker-pool has been created

[root@localhost ~]# lvs docker-vg00
  LV             VG          Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool    docker-vg00 twi-a-t--- 44,93g             0,04   0,06
  docker-storage docker-vg00 -wi-ao---- 25,00g

# Pull a ready Docker image from the public repository:

[root@localhost ~]# docker pull ubuntu:14.04
14.04: Pulling from docker.io/ubuntu
6071b4945dcf: Pull complete
5bff21ba5409: Pull complete
e5855facec0b: Pull complete
8251da35e7a7: Pull complete
Digest: sha256:fb00a642c029fdd339cd7ed5d46ec4ace045c595123a56fc9ba331675ec9da08
Status: Downloaded newer image for docker.io/ubuntu:14.04

# Show all your Docker images. Currently it should only be the Ubuntu image you just downloaded:

[root@localhost ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/ubuntu    14.04               8251da35e7a7        4 days ago          188.3 MB

# Run the downloaded Ubuntu image in an interactive Bourne Again shell

[root@localhost ~]# 
docker run -ti ubuntu:14.04 /bin/bash