Distributed File Storage For CMS

Achieving minimum file redundancy and high availability requires at least two load-balanced application servers. A permanent replication system must keep user-generated files in sync between servers in the pool. We previously developed a system based on “Rsync” or “Csync2”. 

It was reliable, affordable, and did not require specialist tools or skills. It did require occasional maintenance and some configuration during server expansion. Performance could have been better. The biggest drawback of this solution was that the CMS had to initiate the file sync after every modification.

We were looking for a solution where the file synch would take place in the background without the CMS triggering it. Virtual servers with NFS or AWS EFS proved too slow or unaffordable.

A potential solution had to satisfy some predefined essential criteria:

• Performance - file writing must occur on the local server; synchronization occurs in the background in parallel. 
• Reliability - there is no intermediary, and failover is automatic and lossless.
• Simplicity (a single technology that provides all the necessary mechanisms.
• Extensibility - adding new synchronization nodes with minimum changes to the existing configuration.
• Affordability – open-source solutions, use of existing servers.

We found a lot of content about solutions based on DRBD for data replication and Pacemaker for the clustering (read failover), so we tried that approach. All DRBD/Pacemaker solutions require using shared IP addresses to switch to a working server if one of the cluster servers falls over. Our cloud provider offers shared IP addresses, but they break the communication between the servers, negatively impacting DRBD storage cloning.

This solution proved to be somewhat demanding:

• It needed separate block storage on every server.
• We had to organise each block storage into a ‘volume group’, the basis for a ‘logical volume’ – a partition we mounted to the desired application directory (‘/path/to/mount/dir’ in the diagram below). We duplicated that partition across servers using DRBD software (which provided seamless data mirroring between servers).
• Both servers share a single static and a private IP address as an entry point for the NFS service, which works on top of the clustered storage, making files and folders available for the application servers and their services (our CMS, web server, etc.).
• Both servers had Pacemaker cluster software installed, providing failover - if one of the clustered servers goes off, Pacemaker switches shared IP and DRBD to the next available cluster server.

Regardless of how comprehensive this solution seemed, it was cumbersome and difficult to understand. The whole setup was sensitive to the quality of the networking configuration. We decided to push further, in our pursuit of the "ideal" solution.

A promising solution was "GlusterFS" - an open-source, scalable network filesystem suitable for data-intensive tasks such as cloud storage. It can provide data volumes distributed between thousands of servers. This paragraph from its documentation probably best describes it in short:

 "A gluster volume is a collection of servers belonging to a Trusted Storage Pool. A management daemon ("glusterd") runs on each server and manages a brick process ("glusterfsd") which in turn exports the underlying on-disk storage (XFS filesystem). The client process mounts the volume and exposes the storage from all the bricks as a single unified storage namespace to the applications accessing it. The client and brick processes' stacks have various translators loaded in them. I/O from the application is routed to different bricks via these translators."

 Gluster file system supports different types of volumes: 

• The Distributed Glusterfs Volume is the default. It distributes files across various nodes in the volume. There is no data redundancy. It can scale the volume efficiently, but in case of a node failure, it can lead to complete data loss.
• Replicated Glusterfs Volume - It maintains the exact copies of the data on all nodes (at least two nodes constitute a volume). At least one volume must be operative for data access. Such a volume provides data redundancy and improves reliability.
• Distributed Replicated Glusterfs Volume - It distributes files across replicated sets of nodes. It provides high availability of data, redundancy and scaling.
• Dispersed Glusterfs Volume - It strips files across multiple nodes in the volume, with some redundancy added. It provides a configurable level of reliability with minimum waste of space. 
• Distributed Dispersed Glusterfs Volume – Similar to the distributed replicated volumes but uses dispersed sub-volumes. It provides simple scaling and load distribution across nodes.

We opted for the "Replicated Glusterfs Volume" architecture. It proved straightforward and robust enough for our needs.

The diagram of the architecture:


Further in the text, you can find the setup of "GlusterFS" on our application servers in two variants: as a manual install from the console and as an automated Ansible script.

Ansible installation is more straightforward and shorter than console installation.

Manual installation of "GlusterFS" data volume replication from a console

Each step should be performed on both servers unless explicitly stated. 

 UPDATE SERVER

# yum update -y

 SET SERVER HOSTNAME

# hostnamectl set-hostname hostname

Note: hostname should be replaced with actual value.

 INSTALL NECESSARY PACKAGES

# yum install mc wget mlocate epel-release wget -y

 SET SELINUX TO PERMISSIVE MODE

Edit file /etc/selinux/config and change the SELINUX value to “SELINUX=permissive”

# nano /etc/selinux/config

 INSTALL GLUSTERFS

# yum install centos-release-gluster

# yum install glusterfs-server

 START GLUSTERFS

# systemctl start glusterd

 ADD FIREWALL RULES AND RELOAD THE FIREWALL

# firewall-cmd --zone=internal --add-service=glusterfs --permanent

# firewall-cmd --zone=internal --add-source=server1_ip/32 --permanent

# firewall-cmd --zone=internal --add-source=server2_ip/32 --permanent

# firewall-cmd –reload

Note: server1_ip and server2_ip should be replaced with the actual private IP addresses.

 ENABLE AUTOMATIC START OF FIREWALLD AND GLUSTERFS ON BOOT:

# systemctl enable firewalld 

# systemctl enable glusterd

 SET HOSTNAMES

Edit the file /etc/hosts. 

Server1_private_ip           server1_hostname

Server2_private_ip           server2_hostname

 CREATE TRUSTED STORAGE POOL (ONLY ON APPLICATION SERVER 1)

This command will create a network of trusted servers with data sharing:

gluster peer probe server2_hostname

 CREATE A SHARED DIRECTORY FOR FUTURE MOUNTING POINT

"GlusterFS" can mirror data from a directory on the same volume as server files. It can also replicate data from a directory on a separate physical or logical block volume. This time it will mirror data on the server's system (and only) volume. Regardless of the type of underlying volume, it needs a data directory.

 CREATE A DIRECTORY FOR FILES TO BE REPLICATED

# mkdir /path/to/volume/dir

Note: this directory will serve as a physical base for the data on every server mounted later to the actual directory, in our case /path/to/mount/dir.

 CREATE A DISTRIBUTED REPLICATED VOLUME (ONLY ON APPLICATION SERVER 1)

Create a "GlusterFS" replicated data volume, which will be mounted to the destination directory and will be spread among the cluster servers to provide data failover and accessibility in case any of the cluster servers failover.

 # gluster volume create gfs-volume replica 2 server1_hostname: 

 /path/to/volume/dir server2_hostname: /path/to/volume/dir force

 START THE VOLUME

Start the replicated volumes on both servers to start data replication:

 # gluster volume start gfs-volume

 MOUNT THE GLUSTERFS FILESYSTEM

To ensure that the GlusterFS is mounted on every server start, add an entry to /etc/fstab file on both application servers:

 # nano /etc/fstab
 
Enter this line at the end of the file and save it:
 
server1_hostname:/gfs-volume  /path/to/mount/dir  glusterfs defaults,_netdev,backup-volfile-servers=server2_hostname 0 0
 
Note:' server1_hostname' and 'server2_hostname' should be replaced with their actual values, as well as '/path/to/volume/dir' and '/path/to/mount/dir' .
 
Installation of GlusterFS data volume replication with Ansible
---

# Inventory server group selection
- hosts: servers_inventory_group_name
  become: yes
 
  tasks:
 
# UPDATE SERVER
  - name: CentOS security update
    yum:
      name: '*'
      security: yes
      state: latest
 
# INSTALL REQUIRED REPOSITORY
 
# Install EPEL repository. It is required for GlusterFS installation
  - name: Install EPEL repo
    yum: |
      name: https://dl.fedoraproject.org/pub/epel/epel-release-latest-{{ ansible_distribution_major_version }}.noarch.rpm
      state: present
 
# Import EPEL GPG key to verify the signature of package
  - name: Import EPEL GPG key
    rpm_key:
      key: /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-{{ ansible_distribution_major_version }}
      state: present
 
# SET SERVER HOSTNAME
 
  - name: Set server hostname
    hostname: name={{ my_hname }}
 
# Update server's hosts file so the servers can access each other via hostnames rather # than IP addresses
  - name: Update server's hosts file
    lineinfile:
      path: /etc/hosts
      line: "{{ item }}"
      state: present
      backup: yes
    with_items:
      - "{{ my_private_ip }}  {{ my_hname }} {{ my_hname }}.local"
      - "{{ other_private_ip }}  {{ other_hname }} {{ other_hname }}.local"
 
# REBOOT SERVER
 
  - name: Reboot server
    reboot:
      msg: "Reboot initiated by Ansible"
 
# INSTALL GLUSTERFS
 
# Install glusterfs server package 
 - name: Install glusterfs server
    yum:
      name: glusterfs-server
      state: present
 
# Start glusterd service and make it start at boot 
 - name: Start glusterfs service and enable start at boot
    service:
      name: glusterd
      state: started
      enabled: yes
 
  # Add glusterfs service to the Public zone
  - name: Add glusterfs service to Public zone
    ansible.posix.firewalld:
      zone: public
      service: glusterfs
      permanent: yes
      immediate: yes
      state: enabled
 
  # Add source in the firewall to limit access only to other servers in the pool
  - name: Add source in firewalld
    firewalld:
      source: "{{ other_private_ip }}/32"
      zone: public
      immediate: yes
      permanent: yes
      state: enabled
 
  # Reload firewalld service
  - name: Reload firewalld service
    service:
      name: firewalld
      state: reloaded
 
# SET GLUSTERFS
 
  # Create glusterfs brick directory
  - name: Create glusterfs brick dir and set permissions to 775
    file:
      path: /path/to/volume/dir
      owner: root
      group: apache
      state: directory
      mode: '0775'
 
  # Create glusterfs volume
  - name: Create glusterfs volume
    gluster_volume:
      state: present
      name: gfs_volume
      bricks: /path/to/volume/dir
      replicas: 2
      cluster: "{{ groups.servers_inventory_group_name | join(',') }}"
      host: "{{ server1_hostname }}"
      force: yes
    run_once: true
 
  # Mount glusterfs volume
  - name: Mount gluster volume
    mount:
      name: /path/to/mount/dir
      src: "{{ inventory_hostname }}:/gfs_volume"
      fstype: "glusterfs"
      opts: "defaults,_netdev"
      state: "mounted"
 
The above ansible script should install and set GlusterFS on all servers in the pool, listed in  '[servers_inventory_group_name]' section of the default inventory file ('/etc/ansible/hosts'). It contains several variables defined in three separate files, one in the 'group_vars' folder and the other two in the 'host_vars' folder. Variables defined in the 'group_vars' folder are standard for all servers in the pool. They are in a file whose name is identical to the name of the inventory file group that defines the server's IP addresses in the pool. Variables defined in the 'host_vars' folder are server-specific, and for every server in the pool there will be a separate variable file named by its private IP address.
 
Group-specific variables definition:
~/group_vars/servers_inventory_group_name
---
 
# Common variables for application servers on live environment
 
# GlusterFS brick directory
gluster_dir: /actual/path/to/volume/dir
 
# GlusterFS mount directory
mount_dir: /actual/path/to/mount/dir
 
# GlusterFS volume name
gluster_vol: actual_gfs_volume
 
# Server1 hostname
server1_hostname: server1_actual_hostname
 
Host specific variables definition:
~/host_vars/server1_ip
---
 
# First server's hostname
my_hname: server1_actual_hostname
 
# Second server's hostname
other_hname: server2_actual_hostname
 
# This server IP
my_private_ip: 192.168.x.y
 
# Other server IP
other_private_ip: 192.168.x.z
 
~/host_vars/server2_ip
---
 
# First server's hostname
my_hname: server2_actual_hostname
 
# Second server's hostname
other_hname: server1_actual_hostname
 
# This server IP
my_private_ip: 192.168.x.z
 
# Other server IP
other_private_ip: 192.168.x.y