From iRedMail

Revision as of 12:17, 29 June 2011

GlusterFS

Introduction

This tutorial shows how to set up a master-master high-availability failover iRedMail solution with two servers (CentOS 5.x) that use GlusterFS. Each server will be a clone/mirror of the other server, and files will be replicated automatically across both servers. This is not high-availability storage only but also should work as a failover if one of them came down.

• In this tutorial I use two nodes:
  • node-01 (192.168.1.13)
  • node-02 (192.168.1.14)

Each of them will act as a client for its loopback so it's a server of itself; and client for the other node.

• Example: node-01 as a client:
  • client for itself (127.0.0.1)
  • client for node-02 (192.168.1.14)

• Example: node-02 as server:
  • server for itself (127.0.0.1)
  • server for node-01 (192.168.1.13)

• both servers should have the same hostname for future DNS modifications:

node-01#: vi /etc/hosts

[...]
192.168.1.13           mail.node.com mail
[...] 

node-02#: vi /etc/hosts

[...]
192.168.1.14           mail.node.com mail
[...] 

We will use the IP addresses instead of hostnames when configuring the GlusterFS so there is no need for nodes to resolve each other (this will facilitate the process of DNS later)

Setting Up The GlusterFS as (server)

1. mail.node.com (node-01, 192.168.1.13)
2. mail.node.com (node-02, 192.168.1.14)

yum groupinstall 'Development Tools'

yum groupinstall 'Development Libraries'

yum install libibverbs-devel fuse-devel

Then we download the latest GlusterFS release from http://www.gluster.org/download.php and build it as follows:

cd /tmp
wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.9.tar.gz
tar xvfz glusterfs-2.0.9.tar.gz
cd glusterfs-2.0.9
./configure

At the end of the ./configure command, you should see something like this:

[...]
GlusterFS configure summary
===========================
FUSE client        : yes
Infiniband verbs   : yes
epoll IO multiplex : yes
Berkeley-DB        : yes
libglusterfsclient : yes
argp-standalone    : no

[root@mail glusterfs-2.0.9]#

make && make install
ldconfig

glusterfs --version

root@mail glusterfs-2.0.9]# ''glusterfs --version
glusterfs 2.0.9 built on June 26 2011 12:34:50
Repository revision: v2.0.9
Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.''
[root@mail glusterfs-2.0.9]#

Next we create a few directories:

mkdir /data/
mkdir /data/export
mkdir /data/export-ns
mkdir /etc/glusterfs

Now we create the GlusterFS server configuration file /etc/glusterfs/glusterfsd.vol which defines which directory will be exported /data/export and what client is allowed to connect (127.0.0.1,192.168.1.14 on node-01) (127.0.0.1,192.168.1.13 on node-02):

• On node-01:

vi /etc/glusterfs/glusterfsd.vol

volume posix
  type storage/posix
  option directory /data/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow 127.0.0.1,192.168.1.14
  subvolumes brick
end-volume

• on node-02:

volume posix
  type storage/posix
  option directory /data/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow 127.0.0.1,192.168.1.13
  subvolumes brick
end-volume

Please note that it is possible to use wildcards for the IP addresses (like 192.168.*) and that you can specify multiple IP addresses separated by comma (e.g. 192.168.1.13,192.168.1.14).

Afterwards we create the following symlink ...

ln -s /usr/local/sbin/glusterfsd /sbin/glusterfsd

... and then the system startup links for the GlusterFS server and start it:

chkconfig --levels 35 glusterfsd on
/etc/init.d/glusterfsd start

Preparing GlusterFS as(Client)

modprobe fuse

... and create the file /etc/rc.modules with the following contents so that the fuse kernel module will be loaded automatically whenever the system boots:

vi /etc/rc.modules

modprobe fuse 

Make the file executable:

chmod +x /etc/rc.modules 

Next we create the file /etc/glusterfs/glusterfs.vol:

vi /etc/glusterfs/glusterfs.vol

• On node-01:

volume remote1
  type protocol/client
  option transport-type tcp
  option remote-host 127.0.0.1
  option remote-subvolume brick
end-volume

volume remote2
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.1.14
  option remote-subvolume brick
end-volume

volume replicate
  type cluster/replicate
  subvolumes remote1 remote2
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume 

• On node-02:

volume remote1
  type protocol/client
  option transport-type tcp
  option remote-host 127.0.0.1
  option remote-subvolume brick
end-volume

volume remote2
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.1.13
  option remote-subvolume brick
end-volume

volume replicate
  type cluster/replicate
  subvolumes remote1 remote2
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume 

Installing iRedMail

• Please refer to the featured installation for CentOS at: http://code.google.com/p/iredmail/wiki/Installation
• Install iRedMail on both nodes

Things to consider while installing iRedMail (Important)

1. change the mail storage folder to be /mnt/glusterfs

This directory will be automatically created by iRedMail installer so DO NOT create it.

1. If you have choosen to use the iptables rules bundeled with iRedMail so you need to allow port 6996 by running this command on both servers:

iptables -I INPUT -p tcp --dport 6996 -j ACCEPT

1. If you have considered creating /mnt/glusterfs directory manually please make sure that the directory owned by vmail user:

chown -R vmail:vmail /mnt/glusterfs/

Mounting the mail storage directory

Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:

glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs

Or:

mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs

You should now see the new share in the outputs of...

mount

[root@mail ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
glusterfs#/etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse (rw,allow_other,default_permissions,max_read=131072)
[root@mail ~]#

... and...

df -h

[root@mail ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       29G  2.1G   25G   9% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                 187M     0  187M   0% /dev/shm
glusterfs#/etc/glusterfs/glusterfs.vol
                       28G  2.2G   25G   9% /mnt/glusterfs
[root@mail ~]#

(node-01 and node-02 each have 28GB of space for the GlusterFS filesystem, but because the data is mirrored, the client doesn't see 56GB (2 x 28GB), but only 28GB.)

Instead of mounting the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted automatically when the client boots.

Open /etc/fstab and append the following line:

vi /etc/fstab

[...]
/etc/glusterfs/glusterfs.vol  /mnt/glusterfs  glusterfs  defaults  0  0 

This should happen also on both nodes.

MySQL Database replication

Coming soon - you can instead export and import using PHPMyAdmin. (just for now)

Testing

1. Go to: https://mail.node.com/iredadmin, then create users1@node.com
2. list the files under /mnt/glusterfs

ls -l /mnt/glusterfs

Result:

[root@mail glusterfs]# ls -l
total 8
drwx------ 3 vmail vmail 4096 Jun 27 15:04 vmail1
[root@mail glusterfs]#

• Note: if you didn't find the vmail directory do not bother, that's why you didn't run a read command, read command is such case is to access your mail account using the webmail (RoundCube).

Go to http://mail.node.com/mail, then enter your username: user1@node.com and Password: "Secret" then try to list /mnt/glusterfs again.

Try to list /mnt/glusterfs on both nodes, you should see the same data, also try to make some changes like recieving a new message to your mailbox should appear in Maildir/cur/ or Maildir/new

Then try to delete this message and list Maildir/cur/ again on the other node.

• On node-01:

[root@mail new]# ls -l
total 8
-rw------- 1 vmail vmail 1003 Jun 29 11:39 1309336773.M243030P3607.mail.node.com,S=1003,W=1029
[root@mail new]#

• On node-02:

[root@mail new]# ls -l
total 8
-rw------- 1 vmail vmail 1003 Jun 29 11:39 1309336773.M243030P3607.mail.node.com,S=1003,W=1029
[root@mail new]#

ALso try to have a look at /data/export on both nodes:

• On node-01:

[root@mail ~]# ls -l /data/export
total 8
drwx------ 3 vmail vmail 4096 Jun 27 15:04 vmail1
[root@mail ~]#

• On node-02:

[root@mail ~]# ls -l /data/export
total 8
drwx------ 3 vmail vmail 4096 Jun 27 15:04 vmail1
[root@mail ~]#

DNS Modifications

1. Define two A records with the same name and different IP addresses
2. Define MX record with the high priority

dig mail.node.com

;; QUESTION SECTION:
;mail.node.com.                 IN      A

;; ANSWER SECTION:
mail.node.com.          43200       IN      A       192.168.1.13
mail.node.com.          43200       IN      A       192.168.1.14

Now you should add MX record for the domain node.com as following:

;; QUESTION SECTION:
;node.com.                     IN      MX

;; ANSWER SECTION:
node.com. 
            3600    IN      MX      10 mail.node.com.

Links

• GlusterFS: http://www.gluster.org/
• CentOS: http://www.centos.org/