Re: [Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication
Horacio / Ed, Thank you very much for your responses! Don't need to use the mount command on the 2nd node, ok got it. I see now in the replicated mode, when I access the /mnt/ from the server, e.g. du -sh /mnt, that syncs the data to the other nodes. Horacio, thank you for reminding including glusterd in the start up sequence. Actually glusterd was starting, but the fuse module wasn't included at first, but I fixed that. Thanks again! My best to you, ~~Rick - Original Message - From: "Horacio Sanson" To: gluster-users@gluster.org Sent: Monday, November 1, 2010 8:38:31 PM Subject: Re: [Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication On Tuesday 02 November 2010 10:04:27 Rick King wrote: > Ed, thank you for your response! > > >> Are you examining the second node directly, ie not by mounting it? > > This is an interesting question. I am just examining the 2nd node directly, > it wasn't obvious to me that the 2nd node needed to mount the data from > the 1st node. I was just merely expecting the data to be replicate to the > 2nd node. So my rationale is thinking I should run the following command > on the 2nd node: > > mount -t glusterfs hostnameA:/test /mnt > You do not need to mount the data from the first node in the second. As I understand GlusterFS works on the client side. What this means is that you must mount the volume in a client machine using either the glusterfs or native NFS drivers and then when you add a file to the mounted volume it will be replicated to both nodes. Writing directly on one of the server nodes storage (e.g. not through a mount point) will also replicate the file eventually due to GlusterFS self-heal mechanism but this will take longer to take effect. You can always force the replication with the volume rebalance command that is what you are seeing. I had a similar problem with data not being replicated even when using a volume mount point and the problem was that the glusterd daemon was not running in one of the nodes. Make sure the daemon is started: /etc/init.d/glusted start also make sure you configure it to start on system boot (it is not by default). You can check the manual for details: http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Configuring_glusterd_to_Start_Automatically > > The commands I used to create the volume from the server (hostnameA) > > 1) gluster volume create test replica 2 transport tcp hostnameA:/opt > hostnameB:/opt > > 2) gluster volume start test > > 3) mount -t glusterfs hostnameA:/test /mnt > Again this third step is not necessary. Make sure the glusterd daemons are running in all nodes and that you are accessing the volume through the glusterfs client or native NFS client. > Someone sent a message regarding a tutorial that I haven't read yet, so I > am going to work through that tutorial, and see if I can answer some of my > own questions. :) > > Thank you again Ed for the tidbit regarding the latency issue, and your > comment regarding HTPC applications. > > ~~Rick > - Original Message ----- > From: "Ed W" > To: "Gluster Users" > Sent: Monday, November 1, 2010 2:29:57 PM > Subject: Re: [Gluster-users] Possible to use gluster w/ email services + > Tuning for fast replication > > > Right now, I am testing out a 2 node setup, with one server replicating > > data to another node. One thing I noticed was when I created a file or > > directory on the server, the new data does not replicate to the other > > node. The only time data is synced from server to the other node is when > > I run "gluster volume rebalance test start". Is this normal? I had > > envisioned gluster would constantly replicate changes from the server to > > the other nodes, am I off base? > > Are you examining the second node directly, ie not by mounting it? I > think the point is that replication only happens when you "observe" the > second node? > > Glusterfs is targeted for HTPC applications where typically the nodes > are all connected over high performance interlinks. It appears that > performance degrades very quickly as the latency between nodes increases > and so whether the solution works for you is largely going to be > determined by the latency between nodes on your network connection. > > I'm not actually sure what some representative numbers should be? I > have two machines hooked up using bonded-rr intel gigabit cards > (crossover to each other) and these ping at around 0.3ms. However, I > have one other machine on a gigabit connection, hooked up to a switch > and that sometimes drops to around 0.15ms... I believe infiniband will > drop t
Re: [Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication
Ed, thank you for your response!
>> Are you examining the second node directly, ie not by mounting it?
This is an interesting question. I am just examining the 2nd node directly, it
wasn't obvious to me that the 2nd node needed to mount the data from the 1st
node. I was just merely expecting the data to be replicate to the 2nd node. So
my rationale is thinking I should run the following command on the 2nd node:
mount -t glusterfs hostnameA:/test /mnt
The commands I used to create the volume from the server (hostnameA)
1) gluster volume create test replica 2 transport tcp hostnameA:/opt
hostnameB:/opt
2) gluster volume start test
3) mount -t glusterfs hostnameA:/test /mnt
Someone sent a message regarding a tutorial that I haven't read yet, so I am
going to work through that tutorial, and see if I can answer some of my own
questions. :)
Thank you again Ed for the tidbit regarding the latency issue, and your comment
regarding HTPC applications.
~~Rick
- Original Message -
From: "Ed W"
To: "Gluster Users"
Sent: Monday, November 1, 2010 2:29:57 PM
Subject: Re: [Gluster-users] Possible to use gluster w/ email services + Tuning
for fast replication
> Right now, I am testing out a 2 node setup, with one server replicating data
> to another node. One thing I noticed was when I created a file or directory
> on the server, the new data does not replicate to the other node. The only
> time data is synced from server to the other node is when I run "gluster
> volume rebalance test start". Is this normal? I had envisioned gluster would
> constantly replicate changes from the server to the other nodes, am I off
> base?
Are you examining the second node directly, ie not by mounting it? I
think the point is that replication only happens when you "observe" the
second node?
Glusterfs is targeted for HTPC applications where typically the nodes
are all connected over high performance interlinks. It appears that
performance degrades very quickly as the latency between nodes increases
and so whether the solution works for you is largely going to be
determined by the latency between nodes on your network connection.
I'm not actually sure what some representative numbers should be? I
have two machines hooked up using bonded-rr intel gigabit cards
(crossover to each other) and these ping at around 0.3ms. However, I
have one other machine on a gigabit connection, hooked up to a switch
and that sometimes drops to around 0.15ms... I believe infiniband will
drop that latency to some few tens of microseconds?
So basically every file access on my system would suffer a 0.3ms access
latency. This is better than a spining disk with no cache which comes
in more like 3-10ms, but obviously it's still not brilliant
Please let us know how you get on?
Good luck
Ed W
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
DISCLAIMER: This e-mail and any files transmitted with it ('Message') is
intended only for the use of the recepient (s) named and may contain
confidential information. Opinions, conclusion and other information in this
message that do not relate to the official business of King7.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication
Ed, thank you for your response!
>> Are you examining the second node directly, ie not by mounting it?
This is an interesting question. I am just examining the 2nd node directly, it
wasn't obvious to me that the 2nd node needed to mount the data from the 1st
node. I was just merely expecting the data to be replicate to the 2nd node. So
my rationale is thinking I should run the following command on the 2nd node:
mount -t glusterfs hostnameA:/test /mnt
The commands I used to create the volume from the server (hostnameA)
1) gluster volume create test replica 2 transport tcp hostnameA:/opt
hostnameB:/opt
2) gluster volume start test
3) mount -t glusterfs hostnameA:/test /mnt
Someone sent a message regarding a tutorial that I haven't read yet, so I am
going to work through that tutorial, and see if I can answer some of my own
questions. :)
Thank you again Ed for the tidbit regarding the latency issue, and your comment
regarding HTPC applications.
~~Rick
- Original Message -
From: "Ed W"
To: "Gluster Users"
Sent: Monday, November 1, 2010 2:29:57 PM
Subject: Re: [Gluster-users] Possible to use gluster w/ email services + Tuning
for fast replication
> Right now, I am testing out a 2 node setup, with one server replicating data
> to another node. One thing I noticed was when I created a file or directory
> on the server, the new data does not replicate to the other node. The only
> time data is synced from server to the other node is when I run "gluster
> volume rebalance test start". Is this normal? I had envisioned gluster would
> constantly replicate changes from the server to the other nodes, am I off
> base?
Are you examining the second node directly, ie not by mounting it? I
think the point is that replication only happens when you "observe" the
second node?
Glusterfs is targeted for HTPC applications where typically the nodes
are all connected over high performance interlinks. It appears that
performance degrades very quickly as the latency between nodes increases
and so whether the solution works for you is largely going to be
determined by the latency between nodes on your network connection.
I'm not actually sure what some representative numbers should be? I
have two machines hooked up using bonded-rr intel gigabit cards
(crossover to each other) and these ping at around 0.3ms. However, I
have one other machine on a gigabit connection, hooked up to a switch
and that sometimes drops to around 0.15ms... I believe infiniband will
drop that latency to some few tens of microseconds?
So basically every file access on my system would suffer a 0.3ms access
latency. This is better than a spining disk with no cache which comes
in more like 3-10ms, but obviously it's still not brilliant
Please let us know how you get on?
Good luck
Ed W
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
DISCLAIMER: This e-mail and any files transmitted with it ('Message') is
intended only for the use of the recepient (s) named and may contain
confidential information. Opinions, conclusion and other information in this
message that do not relate to the official business of King7.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication
Right now, I am testing out a 2 node setup, with one server replicating data to another node. One thing I noticed was when I created a file or directory on the server, the new data does not replicate to the other node. The only time data is synced from server to the other node is when I run "gluster volume rebalance test start". Is this normal? I had envisioned gluster would constantly replicate changes from the server to the other nodes, am I off base? Are you examining the second node directly, ie not by mounting it? I think the point is that replication only happens when you "observe" the second node? Glusterfs is targeted for HTPC applications where typically the nodes are all connected over high performance interlinks. It appears that performance degrades very quickly as the latency between nodes increases and so whether the solution works for you is largely going to be determined by the latency between nodes on your network connection. I'm not actually sure what some representative numbers should be? I have two machines hooked up using bonded-rr intel gigabit cards (crossover to each other) and these ping at around 0.3ms. However, I have one other machine on a gigabit connection, hooked up to a switch and that sometimes drops to around 0.15ms... I believe infiniband will drop that latency to some few tens of microseconds? So basically every file access on my system would suffer a 0.3ms access latency. This is better than a spining disk with no cache which comes in more like 3-10ms, but obviously it's still not brilliant Please let us know how you get on? Good luck Ed W ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Possible to use gluster w/ email services + Tuning for fast replication
Gluster ver: glusterfs-core/rdma/fuse-3.1.0-1
OS: Centos 5.5 x86_64 (Two VMWare Nodes)
Hello All,
I am relatively new to gluster, I've started diving into Gluster for a couple
of weeks now. I was wondering if anyone is using Gluster with an email server,
e.g. postfix or sendmail. I am curious to know on how fast gluster replicates
files since MTA's do a lot of IO. I'd imagine some gluster tuning involved to
handle a lot files needing to be replicated.
Right now, I am testing out a 2 node setup, with one server replicating data to
another node. One thing I noticed was when I created a file or directory on the
server, the new data does not replicate to the other node. The only time data
is synced from server to the other node is when I run "gluster volume rebalance
test start". Is this normal? I had envisioned gluster would constantly
replicate changes from the server to the other nodes, am I off base?
The commands I used to create the volume from the server (hostnameA)
1) gluster volume create test replica 2 transport tcp hostnameA:/opt
hostnameB:/opt
2) gluster volume start test
3) mount -t glusterfs hostnameA:/test /mnt
My current volume info:
Volume Name: test
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: hostnameA:/opt
Brick2: hostnameB:/opt
Options Reconfigured:
performance.cache-size: 6GB
Greatly appreciate anyone has any pointers on how to improve replication.
My best to you,
~~Rick King
DISCLAIMER: This e-mail and any files transmitted with it ('Message') is
intended only for the use of the recepient (s) named and may contain
confidential information. Opinions, conclusion and other information in this
message that do not relate to the official business of King7.___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
