On Fri, Jun 23, 2017 at 9:10 AM, Pranith Kumar Karampuri < pkara...@redhat.com> wrote:
> > > On Fri, Jun 23, 2017 at 2:23 AM, Pat Haley <pha...@mit.edu> wrote: > >> >> Hi, >> >> Today we experimented with some of the FUSE options that we found in the >> list. >> >> Changing these options had no effect: >> >> gluster volume set test-volume performance.cache-max-file-size 2MB >> gluster volume set test-volume performance.cache-refresh-timeout 4 >> gluster volume set test-volume performance.cache-size 256MB >> gluster volume set test-volume performance.write-behind-window-size 4MB >> gluster volume set test-volume performance.write-behind-window-size 8MB >> >> > This is a good coincidence, I am meeting with write-behind > maintainer(+Raghavendra G) today for the same doubt. I think we will have > something by EOD IST. I will update you. > Sorry, forgot to update you. It seems like there is a bug in Write-behind and Facebook guys sent a patch http://review.gluster.org/16079 to fix the same. But even with that I am not seeing any improvement. May be I am doing something wrong. Will update you if I find anything more. > Changing the following option from its default value made the speed slower >> >> gluster volume set test-volume performance.write-behind off (on by default) >> >> Changing the following options initially appeared to give a 10% increase >> in speed, but this vanished in subsequent tests (we think the apparent >> increase may have been to a lighter workload on the computer from other >> users) >> >> gluster volume set test-volume performance.stat-prefetch on >> gluster volume set test-volume client.event-threads 4 >> gluster volume set test-volume server.event-threads 4 >> >> Can anything be gleaned from these observations? Are there other things >> we can try? >> >> Thanks >> >> Pat >> >> >> >> On 06/20/2017 12:06 PM, Pat Haley wrote: >> >> >> Hi Ben, >> >> Sorry this took so long, but we had a real-time forecasting exercise last >> week and I could only get to this now. >> >> Backend Hardware/OS: >> >> - Much of the information on our back end system is included at the >> top of http://lists.gluster.org/pipermail/gluster-users/2017-April/ >> 030529.html >> - The specific model of the hard disks is SeaGate ENTERPRISE CAPACITY >> V.4 6TB (ST6000NM0024). The rated speed is 6Gb/s. >> - Note: there is one physical server that hosts both the NFS and the >> GlusterFS areas >> >> Latest tests >> >> I have had time to run the tests for one of the dd tests you requested to >> the underlying XFS FS. The median rate was 170 MB/s. The dd results and >> iostat record are in >> >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestXFS/ >> >> I'll add tests for the other brick and to the NFS area later. >> >> Thanks >> >> Pat >> >> >> On 06/12/2017 06:06 PM, Ben Turner wrote: >> >> Ok you are correct, you have a pure distributed volume. IE no replication >> overhead. So normally for pure dist I use: >> >> throughput = slowest of disks / NIC * .6-.7 >> >> In your case we have: >> >> 1200 * .6 = 720 >> >> So you are seeing a little less throughput than I would expect in your >> configuration. What I like to do here is: >> >> -First tell me more about your back end storage, will it sustain 1200 MB / >> sec? What kind of HW? How many disks? What type and specs are the disks? >> What kind of RAID are you using? >> >> -Second can you refresh me on your workload? Are you doing reads / writes >> or both? If both what mix? Since we are using DD I assume you are working >> iwth large file sequential I/O, is this correct? >> >> -Run some DD tests on the back end XFS FS. I normally have >> /xfs-mount/gluster-brick, if you have something similar just mkdir on the >> XFS -> /xfs-mount/my-test-dir. Inside the test dir run: >> >> If you are focusing on a write workload run: >> >> # dd if=/dev/zero of=/xfs-mount/file bs=1024k count=10000 conv=fdatasync >> >> If you are focusing on a read workload run: >> >> # echo 3 > /proc/sys/vm/drop_caches >> # dd if=/gluster-mount/file of=/dev/null bs=1024k count=10000 >> >> ** MAKE SURE TO DROP CACHE IN BETWEEN READS!! ** >> >> Run this in a loop similar to how you did in: >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt >> >> Run this on both servers one at a time and if you are running on a SAN then >> run again on both at the same time. While this is running gather iostat for >> me: >> >> # iostat -c -m -x 1 > iostat-$(hostname).txt >> >> Lets see how the back end performs on both servers while capturing iostat, >> then see how the same workload / data looks on gluster. >> >> -Last thing, when you run your kernel NFS tests are you using the same >> filesystem / storage you are using for the gluster bricks? I want to be >> sure we have an apples to apples comparison here. >> >> -b >> >> >> >> ----- Original Message ----- >> >> From: "Pat Haley" <pha...@mit.edu> <pha...@mit.edu> >> To: "Ben Turner" <btur...@redhat.com> <btur...@redhat.com> >> Sent: Monday, June 12, 2017 5:18:07 PM >> Subject: Re: [Gluster-users] Slow write times to gluster disk >> >> >> Hi Ben, >> >> Here is the output: >> >> [root@mseas-data2 ~]# gluster volume info >> >> Volume Name: data-volume >> Type: Distribute >> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: mseas-data2:/mnt/brick1 >> Brick2: mseas-data2:/mnt/brick2 >> Options Reconfigured: >> nfs.exports-auth-enable: on >> diagnostics.brick-sys-log-level: WARNING >> performance.readdir-ahead: on >> nfs.disable: on >> nfs.export-volumes: off >> >> >> On 06/12/2017 05:01 PM, Ben Turner wrote: >> >> What is the output of gluster v info? That will tell us more about your >> config. >> >> -b >> >> ----- Original Message ----- >> >> From: "Pat Haley" <pha...@mit.edu> <pha...@mit.edu> >> To: "Ben Turner" <btur...@redhat.com> <btur...@redhat.com> >> Sent: Monday, June 12, 2017 4:54:00 PM >> Subject: Re: [Gluster-users] Slow write times to gluster disk >> >> >> Hi Ben, >> >> I guess I'm confused about what you mean by replication. If I look at >> the underlying bricks I only ever have a single copy of any file. It >> either resides on one brick or the other (directories exist on both >> bricks but not files). We are not using gluster for redundancy (or at >> least that wasn't our intent). Is that what you meant by replication >> or is it something else? >> >> Thanks >> >> Pat >> >> On 06/12/2017 04:28 PM, Ben Turner wrote: >> >> ----- Original Message ----- >> >> From: "Pat Haley" <pha...@mit.edu> <pha...@mit.edu> >> To: "Ben Turner" <btur...@redhat.com> <btur...@redhat.com>, "Pranith Kumar >> Karampuri"<pkara...@redhat.com> <pkara...@redhat.com> >> Cc: "Ravishankar N" <ravishan...@redhat.com> <ravishan...@redhat.com>, >> gluster-users@gluster.org, >> "Steve Postma" <spos...@ztechnet.com> <spos...@ztechnet.com> >> Sent: Monday, June 12, 2017 2:35:41 PM >> Subject: Re: [Gluster-users] Slow write times to gluster disk >> >> >> Hi Guys, >> >> I was wondering what our next steps should be to solve the slow write >> times. >> >> Recently I was debugging a large code and writing a lot of output at >> every time step. When I tried writing to our gluster disks, it was >> taking over a day to do a single time step whereas if I had the same >> program (same hardware, network) write to our nfs disk the time per >> time-step was about 45 minutes. What we are shooting for here would be >> to have similar times to either gluster of nfs. >> >> I can see in your test: >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt >> >> You averaged ~600 MB / sec(expected for replica 2 with 10G, {~1200 MB / >> sec} / #replicas{2} = 600). Gluster does client side replication so with >> replica 2 you will only ever see 1/2 the speed of your slowest part of >> the >> stack(NW, disk, RAM, CPU). This is usually NW or disk and 600 is >> normally >> a best case. Now in your output I do see the instances where you went >> down to 200 MB / sec. I can only explain this in three ways: >> >> 1. You are not using conv=fdatasync and writes are actually going to >> page >> cache and then being flushed to disk. During the fsync the memory is not >> yet available and the disks are busy flushing dirty pages. >> 2. Your storage RAID group is shared across multiple LUNS(like in a SAN) >> and when write times are slow the RAID group is busy serviceing other >> LUNs. >> 3. Gluster bug / config issue / some other unknown unknown. >> >> So I see 2 issues here: >> >> 1. NFS does in 45 minutes what gluster can do in 24 hours. >> 2. Sometimes your throughput drops dramatically. >> >> WRT #1 - have a look at my estimates above. My formula for guestimating >> gluster perf is: throughput = NIC throughput or storage(whatever is >> slower) / # replicas * overhead(figure .7 or .8). Also the larger the >> record size the better for glusterfs mounts, I normally like to be at >> LEAST 64k up to 1024k: >> >> # dd if=/dev/zero of=/gluster-mount/file bs=1024k count=10000 >> conv=fdatasync >> >> WRT #2 - Again, I question your testing and your storage config. Try >> using >> conv=fdatasync for your DDs, use a larger record size, and make sure that >> your back end storage is not causing your slowdowns. Also remember that >> with replica 2 you will take ~50% hit on writes because the client uses >> 50% of its bandwidth to write to one replica and 50% to the other. >> >> -b >> >> >> >> >> Thanks >> >> Pat >> >> >> On 06/02/2017 01:07 AM, Ben Turner wrote: >> >> Are you sure using conv=sync is what you want? I normally use >> conv=fdatasync, I'll look up the difference between the two and see if >> it >> affects your test. >> >> >> -b >> >> ----- Original Message ----- >> >> From: "Pat Haley" <pha...@mit.edu> <pha...@mit.edu> >> To: "Pranith Kumar Karampuri" <pkara...@redhat.com> <pkara...@redhat.com> >> Cc: "Ravishankar N" <ravishan...@redhat.com> >> <ravishan...@redhat.com>,gluster-users@gluster.org, >> "Steve Postma" <spos...@ztechnet.com> <spos...@ztechnet.com>, "Ben >> Turner" <btur...@redhat.com> <btur...@redhat.com> >> Sent: Tuesday, May 30, 2017 9:40:34 PM >> Subject: Re: [Gluster-users] Slow write times to gluster disk >> >> >> Hi Pranith, >> >> The "dd" command was: >> >> dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync >> >> There were 2 instances where dd reported 22 seconds. The output from >> the >> dd tests are in >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt >> >> Pat >> >> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote: >> >> Pat, >> What is the command you used? As per the following output, >> it >> seems like at least one write operation took 16 seconds. Which is >> really bad. >> 96.39 1165.10 us 89.00 us*16487014.00 us* >> 393212 >> WRITE >> >> >> On Tue, May 30, 2017 at 10:36 PM, Pat Haley >> <pha...@mit.edu<mailto:pha...@mit.edu> <pha...@mit.edu>> wrote: >> >> >> Hi Pranith, >> >> I ran the same 'dd' test both in the gluster test volume and >> in >> the .glusterfs directory of each brick. The median results >> (12 >> dd >> trials in each test) are similar to before >> >> * gluster test volume: 586.5 MB/s >> * bricks (in .glusterfs): 1.4 GB/s >> >> The profile for the gluster test-volume is in >> >> >> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt >> >> <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt> >> >> <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt> >> >> Thanks >> >> Pat >> >> >> >> >> On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote: >> >> Let's start with the same 'dd' test we were testing with to >> see, >> what the numbers are. Please provide profile numbers for the >> same. From there on we will start tuning the volume to see >> what >> we can do. >> >> On Tue, May 30, 2017 at 9:16 PM, Pat Haley <pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu>> wrote: >> >> >> Hi Pranith, >> >> Thanks for the tip. We now have the gluster volume >> mounted >> under /home. What tests do you recommend we run? >> >> Thanks >> >> Pat >> >> >> >> On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote: >> >> On Tue, May 16, 2017 at 9:20 PM, Pat Haley >> <pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu>> wrote: >> >> >> Hi Pranith, >> >> Sorry for the delay. I never saw received your >> reply >> (but I did receive Ben Turner's follow-up to your >> reply). So we tried to create a gluster volume >> under >> /home using different variations of >> >> gluster volume create test-volume >> mseas-data2:/home/gbrick_test_1 >> mseas-data2:/home/gbrick_test_2 transport tcp >> >> However we keep getting errors of the form >> >> Wrong brick type: transport, use >> <HOSTNAME>:<export-dir-abs-path> >> >> Any thoughts on what we're doing wrong? >> >> >> You should give transport tcp at the beginning I think. >> Anyways, transport tcp is the default, so no need to >> specify >> so remove those two words from the CLI. >> >> >> Also do you have a list of the test we should be >> running >> once we get this volume created? Given the >> time-zone >> difference it might help if we can run a small >> battery >> of tests and post the results rather than >> test-post-new >> test-post... . >> >> >> This is the first time I am doing performance analysis >> on >> users as far as I remember. In our team there are >> separate >> engineers who do these tests. Ben who replied earlier is >> one >> such engineer. >> >> Ben, >> Have any suggestions? >> >> >> Thanks >> >> Pat >> >> >> >> On 05/11/2017 12:06 PM, Pranith Kumar Karampuri >> wrote: >> >> On Thu, May 11, 2017 at 9:32 PM, Pat Haley >> <pha...@mit.edu <mailto:pha...@mit.edu> <pha...@mit.edu>> >> wrote: >> >> >> Hi Pranith, >> >> The /home partition is mounted as ext4 >> /home ext4 defaults,usrquota,grpquota 1 2 >> >> The brick partitions are mounted ax xfs >> /mnt/brick1 xfs defaults 0 0 >> /mnt/brick2 xfs defaults 0 0 >> >> Will this cause a problem with creating a >> volume >> under /home? >> >> >> I don't think the bottleneck is disk. You can do >> the >> same tests you did on your new volume to confirm? >> >> >> Pat >> >> >> >> On 05/11/2017 11:32 AM, Pranith Kumar Karampuri >> wrote: >> >> On Thu, May 11, 2017 at 8:57 PM, Pat Haley >> <pha...@mit.edu <mailto:pha...@mit.edu> <pha...@mit.edu>> >> wrote: >> >> >> Hi Pranith, >> >> Unfortunately, we don't have similar >> hardware >> for a small scale test. All we have is >> our >> production hardware. >> >> >> You said something about /home partition which >> has >> lesser disks, we can create plain distribute >> volume inside one of those directories. After >> we >> are done, we can remove the setup. What do you >> say? >> >> >> Pat >> >> >> >> >> On 05/11/2017 07:05 AM, Pranith Kumar >> Karampuri wrote: >> >> On Thu, May 11, 2017 at 2:48 AM, Pat >> Haley >> <pha...@mit.edu <mailto:pha...@mit.edu> >> <pha...@mit.edu>> >> wrote: >> >> >> Hi Pranith, >> >> Since we are mounting the partitions >> as >> the bricks, I tried the dd test >> writing >> to >> >> <brick-path>/.glusterfs/<file-to-be-removed-after-test>. >> The results without oflag=sync were >> 1.6 >> Gb/s (faster than gluster but not as >> fast >> as I was expecting given the 1.2 Gb/s >> to >> the no-gluster area w/ fewer disks). >> >> >> Okay, then 1.6Gb/s is what we need to >> target >> for, considering your volume is just >> distribute. Is there any way you can do >> tests >> on similar hardware but at a small scale? >> Just so we can run the workload to learn >> more >> about the bottlenecks in the system? We >> can >> probably try to get the speed to 1.2Gb/s >> on >> your /home partition you were telling me >> yesterday. Let me know if that is >> something >> you are okay to do. >> >> >> Pat >> >> >> >> On 05/10/2017 01:27 PM, Pranith Kumar >> Karampuri wrote: >> >> On Wed, May 10, 2017 at 10:15 PM, >> Pat >> Haley <pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu>> wrote: >> >> >> Hi Pranith, >> >> Not entirely sure (this isn't my >> area of expertise). I'll run >> your >> answer by some other people who >> are >> more familiar with this. >> >> I am also uncertain about how to >> interpret the results when we >> also >> add the dd tests writing to the >> /home area (no gluster, still on >> the >> same machine) >> >> * dd test without oflag=sync >> (rough average of multiple >> tests) >> o gluster w/ fuse mount : >> 570 >> Mb/s >> o gluster w/ nfs mount: >> 390 >> Mb/s >> o nfs (no gluster): 1.2 >> Gb/s >> * dd test with oflag=sync >> (rough >> average of multiple tests) >> o gluster w/ fuse mount: >> 5 >> Mb/s >> o gluster w/ nfs mount: >> 200 >> Mb/s >> o nfs (no gluster): 20 >> Mb/s >> >> Given that the non-gluster area >> is >> a >> RAID-6 of 4 disks while each >> brick >> of the gluster area is a RAID-6 >> of >> 32 disks, I would naively expect >> the >> writes to the gluster area to be >> roughly 8x faster than to the >> non-gluster. >> >> >> I think a better test is to try and >> write to a file using nfs without >> any >> gluster to a location that is not >> inside >> the brick but someother location >> that >> is >> on same disk(s). If you are mounting >> the >> partition as the brick, then we can >> write to a file inside .glusterfs >> directory, something like >> >> <brick-path>/.glusterfs/<file-to-be-removed-after-test>. >> >> >> >> I still think we have a speed >> issue, >> I can't tell if fuse vs nfs is >> part >> of the problem. >> >> >> I got interested in the post because >> I >> read that fuse speed is lesser than >> nfs >> speed which is counter-intuitive to >> my >> understanding. So wanted >> clarifications. >> Now that I got my clarifications >> where >> fuse outperformed nfs without sync, >> we >> can resume testing as described >> above >> and try to find what it is. Based on >> your email-id I am guessing you are >> from >> Boston and I am from Bangalore so if >> you >> are okay with doing this debugging >> for >> multiple days because of timezones, >> I >> will be happy to help. Please be a >> bit >> patient with me, I am under a >> release >> crunch but I am very curious with >> the >> problem you posted. >> >> Was there anything useful in the >> profiles? >> >> >> Unfortunately profiles didn't help >> me >> much, I think we are collecting the >> profiles from an active volume, so >> it >> has a lot of information that is not >> pertaining to dd so it is difficult >> to >> find the contributions of dd. So I >> went >> through your post again and found >> something I didn't pay much >> attention >> to >> earlier i.e. oflag=sync, so did my >> own >> tests on my setup with FUSE so sent >> that >> reply. >> >> >> Pat >> >> >> >> On 05/10/2017 12:15 PM, Pranith >> Kumar Karampuri wrote: >> >> Okay good. At least this >> validates >> my doubts. Handling O_SYNC in >> gluster NFS and fuse is a bit >> different. >> When application opens a file >> with >> O_SYNC on fuse mount then each >> write syscall has to be written >> to >> disk as part of the syscall >> where >> as in case of NFS, there is no >> concept of open. NFS performs >> write >> though a handle saying it needs >> to >> be a synchronous write, so >> write() >> syscall is performed first then >> it >> performs fsync(). so an write >> on >> an >> fd with O_SYNC becomes >> write+fsync. >> I am suspecting that when >> multiple >> threads do this write+fsync() >> operation on the same file, >> multiple writes are batched >> together to be written do disk >> so >> the throughput on the disk is >> increasing is my guess. >> >> Does it answer your doubts? >> >> On Wed, May 10, 2017 at 9:35 >> PM, >> Pat Haley <pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu>> >> wrote: >> >> >> Without the oflag=sync and >> only >> a single test of each, the >> FUSE >> is going faster than NFS: >> >> FUSE: >> mseas-data2(dri_nascar)% dd >> if=/dev/zero count=4096 >> bs=1048576 of=zeros.txt >> conv=sync >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) >> copied, 7.46961 s, 575 MB/s >> >> >> NFS >> mseas-data2(HYCOM)% dd >> if=/dev/zero count=4096 >> bs=1048576 of=zeros.txt >> conv=sync >> 4096+0 records in >> 4096+0 records out >> 4294967296 bytes (4.3 GB) >> copied, 11.4264 s, 376 MB/s >> >> >> >> On 05/10/2017 11:53 AM, >> Pranith >> Kumar Karampuri wrote: >> >> Could you let me know the >> speed without oflag=sync >> on >> both the mounts? No need >> to >> collect profiles. >> >> On Wed, May 10, 2017 at >> 9:17 >> PM, Pat Haley >> <pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu>> >> wrote: >> >> >> Here is what I see >> now: >> >> [root@mseas-data2 ~]# >> gluster volume info >> >> Volume Name: >> data-volume >> Type: Distribute >> Volume ID: >> c162161e-2a2d-4dac-b015-f31fd89ceb18 >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: >> mseas-data2:/mnt/brick1 >> Brick2: >> mseas-data2:/mnt/brick2 >> Options Reconfigured: >> diagnostics.count-fop-hits: >> on >> diagnostics.latency-measurement: >> on >> nfs.exports-auth-enable: >> on >> diagnostics.brick-sys-log-level: >> WARNING >> performance.readdir-ahead: >> on >> nfs.disable: on >> nfs.export-volumes: >> off >> >> >> >> On 05/10/2017 11:44 >> AM, >> Pranith Kumar >> Karampuri >> wrote: >> >> Is this the volume >> info >> you have? >> >> >/[root at >> >mseas-data2 >> >> <http://www.gluster.org/mailman/listinfo/gluster-users> >> <http://www.gluster.org/mailman/listinfo/gluster-users> >> ~]# gluster volume >> info >> />//>/Volume Name: >> data-volume />/Type: >> Distribute />/Volume >> ID: >> c162161e-2a2d-4dac-b015-f31fd89ceb18 >> />/Status: Started >> />/Number >> of Bricks: 2 >> />/Transport-type: >> tcp >> />/Bricks: />/Brick1: >> mseas-data2:/mnt/brick1 >> />/Brick2: >> mseas-data2:/mnt/brick2 >> />/Options >> Reconfigured: >> />/performance.readdir-ahead: >> on />/nfs.disable: on >> />/nfs.export-volumes: >> off >> / >> I copied this from >> old >> thread from 2016. >> This >> is >> distribute volume. >> Did >> you change any of the >> options in between? >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> >> <pha...@mit.edu> >> Center for Ocean >> Engineering >> Phone: (617) 253-6824 >> Dept. of Mechanical >> Engineering >> Fax: (617) 253-8125 >> MIT, Room >> 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts >> Avenue >> Cambridge, MA >> 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean >> Engineering >> Phone: (617) 253-6824 >> Dept. of Mechanical >> Engineering >> Fax: (617) 253-8125 >> MIT, Room >> 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering >> Phone: >> (617) 253-6824 >> Dept. of Mechanical Engineering >> Fax: >> (617) 253-8125 >> MIT, Room >> 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering >> Phone: >> (617) 253-6824 >> Dept. of Mechanical Engineering >> Fax: >> (617) 253-8125 >> MIT, Room >> 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering Phone: >> (617) >> 253-6824 >> Dept. of Mechanical Engineering Fax: >> (617) >> 253-8125 >> MIT, Room >> 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering Phone: >> (617) >> 253-6824 >> Dept. of Mechanical Engineering Fax: >> (617) >> 253-8125 >> MIT, Room 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> -- >> Pranith >> >> -- >> >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley >> Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering Phone: (617) >> 253-6824 >> Dept. of Mechanical Engineering Fax: (617) >> 253-8125 >> MIT, Room 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> >> -- >> Pranith >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> >> -- >> Pranith >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email:pha...@mit.edu >> <mailto:pha...@mit.edu> <pha...@mit.edu> >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> >> -- >> Pranith >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> >> >> _______________________________________________ >> Gluster-users mailing >> listGluster-users@gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: pha...@mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> > > > -- > Pranith > -- Pranith
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users