[Gluster-devel] geo-replication issue
Hi I just know someone must have seen this before Geo-replication on 3.6.1, epel, Centos 6.6 Popen: ssh [2014-12-08 22:57:31.433468] E [socket.c:2267:socket_connect_finish] 0-glusterfs: connection to 127.0.0.1:24007 failed (Connection refused) Popen: ssh [2014-12-08 22:57:31.433484] T [socket.c:723:__socket_disconnect] 0-glusterfs: disconnecting 0x22c6550, state=0 gen=0 sock=6 Popen: ssh [2014-12-08 22:57:31.433497] D [socket.c:704:__socket_shutdown] 0-glusterfs: shutdown() returned -1. Transport endpoint is not connected Popen: ssh [2014-12-08 22:57:31.433506] D [socket.c:2344:socket_event_handler] 0-transport: disconnecting now Popen: ssh [2014-12-08 22:57:31.433541] T [cli.c:270:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT Is my google-fu deserting me in my hour of need? BR Jan From: Gong XiaoHui xhg...@wind.com.cn Date: Monday 08 December 2014 at 7:45 AM To: 'gluster-devel@gluster.org' gluster-devel@gluster.org Subject: [Gluster-devel] some issues about geo-replication and gfapi Hi I have two questions with glusterfs: 1.When I use libgfapi to write a file to a volume, which is configured as a geo-replication masterVol. It cannot work well, the geo status is faulty, following is the log: The glusterfs version is 3.6.1 [2014-12-08 13:12:11.708616] I [master(/data_xfs/geo2-master):1330:crawl] _GMaster: finished hybrid crawl syncing, stime: (1418010314, 0) [2014-12-08 13:12:11.709706] I [master(/data_xfs/geo2-master):480:crawlwrap] _GMaster: primary master with volume id d220647c-5730-4cef-a89b-932470c914d2 ... [2014-12-08 13:12:11.735719] I [master(/data_xfs/geo2-master):491:crawlwrap] _GMaster: crawl interval: 3 seconds [2014-12-08 13:12:11.811722] I [master(/data_xfs/geo2-master):1182:crawl] _GMaster: slave's time: (1418010314, 0) [2014-12-08 13:12:11.840826] E [repce(/data_xfs/geo2-master):207:__call__] RepceClient: call 8318:139656072095488:1418015531.84 (entry_ops) failed on peer with OSError [2014-12-08 13:12:11.840990] E [syncdutils(/data_xfs/geo2-master):270:log_raise_exception] top: FAIL: Traceback (most recent call last): File /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py, line 164, in main main_i() File /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py, line 643, in main_i local.service_loop(*[r for r in [remote] if r]) File /usr/libexec/glusterfs/python/syncdaemon/resource.py, line 1344, in service_loop g2.crawlwrap() File /usr/libexec/glusterfs/python/syncdaemon/master.py, line 529, in crawlwrap self.crawl(no_stime_update=no_stime_update) File /usr/libexec/glusterfs/python/syncdaemon/master.py, line 1194, in crawl self.process(changes) File /usr/libexec/glusterfs/python/syncdaemon/master.py, line 946, in process self.process_change(change, done, retry) File /usr/libexec/glusterfs/python/syncdaemon/master.py, line 910, in process_change self.slave.server.entry_ops(entries) File /usr/libexec/glusterfs/python/syncdaemon/repce.py, line 226, in __call__ return self.ins(self.meth, *a) File /usr/libexec/glusterfs/python/syncdaemon/repce.py, line 208, in __call__ raise res OSError: [Errno 12] Cannot allocate memory [2014-12-08 13:12:11.842178] I [syncdutils(/data_xfs/geo2-master):214:finalize] top: exiting. [2014-12-08 13:12:11.843421] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF. [2014-12-08 13:12:11.843682] I [syncdutils(agent):214:finalize] top: exiting. [2014-12-08 13:12:12.103255] I [monitor(monitor):222:monitor] Monitor: worker(/data_xfs/geo2-master) died in startup phase 2.Another question, I cannot configure a geo-replication in 3.6.1 with the method in 3.4.1. Any response is appreciate. 产品及时服务：请在Wind资讯终端上按“F1”键或致电客服专线400-820-Wind(9463) --- 宫晓辉 技术X01部 上海万得信息技术股份有限公司（Wind资讯） Shanghai Wind Information Co., Ltd. 上海市浦东新区福山路33号建工大厦9楼 200120 9/F Jian Gong Mansion,33 Fushan Road, Pudong New Area, Shanghai, P.R.C. 200120 Tel: (0086 21)6888 2280*8310 Fax: (0086 21)6888 2281 Email: xhg...@wind.com.cn http://www.wind.com.cn ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
Are you referring to something else in your request? Meaning, you want /myfile, /dir1/myfile and /dir2/dir3/myfile to fall onto the same bricks/subvolumes and that perchance is what you are looking for? That is EXACTLY what I am looking for. What are my chances? BR Jan ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
I think I have it. Unless I’m totally confused, I can hash ONLY on the filename with: glusterfs --volfile-server=a_server --volfile-id=a_volume \ --xlator-option a_volume-dht.extra_hash_regex='.*[/]' \ /a/mountpoint Correct? Jan From: Jan H Holtzhausen j...@holtztech.info Date: Tuesday 25 November 2014 at 9:06 PM To: gluster-devel@gluster.org Subject: Re: [Gluster-devel] EHT / DHT Are you referring to something else in your request? Meaning, you want /myfile, /dir1/myfile and /dir2/dir3/myfile to fall onto the same bricks/subvolumes and that perchance is what you are looking for? That is EXACTLY what I am looking for. What are my chances? BR Jan ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
Hmm Then something is wrong, If I upload 2 identical files, with different paths they only end up on the same server 1/4 of the time (I have 4 bricks). I’ll test the regex quickly. BR Jan On 2014/11/25, 7:55 PM, Shyam srang...@redhat.com wrote: On 11/25/2014 02:28 PM, Jan H Holtzhausen wrote: I think I have it. Unless I’m totally confused, I can hash ONLY on the filename with: glusterfs --volfile-server=a_server --volfile-id=a_volume \ --xlator-option a_volume-dht.extra_hash_regex='.*[/]' \ /a/mountpoint Correct? The hash of a file does not include the full path, it is on the file name _only_. So any regex will not work when the filename remains constant like myfile. As Jeff explains the option is really to prevent using temporary parts of the name in the hash computation (for rename optimization). In this case, you do not seem to have any tmp parts to the name, like myfile and myfile~ should evaluate to the same hash, so remove all trailing '~' from the name. So I am not sure the above is the option you are looking for. Jan From: Jan H Holtzhausen j...@holtztech.info mailto:j...@holtztech.info Date: Tuesday 25 November 2014 at 9:06 PM To: gluster-devel@gluster.org mailto:gluster-devel@gluster.org Subject: Re: [Gluster-devel] EHT / DHT Are you referring to something else in your request? Meaning, you want /myfile, /dir1/myfile and /dir2/dir3/myfile to fall onto the same bricks/subvolumes and that perchance is what you are looking for? That is EXACTLY whatI am looking for. What are my chances? As far as I know not much out of the box. As Jeff explained, the directory distribution/layout considers the GFID of the directory, hence each of the directories in the above example would/could get different ranges. The file on the other hand remains constant myfile so its hash value remains the same, but due to the distribution range change as above for the directories, it will land on different bricks and not the same one. Out of curiosity, why is this functionality needed? Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
So in a distributed cluster, the GFID tells all bricks what a files preceding directory structure looks like? Where the physical file is saved is a function of the filename ONLY. Therefore My requirement should be met by default, or am I being dense? BR Jan On 2014/11/25, 8:15 PM, Shyam srang...@redhat.com wrote: On 11/25/2014 03:11 PM, Jan H Holtzhausen wrote: STILL doesn’t work … exact same file ends up on 2 different bricks … I must be missing something. All I need is for: /directory1/subdirectory2/foo And /directory2/subdirectoryaaa999/foo To end up on the same brick…. This is not possible is what I was attempting to state in the previous mail. The regex filter is not for this purpose. The hash is always based on the name of the file, but the location is based on the distribution/layout of the directory, which is different for each directory based on its GFID. So there are no options in the code to enable what you seek at present. Why is this needed? Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
As to the why. Filesystem cache hits. Files with the same name tend to be the same files. Regards Jan On 2014/11/25, 8:42 PM, Jan H Holtzhausen j...@holtztech.info wrote: So in a distributed cluster, the GFID tells all bricks what a files preceding directory structure looks like? Where the physical file is saved is a function of the filename ONLY. Therefore My requirement should be met by default, or am I being dense? BR Jan On 2014/11/25, 8:15 PM, Shyam srang...@redhat.com wrote: On 11/25/2014 03:11 PM, Jan H Holtzhausen wrote: STILL doesn’t work … exact same file ends up on 2 different bricks … I must be missing something. All I need is for: /directory1/subdirectory2/foo And /directory2/subdirectoryaaa999/foo To end up on the same brick…. This is not possible is what I was attempting to state in the previous mail. The regex filter is not for this purpose. The hash is always based on the name of the file, but the location is based on the distribution/layout of the directory, which is different for each directory based on its GFID. So there are no options in the code to enable what you seek at present. Why is this needed? Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] EHT / DHT
I could tell you… But Symantec wouldn’t like it….. From: Poornima Gurusiddaiah pguru...@redhat.com Date: Wednesday 26 November 2014 at 7:16 AM To: Jan H Holtzhausen j...@holtztech.info Cc: gluster-devel@gluster.org Subject: Re: [Gluster-devel] EHT / DHT Out of curiosity, what back end and deduplication solution are you using? Regards, Poornima From: Jan H Holtzhausen j...@holtztech.info To: Anand Avati av...@gluster.org, Shyam srang...@redhat.com, gluster-devel@gluster.org Sent: Wednesday, November 26, 2014 3:43:36 AM Subject: Re: [Gluster-devel] EHT / DHT Yes we have deduplication at the filesystem layer BR Jan From: Anand Avati av...@gluster.org Date: Wednesday 26 November 2014 at 12:11 AM To: Jan H Holtzhausen j...@holtztech.info, Shyam srang...@redhat.com, gluster-devel@gluster.org Subject: Re: [Gluster-devel] EHT / DHT Unless there is some sort of de-duplication under the covers happening in the brick, or the files are hardlinks to each other, there is no cache benefit whatsoever by having identical files placed on the same server. Thanks, Avati On Tue Nov 25 2014 at 12:59:25 PM Jan H Holtzhausen j...@holtztech.info wrote: As to the why. Filesystem cache hits. Files with the same name tend to be the same files. Regards Jan On 2014/11/25, 8:42 PM, Jan H Holtzhausen j...@holtztech.info wrote: So in a distributed cluster, the GFID tells all bricks what a files preceding directory structure looks like? Where the physical file is saved is a function of the filename ONLY. Therefore My requirement should be met by default, or am I being dense? BR Jan On 2014/11/25, 8:15 PM, Shyam srang...@redhat.com wrote: On 11/25/2014 03:11 PM, Jan H Holtzhausen wrote: STILL doesn’t work … exact same file ends up on 2 different bricks … I must be missing something. All I need is for: /directory1/subdirectory2/foo And /directory2/subdirectoryaaa999/foo To end up on the same brick…. This is not possible is what I was attempting to state in the previous mail. The regex filter is not for this purpose. The hash is always based on the name of the file, but the location is based on the distribution/layout of the directory, which is different for each directory based on its GFID. So there are no options in the code to enable what you seek at present. Why is this needed? Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
