Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
To confirm: Joe's explanation, uncomfortable as it, looks to be the correct one. When the servers were powered off and restarted (so the gluster processes /had/ to be restarted, the new ones started up and used the correct time format. The 'problem' clients were the ones which were running the updated version; when all the clients were forced to restart the glusterfs; they all appear to be running with the UTC time (tho hard to tell since the number of logged incidents has fallen dramatically). There is a repeating entry in all the server logs tho: 1001 $ tail -3 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2013-12-11 18:26:41.824180] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-12-11 18:26:44.825089] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-12-11 18:26:47.825952] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) Is there a way to detect which client(s) this is coming from? On Tuesday, December 10, 2013 11:23:11 AM Joe Julian wrote: > If I were to hazard a guess, since the timestamp is not configurable and > *is* UTC in 3.4, it would seem that any server that's logging in local time > must not be running 3.4. Sure, it's installed, but the application hasn't > been restarted since it was installed. That's the only thing I can think of > that would allow that behavior. --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
On Tuesday, December 10, 2013 11:23:11 AM Joe Julian wrote: > If I were to hazard a guess, since the timestamp is not configurable and > *is* UTC in 3.4, it would seem that any server that's logging in local time > must not be running 3.4. Sure, it's installed, but the application hasn't > been restarted since it was installed. That's the only thing I can think of > that would allow that behavior. _ Thanks for the info and insight, Joe. Well ... hmmm. I think you're right. For an unrelated reason, this morning, I stopped and restarted the gluster servers and NOW the logs indicate that the servers ARE using UTC. So if they were not using it before, they must have been using the old version ..? The RPMs stated the new version number but I did not check the version of the process that was running. We would have had to have upgraded and then NOT restarted the gluster daemons (or they failed to shut down properly and kept running). The servers weren't rebooted, since they had uptimes of almost a year when I shut them down this morning.. Hmmm again. Now all the clients are reporting UTC as well. ...!!?! hjm --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
On 12/10/2013 10:57 AM, harry mangalam wrote: On Tuesday, December 10, 2013 10:42:28 AM Vijay Bellur wrote: > On 12/10/2013 10:14 AM, harry mangalam wrote: > > Admittedly I should search the source, but I wonder if anyone knows this > > offhand. > > > > Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have > > picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - > > that date string is running about 5-6 hours ahead of the system date and > > all the Gluster servers (which are identical and correct). The time > > advancement does not appear to be identical tho it's hard to tell since > > it only shows on errors and those update irregularly. > > The timestamps in the log file are by default in UTC. That could > possibly explain why the timestamps look advanced in the log file. that seems to make sense. The advanced time on the 4 problems nodes looks to be the correct UTC time, but the others are using /local time/ in their logs, for some reason. And localtime nodes are the ones NOT having problems. ...??! However, this looks to be more of a ROCKS / config problem than a general gluster problem at this point. All the nodes have the md5-identical /etc/localtime, but they seem to be behaving differently as to the logging. Thanks for the pointer. hjm > > All the clients are the same version and all the servers are the same > > (gluster v 3.4.0-8.el6.x86_64 > > > > This would not be of interest except that those 4 clients are losing > > files, unable to reliably do IO, etc on the gluster fs. They don't > > appear to be having problems with NFS mounts, nor with a Fraunhofer FS > > that is also mounted on each node, > > Do you observe anything in the client log files of these machines that > indicate I/O problems? Yes. If I were to hazard a guess, since the timestamp is not configurable and *is* UTC in 3.4, it would seem that any server that's logging in local time must not be running 3.4. Sure, it's installed, but the application hasn't been restarted since it was installed. That's the only thing I can think of that would allow that behavior. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
On Tuesday, December 10, 2013 10:42:28 AM Vijay Bellur wrote: > On 12/10/2013 10:14 AM, harry mangalam wrote: > > Admittedly I should search the source, but I wonder if anyone knows this > > offhand. > > > > Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have > > picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - > > that date string is running about 5-6 hours ahead of the system date and > > all the Gluster servers (which are identical and correct). The time > > advancement does not appear to be identical tho it's hard to tell since > > it only shows on errors and those update irregularly. > > The timestamps in the log file are by default in UTC. That could > possibly explain why the timestamps look advanced in the log file. that seems to make sense. The advanced time on the 4 problems nodes looks to be the correct UTC time, but the others are using /local time/ in their logs, for some reason. And localtime nodes are the ones NOT having problems. ...??! However, this looks to be more of a ROCKS / config problem than a general gluster problem at this point. All the nodes have the md5-identical /etc/localtime, but they seem to be behaving differently as to the logging. Thanks for the pointer. hjm > > All the clients are the same version and all the servers are the same > > (gluster v 3.4.0-8.el6.x86_64 > > > > This would not be of interest except that those 4 clients are losing > > files, unable to reliably do IO, etc on the gluster fs. They don't > > appear to be having problems with NFS mounts, nor with a Fraunhofer FS > > that is also mounted on each node, > > Do you observe anything in the client log files of these machines that > indicate I/O problems? Yes. > > Thanks, > Vijay > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
On Tuesday, December 10, 2013 12:49:25 PM Sharuzzaman Ahmat Raslan wrote: > Hi Harry, > > Did you setup ntp on each of the node, and sync the time to one single > source? Yes, this is done by ROCKS and all the nodes have the identical time. (2admins have checked repeatedly) > Thanks. > > On Tue, Dec 10, 2013 at 12:44 PM, harry mangalam wrote: > > Admittedly I should search the source, but I wonder if anyone knows this > > > > offhand. > > > > > > > > Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have > > picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - that > > date string is running about 5-6 hours ahead of the system date and all > > the > > Gluster servers (which are identical and correct). The time advancement > > does not appear to be identical tho it's hard to tell since it only shows > > on errors and those update irregularly. > > > > > > > > All the clients are the same version and all the servers are the same > > (gluster v 3.4.0-8.el6.x86_64 > > > > > > > > This would not be of interest except that those 4 clients are losing > > files, unable to reliably do IO, etc on the gluster fs. They don't appear > > to be having problems with NFS mounts, nor with a Fraunhofer FS that is > > also mounted on each node, > > > > > > > > Rebooting 2 of them has no effect - they come right back with an advanced > > date. > > > > > > > > > > > > --- > > > > Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine > > > > [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 > > > > 415 South Circle View Dr, Irvine, CA, 92697 [shipping] > > > > MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) > > > > --- > > > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://supercolony.gluster.org/mailman/listinfo/gluster-users --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
On 12/10/2013 10:14 AM, harry mangalam wrote: Admittedly I should search the source, but I wonder if anyone knows this offhand. Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - that date string is running about 5-6 hours ahead of the system date and all the Gluster servers (which are identical and correct). The time advancement does not appear to be identical tho it's hard to tell since it only shows on errors and those update irregularly. The timestamps in the log file are by default in UTC. That could possibly explain why the timestamps look advanced in the log file. All the clients are the same version and all the servers are the same (gluster v 3.4.0-8.el6.x86_64 This would not be of interest except that those 4 clients are losing files, unable to reliably do IO, etc on the gluster fs. They don't appear to be having problems with NFS mounts, nor with a Fraunhofer FS that is also mounted on each node, Do you observe anything in the client log files of these machines that indicate I/O problems? Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
Hi Harry, Did you setup ntp on each of the node, and sync the time to one single source? Thanks. On Tue, Dec 10, 2013 at 12:44 PM, harry mangalam wrote: > Admittedly I should search the source, but I wonder if anyone knows this > offhand. > > > > Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have > picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - that > date string is running about 5-6 hours ahead of the system date and all the > Gluster servers (which are identical and correct). The time advancement > does not appear to be identical tho it's hard to tell since it only shows > on errors and those update irregularly. > > > > All the clients are the same version and all the servers are the same > (gluster v 3.4.0-8.el6.x86_64 > > > > This would not be of interest except that those 4 clients are losing > files, unable to reliably do IO, etc on the gluster fs. They don't appear > to be having problems with NFS mounts, nor with a Fraunhofer FS that is > also mounted on each node, > > > > Rebooting 2 of them has no effect - they come right back with an advanced > date. > > > > > > --- > > Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine > > [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 > > 415 South Circle View Dr, Irvine, CA, 92697 [shipping] > > MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) > > --- > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -- Sharuzzaman Ahmat Raslan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?
Admittedly I should search the source, but I wonder if anyone knows this offhand. Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have picked up an 'advanced date' in the /var/log/glusterfs/gl.log file - that date string is running about 5-6 hours ahead of the system date and all the Gluster servers (which are identical and correct). The time advancement does not appear to be identical tho it's hard to tell since it only shows on errors and those update irregularly. All the clients are the same version and all the servers are the same (gluster v 3.4.0-8.el6.x86_64 This would not be of interest except that those 4 clients are losing files, unable to reliably do IO, etc on the gluster fs. They don't appear to be having problems with NFS mounts, nor with a Fraunhofer FS that is also mounted on each node, Rebooting 2 of them has no effect - they come right back with an advanced date. --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
