Re: Log files with 0.45
I checked with Sam on this and it turns out he created a new subsystem whose output you can control with the "debug optracker" (or --debug-optracker) option (in the same way as the other debug log systems). In 0.45 the output for that system was at inappropriately high levels (1) and it's fixed in our current master (5 now), but you probably want to set "debug optracker = 0" in your config. That should restore things to the way they used to be! (And sorry for the long wait, Danny.) -Greg On Friday, April 20, 2012 at 1:05 PM, Nick Bartos wrote: > Is there a recommended log config for production systems? I'm also > trying to decrease the verbosity in 0.45, using the options specified > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > '1' doesn't end the insane log sprawl I'm seeing. > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > mailto:gregory.far...@dreamhost.com)> wrote: > > On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > > > Hi, > > > > > > did something change with the default log levels for OSDs on v0.45? With > > > 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > > > get by default ~3Gbyte per OSD over 12 hours without any change in the > > > config file. > > > > > > Danny > > I think some extra event notifications got stuck in the logs for OSD > > operations; they're nice for debugging but may well have a log level higher > > than they should. They should be easy to compress a lot, though! > > > > Can you comment on this, Sam? > > -Greg > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majord...@vger.kernel.org > > (mailto:majord...@vger.kernel.org) > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > (mailto:majord...@vger.kernel.org) > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Log files with 0.45
On Fri, 20 Apr 2012, Calvin Morrow wrote: > I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of > logs. I had to change my log rotate config to compress based on size > instead of once a day or I ended up with a full root partition. > > I would love to know if there's a better way to handle it. This will be fixed in 0.46. You can also cherry-pick ac874a62533c3845f9e68201bfd4f7020332da24 from the stable branch to fix this. You can also set 'debug optracker = 0' to turn this down. sage > > Calvin > > On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos wrote: > > Is there a recommended log config for production systems? I'm also > > trying to decrease the verbosity in 0.45, using the options specified > > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > > '1' doesn't end the insane log sprawl I'm seeing. > > > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > > wrote: > >> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > >>> Hi, > >>> > >>> did something change with the default log levels for OSDs on v0.45? With > >>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > >>> get by default ~3Gbyte per OSD over 12 hours without any change in the > >>> config file. > >>> > >>> Danny > >> I think some extra event notifications got stuck in the logs for OSD > >> operations; they're nice for debugging but may well have a log level > >> higher than they should. They should be easy to compress a lot, though! > >> > >> Can you comment on this, Sam? > >> -Greg > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majord...@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majord...@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > >
Re: Log files with 0.45
I'm seeing the same. < 12 hours with 6 OSDs resulted in ~18 GB of logs. I had to change my log rotate config to compress based on size instead of once a day or I ended up with a full root partition. I would love to know if there's a better way to handle it. Calvin On Fri, Apr 20, 2012 at 5:05 PM, Nick Bartos wrote: > Is there a recommended log config for production systems? I'm also > trying to decrease the verbosity in 0.45, using the options specified > here: http://ceph.newdream.net/wiki/Debugging. Setting them down to > '1' doesn't end the insane log sprawl I'm seeing. > > On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum > wrote: >> On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: >>> Hi, >>> >>> did something change with the default log levels for OSDs on v0.45? With >>> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I >>> get by default ~3Gbyte per OSD over 12 hours without any change in the >>> config file. >>> >>> Danny >> I think some extra event notifications got stuck in the logs for OSD >> operations; they're nice for debugging but may well have a log level higher >> than they should. They should be easy to compress a lot, though! >> >> Can you comment on this, Sam? >> -Greg >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Log files with 0.45
Is there a recommended log config for production systems? I'm also trying to decrease the verbosity in 0.45, using the options specified here: http://ceph.newdream.net/wiki/Debugging. Setting them down to '1' doesn't end the insane log sprawl I'm seeing. On Tue, Apr 17, 2012 at 10:09 PM, Greg Farnum wrote: > On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: >> Hi, >> >> did something change with the default log levels for OSDs on v0.45? With >> 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I >> get by default ~3Gbyte per OSD over 12 hours without any change in the >> config file. >> >> Danny > I think some extra event notifications got stuck in the logs for OSD > operations; they're nice for debugging but may well have a log level higher > than they should. They should be easy to compress a lot, though! > > Can you comment on this, Sam? > -Greg > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Log files with 0.45
On Tuesday, April 17, 2012 at 9:53 PM, Danny Kukawka wrote: > Hi, > > did something change with the default log levels for OSDs on v0.45? With > 0.43 and IIRC also 0.44* the logfiles had a acceptable size, but now I > get by default ~3Gbyte per OSD over 12 hours without any change in the > config file. > > Danny I think some extra event notifications got stuck in the logs for OSD operations; they're nice for debugging but may well have a log level higher than they should. They should be easy to compress a lot, though! Can you comment on this, Sam? -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
