OK. Found some loglevel overrides in the monitor and reset them. Restarted the mgr and monitor just in case.
Still getting a lot of stuff that looks like this. Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+0000 7f36d7291700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compact> Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+0000 7f36d7291700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compact> Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+0000 7f36d7291700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compact> Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:51.314+0000 7f36d7291700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compact> Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:49.542670+0000 mgr.xyz1 (mgr.6889303) 177 : cluster [DBG] pgmap v160: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:10:51 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:51.542+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v161: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:10:52 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:51.543403+0000 mgr.xyz1 (mgr.6889303) 178 : cluster [DBG] pgmap v161: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:10:52 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:52.748+0000 7fa1c74a2700 0 [progress INFO root] Processing OSDMap change 20239..20239 Dec 19 17:10:53 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:53.544+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v162: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:10:54 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:53.545649+0000 mgr.xyz1 (mgr.6889303) 179 : cluster [DBG] pgmap v162: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:10:55 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:55.545+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v163: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:10:55 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:10:55.834+0000 7f36de29f700 1 mon.xyz1@1(peon).osd e20239 _set_new_cache_sizes cache_size:1020054731 inc_a> Dec 19 17:10:56 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:55.546197+0000 mgr.xyz1 (mgr.6889303) 180 : cluster [DBG] pgmap v163: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:10:57 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:57.546+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v164: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:10:57 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:57.751+0000 7fa1c74a2700 0 [progress INFO root] Processing OSDMap change 20239..20239 Dec 19 17:10:58 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:57.547657+0000 mgr.xyz1 (mgr.6889303) 181 : cluster [DBG] pgmap v164: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:10:59 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:10:59.548+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v165: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:11:00 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: ::ffff:10.0.1.2 - - [19/Dec/2023:22:11:00] "GET /metrics HTTP/1.1" 200 215073 "" "Prometheus/2.33.4" Dec 19 17:11:00 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:11:00.762+0000 7fa1b6e42700 0 [prometheus INFO cherrypy.access.140332751105776] ::ffff:10.0.1.2 - - [19/De> Dec 19 17:11:00 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: debug 2023-12-19T22:11:00.835+0000 7f36de29f700 1 mon.xyz1@1(peon).osd e20239 _set_new_cache_sizes cache_size:1020054731 inc_a> Dec 19 17:11:00 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:10:59.549152+0000 mgr.xyz1 (mgr.6889303) 182 : cluster [DBG] pgmap v165: 649 pgs: 1 active+clean+scrubbin> Dec 19 17:11:01 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mgr-xyz1[1906755]: debug 2023-12-19T22:11:01.548+0000 7fa1d9fc7700 0 log_channel(cluster) log [DBG] : pgmap v166: 649 pgs: 1 active+clean+scrubbi> Dec 19 17:11:02 xyz1.mousetech.com ceph-278fcd86-0861-11ee-a7df- 9c5c8e86cf8f-mon-xyz1[1906961]: cluster 2023-12-19T22:11:01.549680+0000 mgr.xyz1 (mgr.6889303) 183 : cluster [DBG] pgmap v166: 649 pgs: 1 active+clean+scrubbin> On Tue, 2023-12-19 at 16:21 -0500, Wesley Dillingham wrote: > "ceph daemon" commands need to be run local to the machine where the > daemon > is running. So in this case if you arent on the node where osd.1 > lives it > wouldnt work. "ceph tell" should work anywhere there is a > client.admin key. > > > Respectfully, > > *Wes Dillingham* > w...@wesdillingham.com > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > > > On Tue, Dec 19, 2023 at 4:02 PM Tim Holloway <t...@mousetech.com> > wrote: > > > Ceph version is Pacific (16.2.14), upgraded from a sloppy Octopus. > > > > I ran afoul of all the best bugs in Octopus, and in the process > > switched on a lot of stuff better left alone, including some > > detailed > > debug logging. Now I can't turn it off. > > > > I am confidently informed by the documentation that the first step > > would be the command: > > > > ceph daemon osd.1 config show | less > > > > But instead of config information I get back: > > > > Can't get admin socket path: unable to get conf option admin_socket > > for > > osd: b"error parsing 'osd': expected string of the form TYPE.ID, > > valid > > types are: auth, mon, osd, mds, mgr, client\n" > > > > Which seems to be kind of insane. > > > > Attempting to get daemon config info on a monitor on that machine > > gives: > > > > admin_socket: exception getting command descriptions: [Errno 2] No > > such > > file or directory > > > > Which doesn't help either. > > > > Anyone got an idea? > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
