The daily log rotation. -Ben
On Wed, Aug 30, 2017 at 3:09 PM, Bryan Banister <bbanis...@jumptrading.com> wrote: > Looking at the systemd service it does show that twice, at roughly the > same time and one day apart, the service did receive a HUP signal: > > Aug 29 16:31:02 carf-ceph-osd02 radosgw[130050]: 2017-08-29 > 16:31:02.528559 7fffc641c700 -1 received signal: Hangup from PID: 73176 > task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse > radosgw UID: 0 > > Aug 30 16:32:02 carf-ceph-osd02 radosgw[130050]: 2017-08-30 > 16:32:02.529825 7fffc641c700 -1 received signal: Hangup from PID: 48062 > task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse > radosgw UID: 0 > > > > Any idea what would do this? > > > > I'll be updating the version to 12.2.0 shortly, > > -Bryan > > > > -----Original Message----- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Bryan Banister > Sent: Wednesday, August 30, 2017 3:42 PM > To: Yehuda Sadeh-Weinraub <yeh...@redhat.com> > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4 > > > > Note: External Email > > ------------------------------------------------- > > > > We are not sending a HUP signal that we know about. We were not modifying > our configuration. However all user accounts in the RGW were lost! > > -Bryan > > > > -----Original Message----- > > From: Yehuda Sadeh-Weinraub [mailto:yeh...@redhat.com <yeh...@redhat.com>] > > Sent: Wednesday, August 30, 2017 3:30 PM > > To: Bryan Banister <bbanis...@jumptrading.com> > > Cc: ceph-users@lists.ceph.com > > Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4 > > > > Note: External Email > > ------------------------------------------------- > > > > On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister > > <bbanis...@jumptrading.com> wrote: > > > Not sure what’s happening but we started to but a decent load on the > RGWs we > > > have setup and we were seeing failures with the following kind of > > > fingerprint: > > > > > > > > > > > > 2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends > > > paused > > > > > > > Are you modifying configuration? Could be that something is sending > > HUP singal to the radosgw process. We disabled this behavior (process > > dynamic reconfig after HUP) in 12.2.0. > > > > Yehuda > > > > > 2017-08-29 17:06:22.072359 7fffacbe9700 1 civetweb: 0x555556add000: > > > 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT > > > /blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" > 1 0 > > > - Minio (linux; amd64) minio-go/3.0.0 > > > > > > 2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard, > > > completion_mgr.get_next() returned ret=-125 > > > > > > 2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store > closed > > > > > > 2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog > > > history: buffer::end_of_buffer > > > > > > 2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: > (5) > > > Input/output error > > > > > > 2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating > new > > > store > > > > > > 2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register > > > rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version > 12.1.4 > > > (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R) > > > Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat > > > Enterprise Linux Server 7.3 > > > (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80 > > > num_threads=1024,frontend_type#0=civetweb,hos > > > > > > tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT > > > 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_ > swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id= > b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01, > zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us} > > > > > > 2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing > > > initialization of new store > > > > > > 2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST > > > subsystem init > > > > > > 2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user > > > subsystem init > > > > > > 2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user > > > subsystem init > > > > > > 2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage > > > subsystem init > > > > > > 2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming > > > frontends with new realm configuration. > > > > > > 2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request > > > req=0x7fffad3e4190 ===== > > > > > > 2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request > > > req=0x7fffacbe3190 ===== > > > > > > 2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done > > > req=0x7fffacbe3190 op status=0 http_status=403 ====== > > > > > > 2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done > > > req=0x7fffad3e4190 op status=0 http_status=403 ====== > > > > > > > > > > > > Any help understanding how to fix this would be greatly appreciated! > > > > > > -Bryan > > > > > > > > > ________________________________ > > > > > > Note: This email is for the confidential use of the named addressee(s) > only > > > and may contain proprietary, confidential or privileged information. If > you > > > are not the intended recipient, you are hereby notified that any review, > > > dissemination or copying of this email is strictly prohibited, and to > please > > > notify the sender immediately and destroy this email and any attachments. > > > Email transmission cannot be guaranteed to be secure or error-free. The > > > Company, therefore, does not make any guarantees as to the completeness > or > > > accuracy of this email or any attachments. This email is for > informational > > > purposes only and does not constitute a recommendation, offer, request or > > > solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > > > of transaction of a financial product. > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ------------------------------ > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
