Re: [ceph-users] XFS attempt to access beyond end of device
Blair,
I should clarify that I am *now* aware of your support case =D
For anyone willing to run a systemtap the following should give us more
information about the problem.
stap --all-modules -e 'probe kernel.function("handle_bad_sector"){
printf("handle_bad_sector(): ARGS is %s\n",$$parms$$); print_backtrace()}'
In order to run this you will need to install some non-trivial packages such as
the kernel debuginfo package and kernel-devel. This is generally best
accomplished as follows, at least on rpm based systems.
(yum|dnf) install systemtap
stap-prep
The systemtap needs to be running when the error is generated as it monitors
calls to "handle_bad_sector" which is the function generating the error message.
Once that function is called the probe will dump all information about the bio
structure passed as a parameter to "handle_bad_sector" as well as dumping the
call stack. This would give us a good idea of the specific code involved.
On Sat, Jul 22, 2017 at 9:45 AM, Brad Hubbard wrote:
> On Sat, Jul 22, 2017 at 9:38 AM, Blair Bethwaite
> wrote:
>> Hi Brad,
>>
>> On 22 July 2017 at 09:04, Brad Hubbard wrote:
>>> Could you share what kernel/distro you are running and also please test
>>> whether
>>> the error message can be triggered by running the "blkid" command?
>>
>> I'm seeing it on RHEL7.3 (3.10.0-514.2.2.el7.x86_64). See Red Hat
>> support case #01891011 for sosreport etc.
>
> Thanks Blair,
>
> I'm aware of your case and the Bugzilla created from it and we are
> investigating further.
>
>>
>> No, blkid does not seem to trigger it. So far I haven't figured out
>> what does. It seems to be showing up roughly once for each disk every
>
> Thanks, that appears to exclude any link to an existing Bugzilla that
> was suggested as being related.
>
>> 1-2 weeks, and there is a clear time correlation across the hosts
>> experiencing it.
>>
>> --
>> Cheers,
>> ~Blairo
>
>
>
> --
> Cheers,
> Brad
--
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] dealing with incomplete PGs while using bluestore
Thats exactly what I am doing the only difference is that i didnt need
to do step1 since for me the dev was already mounted in
/var/lib/ceph/ceph-### but remaining steps are exactly what i am doing.
it seems to me the PG got corrupted in my case in all copies and thats
what causing it to refuse to start witht he imported PG
Glad it worked for you
I am now marking the PGs as complete (data loss)
it will be long night today :)
On 7/22/2017 8:03 PM, Daniel K wrote:
I am in the process of doing exactly what you are -- this worked for me:
1. mount the first partition of the bluestore drive that holds the
missing PGs (if it's not already mounted)
> mkdir /mnt/tmp
> mount /dev/sdb1 /mnt/tmp
2. export the pg to a suitable temporary storage location:
> ceph-objectstore-tool --data-path /mnt/tmp --pgid 1.24 --op export
--file /mnt/sdd1/recover.1.24
3. find the acting osd
> ceph health detail |grep incomplete
PG_DEGRADED Degraded data redundancy: 23 pgs unclean, 23 pgs incomplete
pg 1.24 is incomplete, acting [18,13]
pg 4.1f is incomplete, acting [11]
...
4. set noout
> ceph osd set noout
5. Find the OSD and log into it -- I used 18 here.
> ceph osd find 18
{
"osd": 18,
"ip": "10.0.15.54:6801/9263 ",
"crush_location": {
"building": "building-dc",
"chassis": "chassis-dc400f5-10",
"city": "city",
"floor": "floor-dc4",
"host": "stor-vm4",
"rack": "rack-dc400f5",
"region": "cfl",
"room": "room-dc400",
"root": "default",
"row": "row-dc400f"
}
}
> ssh user@10.0.15.54
6. copy the file to somewhere accessible by the new(acting) osd
> scp user@10.0.14.51:/mnt/sdd1/recover.1.24 /tmp/recover.1.24
7. stop the osd
> service ceph-osd@18 stop
8. import the file using ceph-objectstore-tool
> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-18 --op
import --file /tmp/recover.1.24
9. start the osd
> service-osd@18 start
this worked for me -- not sure if this is the best way or if I took
any extra steps and I have yet to validate that the data is good.
I based this partially off your original email, and the guide here
http://ceph.com/geen-categorie/incomplete-pgs-oh-my/
On Sat, Jul 22, 2017 at 4:46 PM, mofta7y > wrote:
Hi All,
I have a situation here.
I have an EC pool that is having cache tier pool (the cache tier
is replicated with size 2).
Had an issue on the pool and the crush map got changed after
rebooting some OSD in any case I lost 4 cache ties OSDs
those lost OSDs are not really lost they look fine to me but
bluestore is giving me exception when starting them i cant deal
with it. (will open question about that exception as well)
So now i have 14 incomplete Pgs on the caching tier.
I am trying to recover them using ceph-objectstore-tool
the extraction and import works nice with no issues but the OSD
fail to start after wards with same issue as the original OSD .
after importing the PG on the acting OSD i get the exact same
exception I was getting while trying to start the failed OSD
removing that import resolve the issue.
So the question is how can use ceph-objectstore-tool to import in
bluestore as i think i am missing somthing here
here is the procedure and the steps i used
1- stop old osd (it cannot start anyway)
2- use this command to extract the pg i need
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-116
--pgid 15.371 --op export --file /tmp/recover.15.371
that command work
3- check what is the acting OSD for the pg
4- stop the acting OSD
5- delete the current folder with same og name
6- use this command
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-78 --op
import /tmp/recover.15.371
the error i got in both cases is this bluestore error
Jul 22 16:35:20 alm9 ceph-osd[3799171]: -257> 2017-07-22
16:20:19.544195 7f7157036a40 -1 osd.116 119691 log_to_monitors
{default=true}
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 0> 2017-07-22
16:35:20.142143 7f713c597700 -1
/tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc: In
function 'virtual int BitMapAllocator::reserve(uint64_t)' thread
7f713c597700 time 2017-07-22 16:35:20.139309
Jul 22 16:35:20 alm9 ceph-osd[3799171]:
/tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc: 82:
FAILED assert(!(need % m_block_size))
Jul 22 16:35:20 alm9 ceph-osd[3799171]: ceph version 11.2.0
(f223e27eeb35991352ebc1f67423d4ebc252adb7)
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x562b84558380]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 2:
(BitMapAllocator::reserve(unsigned long)+0x2ab) [0x562b8437c5cb]
Jul 22
Re: [ceph-users] dealing with incomplete PGs while using bluestore
I am in the process of doing exactly what you are -- this worked for me:
1. mount the first partition of the bluestore drive that holds the missing
PGs (if it's not already mounted)
> mkdir /mnt/tmp
> mount /dev/sdb1 /mnt/tmp
2. export the pg to a suitable temporary storage location:
> ceph-objectstore-tool --data-path /mnt/tmp --pgid 1.24 --op export --file
/mnt/sdd1/recover.1.24
3. find the acting osd
> ceph health detail |grep incomplete
PG_DEGRADED Degraded data redundancy: 23 pgs unclean, 23 pgs incomplete
pg 1.24 is incomplete, acting [18,13]
pg 4.1f is incomplete, acting [11]
...
4. set noout
> ceph osd set noout
5. Find the OSD and log into it -- I used 18 here.
> ceph osd find 18
{
"osd": 18,
"ip": "10.0.15.54:6801/9263",
"crush_location": {
"building": "building-dc",
"chassis": "chassis-dc400f5-10",
"city": "city",
"floor": "floor-dc4",
"host": "stor-vm4",
"rack": "rack-dc400f5",
"region": "cfl",
"room": "room-dc400",
"root": "default",
"row": "row-dc400f"
}
}
> ssh user@10.0.15.54
6. copy the file to somewhere accessible by the new(acting) osd
> scp user@10.0.14.51:/mnt/sdd1/recover.1.24 /tmp/recover.1.24
7. stop the osd
> service ceph-osd@18 stop
8. import the file using ceph-objectstore-tool
> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-18 --op import
--file /tmp/recover.1.24
9. start the osd
> service-osd@18 start
this worked for me -- not sure if this is the best way or if I took any
extra steps and I have yet to validate that the data is good.
I based this partially off your original email, and the guide here
http://ceph.com/geen-categorie/incomplete-pgs-oh-my/
On Sat, Jul 22, 2017 at 4:46 PM, mofta7y wrote:
> Hi All,
>
> I have a situation here.
>
> I have an EC pool that is having cache tier pool (the cache tier is
> replicated with size 2).
>
> Had an issue on the pool and the crush map got changed after rebooting
> some OSD in any case I lost 4 cache ties OSDs
>
> those lost OSDs are not really lost they look fine to me but bluestore is
> giving me exception when starting them i cant deal with it. (will open
> question about that exception as well)
>
> So now i have 14 incomplete Pgs on the caching tier.
>
>
> I am trying to recover them using ceph-objectstore-tool
>
> the extraction and import works nice with no issues but the OSD fail to
> start after wards with same issue as the original OSD .
>
> after importing the PG on the acting OSD i get the exact same exception I
> was getting while trying to start the failed OSD
>
> removing that import resolve the issue.
>
>
> So the question is how can use ceph-objectstore-tool to import in
> bluestore as i think i am missing somthing here
>
>
> here is the procedure and the steps i used
>
> 1- stop old osd (it cannot start anyway)
>
> 2- use this command to extract the pg i need
>
> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-116 --pgid 15.371
> --op export --file /tmp/recover.15.371
>
> that command work
>
> 3- check what is the acting OSD for the pg
>
> 4- stop the acting OSD
>
> 5- delete the current folder with same og name
>
> 6- use this command
>
> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-78 --op import
> /tmp/recover.15.371
> the error i got in both cases is this bluestore error
>
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: -257> 2017-07-22 16:20:19.544195
> 7f7157036a40 -1 osd.116 119691 log_to_monitors {default=true}
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 0> 2017-07-22 16:35:20.142143
> 7f713c597700 -1 /tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc:
> In function 'virtual int BitMapAllocator::reserve(uint64_t)' thread
> 7f713c597700 time 2017-07-22 16:35:20.139309
> Jul 22 16:35:20 alm9 ceph-osd[3799171]:
> /tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc:
> 82: FAILED assert(!(need % m_block_size))
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: ceph version 11.2.0
> (f223e27eeb35991352ebc1f67423d4ebc252adb7)
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x80) [0x562b84558380]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 2: (BitMapAllocator::reserve(unsigned
> long)+0x2ab) [0x562b8437c5cb]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 3: (BlueFS::reclaim_blocks(unsigned
> int, unsigned long, std::vector mempool::pool_allocator<(mempool::pool_index_t)7,
> AllocExtent> >*)+0x22a) [0x562b8435109a]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 4: (BlueStore::_balance_bluefs_fr
> eespace(std::vector >*)+0x28e) [0x562b84270dae]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 5:
> (BlueStore::_kv_sync_thread()+0x164a)
> [0x562b84273eea]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 6:
> (BlueStore::KVSyncThread::entry()+0xd)
> [0x562b842ad9dd]
> Jul 22 16:35:20 alm9 ceph-osd[3799171]: 7: (()+0x76ba)
[ceph-users] dealing with incomplete PGs while using bluestore
Hi All,
I have a situation here.
I have an EC pool that is having cache tier pool (the cache tier is
replicated with size 2).
Had an issue on the pool and the crush map got changed after rebooting
some OSD in any case I lost 4 cache ties OSDs
those lost OSDs are not really lost they look fine to me but bluestore
is giving me exception when starting them i cant deal with it. (will
open question about that exception as well)
So now i have 14 incomplete Pgs on the caching tier.
I am trying to recover them using ceph-objectstore-tool
the extraction and import works nice with no issues but the OSD fail to
start after wards with same issue as the original OSD .
after importing the PG on the acting OSD i get the exact same exception
I was getting while trying to start the failed OSD
removing that import resolve the issue.
So the question is how can use ceph-objectstore-tool to import in
bluestore as i think i am missing somthing here
here is the procedure and the steps i used
1- stop old osd (it cannot start anyway)
2- use this command to extract the pg i need
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-116 --pgid
15.371 --op export --file /tmp/recover.15.371
that command work
3- check what is the acting OSD for the pg
4- stop the acting OSD
5- delete the current folder with same og name
6- use this command
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-78 --op import
/tmp/recover.15.371
the error i got in both cases is this bluestore error
Jul 22 16:35:20 alm9 ceph-osd[3799171]: -257> 2017-07-22
16:20:19.544195 7f7157036a40 -1 osd.116 119691 log_to_monitors
{default=true}
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 0> 2017-07-22
16:35:20.142143 7f713c597700 -1
/tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc: In function
'virtual int BitMapAllocator::reserve(uint64_t)' thread 7f713c597700
time 2017-07-22 16:35:20.139309
Jul 22 16:35:20 alm9 ceph-osd[3799171]:
/tmp/buildd/ceph-11.2.0/src/os/bluestore/BitMapAllocator.cc: 82: FAILED
assert(!(need % m_block_size))
Jul 22 16:35:20 alm9 ceph-osd[3799171]: ceph version 11.2.0
(f223e27eeb35991352ebc1f67423d4ebc252adb7)
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 1:
(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x562b84558380]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 2:
(BitMapAllocator::reserve(unsigned long)+0x2ab) [0x562b8437c5cb]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 3:
(BlueFS::reclaim_blocks(unsigned int, unsigned long,
std::vector
>*)+0x22a) [0x562b8435109a]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 4:
(BlueStore::_balance_bluefs_freespace(std::vector*)+0x28e) [0x562b84270dae]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 5:
(BlueStore::_kv_sync_thread()+0x164a) [0x562b84273eea]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 6:
(BlueStore::KVSyncThread::entry()+0xd) [0x562b842ad9dd]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 7: (()+0x76ba) [0x7f71560c76ba]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: 8: (clone()+0x6d) [0x7f71547953dd]
Jul 22 16:35:20 alm9 ceph-osd[3799171]: NOTE: a copy of the executable,
or `objdump -rdS ` is needed to interpret this.
if any one have any idea how to restore those PGs please point me to the
right direction
by the way resarting the folder that i deleted in step5 manually make
the osd go up again
Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Luminous: ceph mgr crate error - mon disconnected
Hi,
I have upgraded from kraken version with a simple "yum upgrade command".
Later the upgrade, I'd like to deploy the mgr daemon on one node of my ceph
infrastrucute.
But, for any reason, It gets stuck!
Let's see the complete set of commands:
[root@vdicnode01 ~]# ceph -s
cluster:
id: 656e84b2-9192-40fe-9b81-39bd0c7a3196
health: HEALTH_WARN
*no active mgr*
services:
mon: 1 daemons, quorum vdicnode01
mgr: no daemons active
osd: 2 osds: 2 up, 2 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs:
[root@vdicnode01 ~]# su - vdicceph
Last login: Sat Jul 22 12:50:38 CEST 2017 on pts/0
[vdicceph@vdicnode01 ~]$ cd ceph
*[vdicceph@vdicnode01 ceph]$ ceph-deploy --username vdicceph -v mgr create
vdicnode02.local*
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/vdicceph/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.38): /bin/ceph-deploy --username
vdicceph -v mgr create vdicnode02.local
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : vdicceph
[ceph_deploy.cli][INFO ] verbose : True
[ceph_deploy.cli][INFO ] mgr :
[('vdicnode02.local', 'vdicnode02.local')]
[ceph_deploy.cli][INFO ] overwrite_conf: False
[ceph_deploy.cli][INFO ] subcommand: create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func :
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts
vdicnode02.local:vdicnode02.local
[vdicnode02.local][DEBUG ] connection detected need for sudo
[vdicnode02.local][DEBUG ] connected to host: vdicceph@vdicnode02.local
[vdicnode02.local][DEBUG ] detect platform information from remote host
[vdicnode02.local][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vdicnode02.local
[vdicnode02.local][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[vdicnode02.local][DEBUG ] create path if it doesn't exist
[vdicnode02.local][INFO ] Running command: sudo ceph --cluster ceph --name
client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring
auth get-or-create mgr.vdicnode02.local mon allow profile mgr osd allow *
mds allow * -o /var/lib/ceph/mgr/ceph-vdicnode02.local/keyring
[vdicnode02.local][WARNIN] No data was received after 300 seconds,
disconnecting...
[vdicnode02.local][INFO ] Running command: sudo systemctl enable
ceph-mgr@vdicnode02.local
[vdicnode02.local][WARNIN] Created symlink from
/etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@vdicnode02.local.service
to /usr/lib/systemd/system/ceph-mgr@.service.
[vdicnode02.local][INFO ] Running command: sudo systemctl start
ceph-mgr@vdicnode02.local
[vdicnode02.local][INFO ] Running command: sudo systemctl enable
ceph.target
*[vdicceph@vdicnode01 ceph]$ sudo ceph -s --verbose --watch-warn
--watch-error*
parsed_args: Namespace(admin_socket=None, admin_socket_nope=None,
cephconf=None, client_id=None, client_name=None, cluster=None,
cluster_timeout=None, completion=False, help=False, input_file=None,
output_file=None, output_format=None, status=True, verbose=True,
version=False, watch=False, watch_channel='cluster', watch_debug=False,
watch_error=True, watch_info=False, watch_sec=False, watch_warn=True),
childargs: []
< no response for ever >
Anybody has experienced the same issue? how can I make my ceph work again?
Thanks a lot.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
