Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

2019-07-23 Thread Armin Ranjbar 
Thanks JC!

I can confirm that I was able to first upgrade to jewel, and then directly
upgrade to Luminous. thank you all!

---
Armin ranjbar



On Tue, Jul 23, 2019 at 1:22 AM JC Lopez  wrote:

> First link should be this one
> http://docs.ceph.com/docs/jewel/install/upgrading-ceph/#upgrade-procedures 
> rather
> than
> http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures to
> be consistent.
>
> JC
>
> On Jul 22, 2019, at 13:38, JC Lopez  wrote:
>
> Hi
>
> you’ll have to go from Hammer to Jewel then from Jewel to Luminous for a
> smooth upgrade.
> -
> http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures
> -
> http://docs.ceph.com/docs/luminous/release-notes/#upgrading-from-pre-jewel-releases-like-hammer
>
> Make sure to check any special upgrade requirement from the release notes.
> -
> http://docs.ceph.com/docs/jewel/release-notes/#upgrading-from-infernalis-or-hammer
> -
> http://docs.ceph.com/docs/luminous/release-notes/#upgrade-from-jewel-or-kraken
>
> Regards
> JC
>
> On Jul 22, 2019, at 12:20, Armin Ranjbar  wrote:
>
> Dear Everyone,
>
> First of all, guys, seriously, Thank you for Ceph.
>
> now to the problem, upgrading ceph from 0.94.6
> (e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f
> (9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to
> upgrade it's database, problem is gone if i --force-sync.
>
> This is the message:
> terminate called after throwing an instance of
> 'ceph::buffer::malformed_input'
>   what():  buffer::malformed_input: void
> object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of
> struct encoding
> *** Caught signal (Aborted) **
>
> attached is full log, the output of:
> ceph-mon --debug_mon 100 -i node-1 -d
>
> ---
> Armin ranjbar
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

2019-07-22 Thread JC Lopez 
First link should be this one 
http://docs.ceph.com/docs/jewel/install/upgrading-ceph/#upgrade-procedures 
 
rather than 
http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures 
 to 
be consistent.

JC

> On Jul 22, 2019, at 13:38, JC Lopez  wrote:
> 
> Hi 
> 
> you’ll have to go from Hammer to Jewel then from Jewel to Luminous for a 
> smooth upgrade.
> - http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures 
> 
> - 
> http://docs.ceph.com/docs/luminous/release-notes/#upgrading-from-pre-jewel-releases-like-hammer
>  
> 
> 
> Make sure to check any special upgrade requirement from the release notes.
> - 
> http://docs.ceph.com/docs/jewel/release-notes/#upgrading-from-infernalis-or-hammer
>  
> 
> - 
> http://docs.ceph.com/docs/luminous/release-notes/#upgrade-from-jewel-or-kraken
>  
> 
> 
> Regards
> JC
> 
>> On Jul 22, 2019, at 12:20, Armin Ranjbar > > wrote:
>> 
>> Dear Everyone,
>> 
>> First of all, guys, seriously, Thank you for Ceph.
>> 
>> now to the problem, upgrading ceph from 0.94.6 
>> (e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f 
>> (9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to upgrade 
>> it's database, problem is gone if i --force-sync.
>> 
>> This is the message:
>> terminate called after throwing an instance of 
>> 'ceph::buffer::malformed_input'
>>   what():  buffer::malformed_input: void 
>> object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of 
>> struct encoding
>> *** Caught signal (Aborted) **
>> 
>> attached is full log, the output of:
>> ceph-mon --debug_mon 100 -i node-1 -d
>> 
>> ---
>> Armin ranjbar
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

2019-07-22 Thread JC Lopez 
Hi 

you’ll have to go from Hammer to Jewel then from Jewel to Luminous for a smooth 
upgrade.
- http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures 

- 
http://docs.ceph.com/docs/luminous/release-notes/#upgrading-from-pre-jewel-releases-like-hammer
 


Make sure to check any special upgrade requirement from the release notes.
- 
http://docs.ceph.com/docs/jewel/release-notes/#upgrading-from-infernalis-or-hammer
 

- 
http://docs.ceph.com/docs/luminous/release-notes/#upgrade-from-jewel-or-kraken 


Regards
JC

> On Jul 22, 2019, at 12:20, Armin Ranjbar  wrote:
> 
> Dear Everyone,
> 
> First of all, guys, seriously, Thank you for Ceph.
> 
> now to the problem, upgrading ceph from 0.94.6 
> (e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f 
> (9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to upgrade 
> it's database, problem is gone if i --force-sync.
> 
> This is the message:
> terminate called after throwing an instance of 'ceph::buffer::malformed_input'
>   what():  buffer::malformed_input: void 
> object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of 
> struct encoding
> *** Caught signal (Aborted) **
> 
> attached is full log, the output of:
> ceph-mon --debug_mon 100 -i node-1 -d
> 
> ---
> Armin ranjbar
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] MON crashing when upgrading from Hammer to Luminous

2019-07-22 Thread Armin Ranjbar 
Dear Everyone,

First of all, guys, seriously, Thank you for Ceph.

now to the problem, upgrading ceph from 0.94.6
(e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f
(9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to
upgrade it's database, problem is gone if i --force-sync.

This is the message:
terminate called after throwing an instance of
'ceph::buffer::malformed_input'
  what():  buffer::malformed_input: void
object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of
struct encoding
*** Caught signal (Aborted) **

attached is full log, the output of:
ceph-mon --debug_mon 100 -i node-1 -d

---
Armin ranjbar
2019-07-22 19:17:54.429120 7f2064488f40  0 ceph version 12.2.12-218-g9fd889f 
(9fd889fe09c652512ca78854702d5ad9bf3059bb) luminous (stable), process ceph-mon, 
pid 908122
2019-07-22 19:17:54.429229 7f2064488f40  0 pidfile_write: ignore empty 
--pid-file
2019-07-22 19:17:54.438472 7f2064488f40  0 load: jerasure load: lrc load: isa 
2019-07-22 19:17:54.438908 7f2064488f40  1 leveldb: Recovering log #4402802
2019-07-22 19:17:54.489204 7f2064488f40  1 leveldb: Delete type=0 #4402802

2019-07-22 19:17:54.489263 7f2064488f40  1 leveldb: Delete type=3 #4402801

2019-07-22 19:17:54.489547 7f2064488f40 10 obtain_monmap
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
  what():  buffer::malformed_input: void 
object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of 
struct encoding
*** Caught signal (Aborted) **
 in thread 7f2064488f40 thread_name:ceph-mon
2019-07-22 19:17:54.489654 7f2064488f40 10 obtain_monmap read last committed 
monmap ver 3
2019-07-22 19:17:54.490558 7f2064488f40  0 starting mon.node-1 rank 2 at public 
addr 192.168.1.16:6789/0 at bind addr 192.168.1.16:6789/0 mon_data 
/var/lib/ceph/mon/ceph-node-1 fsid cf635990-70fa-43ed-978d-96f92f9ccc92
2019-07-22 19:17:54.490737 7f2064488f40  0 starting mon.node-1 rank 2 at 
192.168.1.16:6789/0 mon_data /var/lib/ceph/mon/ceph-node-1 fsid 
cf635990-70fa-43ed-978d-96f92f9ccc92
2019-07-22 19:17:54.491279 7f2064488f40  1 mon.node-1@-1(probing) e3 preinit 
fsid cf635990-70fa-43ed-978d-96f92f9ccc92
2019-07-22 19:17:54.491351 7f2064488f40 10 mon.node-1@-1(probing) e3 check_fsid 
cluster_uuid contains 'cf635990-70fa-43ed-978d-96f92f9ccc92'
2019-07-22 19:17:54.491363 7f2064488f40 10 mon.node-1@-1(probing) e3 features 
compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos 
with k/v store (v0.?),4=support erasure code pools,5=new-style osdmap 
encoding,6=support isa/lrc erasure code}
2019-07-22 19:17:54.491371 7f2064488f40 10 mon.node-1@-1(probing) e3 
calc_quorum_requirements required_features 18416819765248
2019-07-22 19:17:54.491374 7f2064488f40 10 mon.node-1@-1(probing) e3 
required_features 18416819765248
2019-07-22 19:17:54.491381 7f2064488f40 10 mon.node-1@-1(probing) e3 
has_ever_joined = 1
2019-07-22 19:17:54.491411 7f2064488f40 10 mon.node-1@-1(probing) e3 
sync_last_committed_floor 0
2019-07-22 19:17:54.491413 7f2064488f40 10 mon.node-1@-1(probing) e3 init_paxos
2019-07-22 19:17:54.491516 7f2064488f40  1 mon.node-1@-1(probing).mds e0 Unable 
to load 'last_metadata'
2019-07-22 19:17:54.491558 7f2064488f40 10 mon.node-1@-1(probing).health init
2019-07-22 19:17:54.491574 7f2064488f40 10 mon.node-1@-1(probing) e3 
refresh_from_paxos
2019-07-22 19:17:54.491608 7f2064488f40  1 
mon.node-1@-1(probing).paxosservice(pgmap 21727587..21728259) refresh upgraded, 
format 0 -> 1
2019-07-22 19:17:54.491612 7f2064488f40  1 mon.node-1@-1(probing).pg v0 
on_upgrade discarding in-core PGMap
2019-07-22 19:17:54.491635 7f2064488f40 10 mon.node-1@-1(probing).pg v0 
update_from_paxos v0, read_full
2019-07-22 19:17:54.491638 7f2064488f40 10 mon.node-1@-1(probing).pg v0 
read_pgmap_meta
 ceph version 12.2.12-218-g9fd889f (9fd889fe09c652512ca78854702d5ad9bf3059bb) 
luminous (stable)
 1: (()+0x96b249) [0x7f2063e73249]
 2: (()+0x10330) [0x7f20628a0330]
 3: (gsignal()+0x37) [0x7f2060e8bc37]
 4: (abort()+0x148) [0x7f2060e8f028]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f206179a535]
 6: (()+0x5e6d6) [0x7f20617986d6]
 7: (()+0x5e703) [0x7f2061798703]
 8: (()+0x5e922) [0x7f2061798922]
 9: (object_stat_sum_t::decode(ceph::buffer::list::iterator&)+0x650) 
[0x7f2063c81be0]
 10: (object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x4f) 
[0x7f2063c9627f]
 11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x1d5) [0x7f2063c96965]
 12: (PGMap::update_pg(pg_t, ceph::buffer::list&)+0xf4) [0x7f20639d93b4]
 13: (PGMonitor::read_pgmap_full()+0x161) [0x7f20639a8a81]
 14: (PGMonitor::update_from_paxos(bool*)+0x699) [0x7f20639b0479]
 15: (PaxosService::refresh(bool*)+0x1a3) [0x7f2063a55103]
 16: (Monitor::refresh_from_paxos(bool*)+0x183) [0x7f206390cd53]
 17: (Monitor::init_paxos()+0xfd) [0x7f206390d12d]
 18: (Monitor::preinit()+0xa7e) [0x7f206390dbee]
 19: (main()+0x3bf4) [0x7f206383cde4]
 20: (__libc_start_main()+0xf5) [0x7f2060e76f45]
 21: