Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

[Mark Kirkwood]

On 13/07/14 18:38, Andrija Panic wrote:

Hi Mark,
actually, CEPH is running fine, and I have deployed NEW host (new
compile libvirt with ceph 0.8 devel, and newer kernel) - and it works...
so migrating some VMs to this new host...

I have 3 physical hosts, that are both MON and 2x OSD per host, all3
don't work-cloudstack/libvirt...

Any suggestion on need to recompile libvirt ? I got info from Wido, that
libvirt does NOT need to be recompiled



Looking at the differences between src/include/ceph_features.h in 0.72 
and 0.81 [1] (note, not *quite* the same version as you are using), 
there's erasure codes and other new features that are advertised by the 
later version that the client will need to match. Now *some* of these 
(crush tunables) can be switched off via:


$ ceph osd crush tunables legacy

...which would have been worth a try, but my guess is would not have 
worked, as (for example) I *don't* think erasure codes feature can be 
switched off. Hence, unless I'm mistaken (which is always possible) I 
think you did in fact need to recompile.


regards

Mark


[1] e.g:
--- ceph_features.h.72  2014-07-13 19:00:36.805825203 +1200
+++ ceph_features.h.81  2014-07-13 19:02:22.065826068 +1200
@@ -40,6 +40,18 @@
 #define CEPH_FEATURE_MON_SCRUB  (1ULL<<33)
 #define CEPH_FEATURE_OSD_PACKED_RECOVERY (1ULL<<34)
 #define CEPH_FEATURE_OSD_CACHEPOOL (1ULL<<35)
+#define CEPH_FEATURE_CRUSH_V2  (1ULL<<36)  /* new indep; SET_* steps */
+#define CEPH_FEATURE_EXPORT_PEER   (1ULL<<37)
+#define CEPH_FEATURE_OSD_ERASURE_CODES (1ULL<<38)
+#define CEPH_FEATURE_OSD_TMAP2OMAP (1ULL<<38)   /* overlap with EC */
+/* The process supports new-style OSDMap encoding. Monitors also use
+   this bit to determine if peers support NAK messages. */
+#define CEPH_FEATURE_OSDMAP_ENC(1ULL<<39)
+#define CEPH_FEATURE_MDS_INLINE_DATA (1ULL<<40)
+#define CEPH_FEATURE_CRUSH_TUNABLES3 (1ULL<<41)
+#define CEPH_FEATURE_OSD_PRIMARY_AFFINITY (1ULL<<41)  /* overlap w/ 
tunables3 */

+#define CEPH_FEATURE_MSGR_KEEPALIVE2   (1ULL<<42)
+#define CEPH_FEATURE_OSD_POOLRESEND(1ULL<<43)

 /*
  * The introduction of CEPH_FEATURE_OSD_SNAPMAPPER caused the feature
@@ -102,7 +114,16 @@
 CEPH_FEATURE_OSD_SNAPMAPPER |  \
 CEPH_FEATURE_MON_SCRUB |   \
 CEPH_FEATURE_OSD_PACKED_RECOVERY | \
-CEPH_FEATURE_OSD_CACHEPOOL | \
+CEPH_FEATURE_OSD_CACHEPOOL |   \
+CEPH_FEATURE_CRUSH_V2 |\
+CEPH_FEATURE_EXPORT_PEER | \
+ CEPH_FEATURE_OSD_ERASURE_CODES |   \
+CEPH_FEATURE_OSDMAP_ENC |  \
+CEPH_FEATURE_MDS_INLINE_DATA | \
+CEPH_FEATURE_CRUSH_TUNABLES3 | \
+CEPH_FEATURE_OSD_PRIMARY_AFFINITY |\
+CEPH_FEATURE_MSGR_KEEPALIVE2 | \
+CEPH_FEATURE_OSD_POOLRESEND |  \
 0ULL)

 #define CEPH_FEATURES_SUPPORTED_DEFAULT  CEPH_FEATURES_ALL
@@ -112,6 +133,8 @@
  */
 #define CEPH_FEATURES_CRUSH\
(CEPH_FEATURE_CRUSH_TUNABLES |  \
-CEPH_FEATURE_CRUSH_TUNABLES2)
+CEPH_FEATURE_CRUSH_TUNABLES2 | \
+CEPH_FEATURE_CRUSH_TUNABLES3 | \
+CEPH_FEATURE_CRUSH_V2)

 #endif

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

[Mark Kirkwood]

On 13/07/14 19:15, Mark Kirkwood wrote:

On 13/07/14 18:38, Andrija Panic wrote:



Any suggestion on need to recompile libvirt ? I got info from Wido, that
libvirt does NOT need to be recompiled



Thinking about this a bit more - Wido *may* have meant:

- *libvirt* does not need to be rebuild
- ...but you need to get/build a later ceph client i.e - 0.80

Of course depending on how your libvirt build was set up (e.g static 
linkage), this *might* have meant you needed to rebuild it too.


Regards

Mark

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [URGENT]. Can't connect to CEPH after upgrade from 0.72 to 0.80

[Andrija Panic]

Hi Mark,

update:

after restarting libvirtd and cloudstack-agent and management server God
know how many times - it WORKS now !

Not sure what is happening here, but it works again... I know for sure it
was not CEPH cluster, since it was fine, and accessible via qemu-img, etc...

Thanks Mark for your time for my issue...
Best.
Andrija




On 13 July 2014 10:20, Mark Kirkwood  wrote:

> On 13/07/14 19:15, Mark Kirkwood wrote:
>
>> On 13/07/14 18:38, Andrija Panic wrote:
>>
>
>  Any suggestion on need to recompile libvirt ? I got info from Wido, that
>>> libvirt does NOT need to be recompiled
>>>
>>>
> Thinking about this a bit more - Wido *may* have meant:
>
> - *libvirt* does not need to be rebuild
> - ...but you need to get/build a later ceph client i.e - 0.80
>
> Of course depending on how your libvirt build was set up (e.g static
> linkage), this *might* have meant you needed to rebuild it too.
>
> Regards
>
> Mark
>
>


-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Andrija Panic]

Hi,

after seting ceph upgrade (0.72.2 to 0.80.3) I have issued "ceph osd crush
tunables optimal" and after only few minutes I have added 2 more OSDs to
the CEPH cluster...

So these 2 changes were more or a less done at the same time - rebalancing
because of tunables optimal, and rebalancing because of adding new OSD...

Result - all VMs living on CEPH storage have gone mad, no disk access
efectively, blocked so to speak.

Since this rebalancing took 5h-6h, I had bunch of VMs down for that long...

Did I do wrong by causing "2 rebalancing" to happen at the same time ?
Is this behaviour normal, to cause great load on all VMs because they are
unable to access CEPH storage efectively ?

Thanks for any input...
-- 

Andrija Panić
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mixing CEPH versions on new ceph nodes...

[Andrija Panic]

Hi Wido,

you said previously:
  Upgrade the packages, but don't restart the daemons yet, then:
  1. Restart the mon leader
  2. Restart the two other mons
  3. Restart all the OSDs one by one

But in reality (yum update or by using ceph-deploy install nodename) -
the package manager does restart ALL ceph services on that node by its
own...
So, I have upgraded - MON leader and 2 OSD on this 1st upgraded host were
restarted, folowed by doing the same with other 2 servers (1 MON peon and 2
OSD per host).

Is this perhaps a package (RPM) bug - restarting daemons automatically ?
Since it makes sense to have all MONs updated first, and than OSD (and
perhaps after that MDS if using it...)

Upgraded to 0.80.3 release btw.

Thanks for your help again.
Andrija



On 3 July 2014 15:21, Andrija Panic  wrote:

> Thanks again a lot.
>
>
> On 3 July 2014 15:20, Wido den Hollander  wrote:
>
>> On 07/03/2014 03:07 PM, Andrija Panic wrote:
>>
>>> Wido,
>>> one final question:
>>> since I compiled libvirt1.2.3 usinfg ceph-devel 0.72 - do I need to
>>> recompile libvirt again now with ceph-devel 0.80 ?
>>>
>>> Perhaps not smart question, but need to make sure I don't screw
>>> something...
>>>
>>
>> No, no need to. The librados API didn't change in case you are using RBD
>> storage pool support.
>>
>> Otherwise it just talks to Qemu and that talks to librbd/librados.
>>
>> Wido
>>
>>  Thanks for your time,
>>> Andrija
>>>
>>>
>>> On 3 July 2014 14:27, Andrija Panic >> > wrote:
>>>
>>> Thanks a lot Wido, will do...
>>>
>>> Andrija
>>>
>>>
>>> On 3 July 2014 13:12, Wido den Hollander >> > wrote:
>>>
>>> On 07/03/2014 10:59 AM, Andrija Panic wrote:
>>>
>>> Hi Wido, thanks for answers - I have mons and OSD on each
>>> host...
>>> server1: mon + 2 OSDs, same for server2 and server3.
>>>
>>> Any Proposed upgrade path, or just start with 1 server and
>>> move along to
>>> others ?
>>>
>>>
>>> Upgrade the packages, but don't restart the daemons yet, then:
>>>
>>> 1. Restart the mon leader
>>> 2. Restart the two other mons
>>> 3. Restart all the OSDs one by one
>>>
>>> I suggest that you wait for the cluster to become fully healthy
>>> again before restarting the next OSD.
>>>
>>> Wido
>>>
>>> Thanks again.
>>> Andrija
>>>
>>>
>>> On 2 July 2014 16:34, Wido den Hollander >> 
>>> >> wrote:
>>>
>>>  On 07/02/2014 04:08 PM, Andrija Panic wrote:
>>>
>>>  Hi,
>>>
>>>  I have existing CEPH cluster of 3 nodes, versions
>>> 0.72.2
>>>
>>>  I'm in a process of installing CEPH on 4th node,
>>> but now CEPH
>>>  version is
>>>  0.80.1
>>>
>>>  Will this make problems running mixed CEPH versions
>>> ?
>>>
>>>
>>>  No, but the recommendation is not to have this running
>>> for a very
>>>  long period. Try to upgrade all nodes to the same
>>> version within a
>>>  reasonable amount of time.
>>>
>>>
>>>  I intend to upgrade CEPH on exsiting 3 nodes anyway
>>> ?
>>>  Recommended steps ?
>>>
>>>
>>>  Always upgrade the monitors first! Then to the OSDs one
>>> by one.
>>>
>>>  Thanks
>>>
>>>  --
>>>
>>>  Andrija Panić
>>>
>>>
>>>  ___
>>>
>>>  ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> >> >
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph._
>>> ___com
>>> 
>>>
>>>
>>>
>>> >> >
>>>
>>>
>>>
>>>  --
>>>  Wido den Hollander
>>>  42on B.V.
>>>  Ceph trainer and consultant
>>>
>>>  Phone: +31 (0)20 700 9902
>>> 
>>> 
>>>  Skype: contact42on
>>>  ___
>>>
>>>  ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> >> >
>>> h