Re: [ClusterLabs] Upgrade corosync problem

[Salvatore D'angelo]

Hi All,

Today I tested the two suggestions you gave me. Here what I did. 
In the script where I create my 5 machines cluster (I use three nodes for 
pacemaker PostgreSQL cluster and two nodes for glusterfs that we use for 
database backup and WAL files).

FIRST TEST
——
I added the —shm-size=512m to the “docker create” command. I noticed that as 
soon as I start it the shm size is 512m and I didn’t need to add the entry in 
/etc/fstab. However, I did it anyway:

tmpfs  /dev/shm  tmpfs   defaults,size=512m   0   0

and then
mount -o remount /dev/shm

Then I uninstalled all pieces of software (crmsh, resource agents, corosync and 
pacemaker) and installed the new one.
Started corosync and pacemaker but same problem occurred.

SECOND TEST
———
stopped corosync and pacemaker
uninstalled corosync
build corosync with --enable-small-memory-footprint and installed it
starte corosync and pacemaker

IT WORKED.

I would like to understand now why it didn’t worked in first test and why it 
worked in second. Which kind of memory is used too much here? /dev/shm seems 
not the problem, I allocated 512m on all three docker images (obviously on my 
single Mac) and enabled the container option as you suggested. Am I missing 
something here?

Now I want to use Docker for the moment only for test purpose so it could be ok 
to use the --enable-small-memory-footprint, but there is something I can do to 
have corosync working even without this option?


The reason I am asking this is that, in the future, it could be possible we 
deploy in production our cluster in containerised way (for the moment is just 
an idea). This will save a lot of time in developing, maintaining and deploying 
our patch system. All prerequisites and dependencies will be enclosed in 
container and if IT team will do some maintenance on bare metal (i.e. install 
new dependencies) it will not affects our containers. I do not see a lot of 
performance drawbacks in using container. The point is to understand if a 
containerised approach could save us lot of headache about maintenance of this 
cluster without affect performance too much. I am notice in Cloud environment 
this approach in a lot of contexts.


> On 2 Jul 2018, at 08:54, Christine Caulfield  wrote:
> 
> On 29/06/18 17:20, Jan Pokorný wrote:
>> On 29/06/18 10:00 +0100, Christine Caulfield wrote:
>>> On 27/06/18 08:35, Salvatore D'angelo wrote:
 One thing that I do not understand is that I tried to compare corosync
 2.3.5 (the old version that worked fine) and 2.4.4 to understand
 differences but I haven’t found anything related to the piece of code
 that affects the issue. The quorum tool.c and cfg.c are almost the same.
 Probably the issue is somewhere else.
 
>>> 
>>> This might be asking a bit much, but would it be possible to try this
>>> using Virtual Machines rather than Docker images? That would at least
>>> eliminate a lot of complex variables.
>> 
>> Salvatore, you can ignore the part below, try following the "--shm"
>> advice in other part of this thread.  Also the previous suggestion
>> to compile corosync with --small-memory-footprint may be of help,
>> but comes with other costs (expect lower throughput).
>> 
>> 
>> Chrissie, I have a plausible explanation and if it's true, then the
>> same will be reproduced wherever /dev/shm is small enough.
>> 
>> If I am right, then the offending commit is
>> https://github.com/corosync/corosync/commit/238e2e62d8b960e7c10bfa0a8281d78ec99f3a26
>> (present since 2.4.3), and while it arranges things for the better
>> in the context of prioritized, low jitter process, it all of
>> a sudden prevents as-you-need memory acquisition from the system,
>> meaning that the memory consumption constraints are checked immediately
>> when the memory is claimed (as it must fit into dedicated physical
>> memory in full).  Hence this impact we likely never realized may
>> be perceived as a sort of a regression.
>> 
>> Since we can calculate the approximate requirements statically, might
>> be worthy to add something like README.requirements, detailing how much
>> space will be occupied for typical configurations at minimum, e.g.:
>> 
>> - standard + --small-memory-footprint configuration
>> - 2 + 3 + X nodes (5?)
>> - without any service on top + teamed with qnetd + teamed with
>>  pacemaker atop (including just IPC channels between pacemaker
>>  daemons and corosync's CPG service, indeed)
>> 
> 
> That is possible explanation I suppose, yes.it 's not 
> something we can
> sensibly revert because it was already fixing another regression!
> 
> 
> I like the idea of documenting the /dev/shm requrements - that would
> certainly help with other people using containers - Salvatore mentioned
> earlier that there was nothing to guide him about the size needed. I'll
> raise an issue in github to cover it. Your input on how to do it for
> containers would also be helpful.
> 
> Chrissie
> 

Re: [ClusterLabs] Upgrade corosync problem

[Christine Caulfield]

On 29/06/18 17:20, Jan Pokorný wrote:
> On 29/06/18 10:00 +0100, Christine Caulfield wrote:
>> On 27/06/18 08:35, Salvatore D'angelo wrote:
>>> One thing that I do not understand is that I tried to compare corosync
>>> 2.3.5 (the old version that worked fine) and 2.4.4 to understand
>>> differences but I haven’t found anything related to the piece of code
>>> that affects the issue. The quorum tool.c and cfg.c are almost the same.
>>> Probably the issue is somewhere else.
>>>
>>
>> This might be asking a bit much, but would it be possible to try this
>> using Virtual Machines rather than Docker images? That would at least
>> eliminate a lot of complex variables.
> 
> Salvatore, you can ignore the part below, try following the "--shm"
> advice in other part of this thread.  Also the previous suggestion
> to compile corosync with --small-memory-footprint may be of help,
> but comes with other costs (expect lower throughput).
> 
> 
> Chrissie, I have a plausible explanation and if it's true, then the
> same will be reproduced wherever /dev/shm is small enough.
> 
> If I am right, then the offending commit is
> https://github.com/corosync/corosync/commit/238e2e62d8b960e7c10bfa0a8281d78ec99f3a26
> (present since 2.4.3), and while it arranges things for the better
> in the context of prioritized, low jitter process, it all of
> a sudden prevents as-you-need memory acquisition from the system,
> meaning that the memory consumption constraints are checked immediately
> when the memory is claimed (as it must fit into dedicated physical
> memory in full).  Hence this impact we likely never realized may
> be perceived as a sort of a regression.
> 
> Since we can calculate the approximate requirements statically, might
> be worthy to add something like README.requirements, detailing how much
> space will be occupied for typical configurations at minimum, e.g.:
> 
> - standard + --small-memory-footprint configuration
> - 2 + 3 + X nodes (5?)
> - without any service on top + teamed with qnetd + teamed with
>   pacemaker atop (including just IPC channels between pacemaker
>   daemons and corosync's CPG service, indeed)
> 

That is possible explanation I suppose, yes.it's not something we can
sensibly revert because it was already fixing another regression!


I like the idea of documenting the /dev/shm requrements - that would
certainly help with other people using containers - Salvatore mentioned
earlier that there was nothing to guide him about the size needed. I'll
raise an issue in github to cover it. Your input on how to do it for
containers would also be helpful.

Chrissie
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync 2.4 CPG config change callback

[Jan Friesse]

Hi Thomas,

Hi,

Am 04/25/2018 um 09:57 AM schrieb Jan Friesse:

Thomas Lamprecht napsal(a):

On 4/24/18 6:38 PM, Jan Friesse wrote:

On 4/6/18 10:59 AM, Jan Friesse wrote:

Thomas Lamprecht napsal(a):

Am 03/09/2018 um 05:26 PM schrieb Jan Friesse:
I've tested it too and yes, you are 100% right. Bug is there and 
it's
pretty easy to reproduce when node with lowest nodeid is paused. 
It's

slightly harder when node with higher nodeid is paused.



Do you were able to make some progress on this issue?


Ya, kind of. Sadly I had to work on different problem, but I'm 
expecting to sent patch next week.




I guess the different problems where the ones related to the issued 
CVEs :)


Yep.

Also I've spent quite a lot of the time thinking about best possible 
solution. CPG is quite old, it was full of weird bugs and risk of 
breakage is very high.


Anyway, I've decided to not to try hack what is apparently broken 
and just go for risky but proper solution (= needs a LOT more 
testing, but so far looks good).




I did not looked deep into how your revert plays out with the
mentioned commits of the heuristics approach, but this fix would
mean to bring corosync back to a state it had already, and thus
was already battle tested?


Yep, but not fully. Important change was to use joinlists as 
authoritative source of information about other node clients, so I 
believe that solved problems which should had been "solved" by 
downlist heuristics.





Patch and approach seems good to me, with my limited knowledge,
when looking at the various "bandaid" fix commits you mentioned.


Patch is in PR (needle): https://github.com/corosync/corosync/pull/347



Much thanks! First tests work well here.
I could not yet reproduce the problem with the patch applied in both,
testcpg and our cluster configuration file system.


That's good to hear :)



I'll let it run


Perfect.




Just wanted to give some quick feedback.
We deployed this to your community repository about a week ago (after
another week of successful testing), we had no negative feedback or
issues reported or seen yet, with (strong lower bound) > 10k systems
running the fix by now.


Thanks, that's exciting news.



I saw just now that you merged it into needle and master, so, while a 
bit late, this just backs the confidence into the fix up.


Definitively not late until it's released :)



Much thanks for your, and the reviewers, work!


Yep, you are welcomed.

Honza



cheers,
Thomas



___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: [questionnaire] Do you overload pacemaker's meta-attributes to track your own data?

[Ulrich Windl]

>>> Jan Pokorný  schrieb am 28.06.2018 um 20:23 in
Nachricht
<20180628182329.gb4...@redhat.com>:
[...]
> * * *
> 
> So now the promised questions; just send a reply where you [x] tick
> your selections for the questions below, possibly with some more
> commentary on the topic, and preferrably on‑list (single of your
> choice is enough):
> 
> 1. In your cluster configurations, do you carry meta‑attributes
>other than those recognized by pacemaker?
> 
>[ ] no

--> NO

> 
>[ ] yes (if so, can you specify whether for said constraints
> rules, as a way to permanently attach some kind of
> administrative piece of information, whether you
> have the whole custom tooling around this, etc.?)
> 
> 2. How do you feel about said meta‑attributes' namespace separation
>proposal (as summarized in documentation edit per above link)?
> 
>[ ] no feelings/not related to my use cases (e.g., haven't used
>custom meta‑attributes possibility before, no inclination to
>use that in the future)

--> X (the above)

> 
>[ ] too cumbersome, better to live with the risk of the future
>clashes now (and with the risk of distant future automatic
>upgrade doing accidentally a wrong thing)
> 
>[ ] acceptable, but only as an interim solution
> 
>[ ] acceptable without complaints
> 
> * * *
> 
> I am also keen to hear if this rather non‑intrusive change alone could
> be a show‑stopper for some.
> 
> Thanks for your feedback!
> 
> 
> [1] e.g. for primitive resources:
> 
> https://clusterlabs.org/pacemaker/doc/en‑US/Pacemaker/1.1/html‑single/Pacem

> aker_Explained/index.html#_resource_meta_attributes
> [2] 
> https://clusterlabs.org/pacemaker/doc/en‑US/Pacemaker/1.1/html‑single/Pacem

> aker_Explained/index.html#_node_attribute_expressions
> 
> ‑‑ 
> Jan (Poki)



___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org