Re: [Gluster-users] Filled up file system and glusterd not working anymore. To probe or not to probe?

2024-01-09 Thread Ilias Chasapakis forumZFD

Dear all,

first of all thanks to Stefan (Kania) for the feedback. So detaching the 
peer, removing brick and downgrading to replica 2 until you add a new 
brick as you would do with any otherwise faulty node.


So here is how we proceeded:

 * check mounts where the faulty (ex-filled-up) node is present and
   remove it from there
 * detach "bad" peer with:  gluster peer detach 
 * remove brick: gluster volume remove-brick  replica 2
   : force

So in our case we are ready to add a new brick to restore our replica 3.

This should be safer (and in the end as simple) than just copying the 
missing files probably bringing along meta-data inconsistencies. Which 
was also the doubt expressed on this post.


Am 08.01.24 um 11:13 schrieb Ilias Chasapakis forumZFD:

Dear all,

we have a replica 3 configuration an issue after that the file system 
filled up. The gluster daemon is not starting anymore on the affected 
node and the inconsistency that we noticed is that the 
/var/lib/glusterd/peers does contain only one "good" node and the 
other one is missing.


Now what we would like to try is to probe from the affected peer the 
missing one. We also thought about just copying the missing node 
config file /var/lib/glusterd/peers here but we think this might be 
more inconsistent because perhaps of the time passed between finding 
out the issue and when the fill up had occurred (if this has 
relevance). We assume probing will fill that missing entry in the 
affected gluster and will sync to it. What we want to avoid is that 
this happens the other way around.


So in brief the question is if in our situation probing is too risky 
and we should try other methods (removing-adding brick) or anything 
that you might want to suggest from your experience.


Many thanks in advance

Ilias






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge:https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


--
forumZFD
Entschieden für Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany

Tel 0221 91273243 | Fax 0221 91273299 |http://www.forumZFD.de

Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS



OpenPGP_signature.asc
Description: OpenPGP digital signature




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


[Gluster-users] Filled up file system and glusterd not working anymore. To probe or not to probe?

2024-01-08 Thread Ilias Chasapakis forumZFD

Dear all,

we have a replica 3 configuration an issue after that the file system 
filled up. The gluster daemon is not starting anymore on the affected 
node and the inconsistency that we noticed is that the 
/var/lib/glusterd/peers does contain only one "good" node and the other 
one is missing.


Now what we would like to try is to probe from the affected peer the 
missing one. We also thought about just copying the missing node config 
file /var/lib/glusterd/peers here but we think this might be more 
inconsistent because perhaps of the time passed between finding out the 
issue and when the fill up had occurred (if this has relevance). We 
assume probing will fill that missing entry in the affected gluster and 
will sync to it. What we want to avoid is that this happens the other 
way around.


So in brief the question is if in our situation probing is too risky and 
we should try other methods (removing-adding brick) or anything that you 
might want to suggest from your experience.


Many thanks in advance

Ilias

--
forumZFD
Entschieden für Frieden | Committed to Peace

Ilias Chasapakis
Referent IT | IT Consultant

Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany

Tel 0221 91273243 | Fax 0221 91273299 | http://www.forumZFD.de

Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln

Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00   BIC GENODEM1GLS



OpenPGP_signature.asc
Description: OpenPGP digital signature




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users