On 08/12/2014 11:29 AM, Harshavardhana wrote:
This is a standard problem where there are split-brains in distributed
systems. For example even in git there are cases where it gives up asking
users to fix the file i.e. merge conflicts. If the user doesn't want
split-brains they should move to replica-3 and enable client-quorum. But if
the user made a conscious decision to live with split-brain problems
favouring availability/using replica-2, then split-brains do happen and it
needs user intervention. All we are trying to do is to make this process a
bit painless by coming up with meaningful policies.
Agreed, split brains do require manual intervention no one argues about that,
but it shouldn't be quite as tedious as GlusterFS wants it to be.
We are on same page.
I do agree that it is way simpler than perhaps some other distributed
filesystems but at
any point we ask some one to write a script to fix our internal
structure - that is not a
feature its a bug.
We are not asking them to write a script with this solution.
We all appreciate the effort, but my wish we incorporate some pain
points which we have
seen personally over the years and fix it right when we are at it.
If the user knows his workload is append only and there are split-brains the
only command he needs to execute is:
'gluster volume heal <volname> split-brain bigger-file'
no grep, no finding file paths, nothing.
Adding to this - we need to provide additional sanity checks that
split brains were
indeed fixed - since this looks quite destructive operation, are you
planning a rollback
at any point during this process?
User can find if split-brain is resolved by executing 'gluster volume
heal <volname> info split-brain'. The file/gfid shouldn't show up there.
There is no rollback until we integrate it with trash. When we integrate
it with trash the file that will be over-written/deleted will be moved
to trash.
There were also instances where the user knows the brick he/she would like
to be the source but he/she is worried that old brick which comes back up
would cause split-brains so he/she had to erase the whole brick which was
down and bring it back up.
Instead we can suggest him/her to use 'gluster volume heal <VOLNAME>
split-brain source-brick <brick_name>' after bringing the brick back up so
that not all the contents needs to be healed.
1) gluster volume heal <volname> info split-brain should give output in some
'format' giving stat/pending-matrix etc for all the files in split-brain.
- Unfortunately we still don't have a way to provide with file paths
without doing 'find' on the bricks.
Critical setups require fixing split-brain with quick turn around no
one really has the
luxury running a find on a large volume. So i still do not understand,
if a 'find' can do
a gfid --> inum --> path - how hard it is for Gluster management
daemon to know this?
just to provide better tooling?
Not sure. We can figure this out by traversing up the softlinks for
directories. But for files there is no way to find the parent at the moment.
Pranith
-- Harsha
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
