[
https://issues.apache.org/jira/browse/TS-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167597#comment-13167597
]
B Wyatt edited comment on TS-949 at 12/12/11 5:05 PM:
------------------------------------------------------
Thanks John. I think the new patch should be more stable. I apologize for the
misread of the previous patch, all of my volumes are matched in size so I had
erroneously tuned out the inclusion of vol->len in the initial value of
forvol[i].
All of this digging has brought up a new related issue (that I am pretty sure
we cannot address at this level): Object loss when adding volumes. The hash is
now consistent, however when a new volume supersedes an existing volume in the
hash, any object that maps to that bucket but currently stored on the old
volume will become inaccessible. I will probably create a new issue for that
as this one is solved in my book.
[Edit: removed statement about comments... they are there... my coffee is not,
apparently]
was (Author: wanderingbort):
Thanks John. I think the new patch should be more stable. I apologize for
the misread of the previous patch, all of my volumes are matched in size so I
had erroneously tuned out the inclusion of vol->len in the initial value of
forvol[i].
While I am not an enforcer of code quality, I think the particulars of this
method should at the very least be documented in the patched code. I'll let
someone else decide whether it is worth the effort to "pretty" it up.
All of this digging has brought up a new related issue (that I am pretty sure
we cannot address at this level): Object loss when adding volumes. The hash is
now consistent, however when a new volume supersedes an existing volume in the
hash, any object that maps to that bucket but currently stored on the old
volume will become inaccessible. I will probably create a new issue for that
as this one is solved in my book.
> key->volume hash table is not consistent when a disk is marked as bad or
> removed due to failure
> -----------------------------------------------------------------------------------------------
>
> Key: TS-949
> URL: https://issues.apache.org/jira/browse/TS-949
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 3.1.0
> Environment: Multi-volume cache with apparently faulty drives
> Reporter: B Wyatt
> Assignee: John Plevyak
> Fix For: 3.1.2
>
> Attachments: TS-949-jp-1.patch, TS-949-jp2.patch, TS949-BW-p1.patch
>
>
> The method for resolving collisions when distributing hash-table space to
> volumes for the object_key->volume hash table creates inconsistency when a
> disk is determined to be bad, or when a failed disk is removed from the
> volume.config.
> Background:
> The hash space is distributed by round robin draft where each volume "drafts"
> a random index in the hash table until the hash space is exhausted. The
> random order in which a given volume drafts hash table slots is consistent
> across reboot/crash/disk-failure, however when a volume attempts to draft a
> slot which has already been occupied, it skips to its next random pick and
> attempts to draft that slot until it finds an open slot. This ensures that
> the hash is partitioned evenly between volumes.
> The issue:
> Resolving slot contention breaks the consistency as it is dependent on the
> order that the volumes draft. When rebuilding the hash after disk failure or
> reboot with fewer drives, a volume may secure an index that was previously
> occupied by the dead-disk. In the old hash, the surviving volume would have
> selected another random index due to contention. If this index is taken, by
> the next draft round it will represent an inconsistent key->volume result.
> The effects of one inconsistency will then cascade as whichever volume
> occupies that index after removing a dead disk is now behind on its draft
> sequence as well.
> An Example:
> ||Disk||Draft Sequence||
> |A|1,4,7,5|
> |B|4,2,8,1|
> |C|3,7,5,2|
> Pre-failure Hash Table after 2 rounds of draft:
> |A|B|C|B|C|?|A|?|
> Post-failure of drive B Hash Table after 3 rounds of draft:
> |A|C|C|A|{color:red}A{color}|?|{color:red}C{color}|?|
> Two slots have become inconsistent and more will probably follow. These
> inconsistencies become objects stored in a volume but lost to the top level
> cache for open/lookup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
