Adar Dembo has posted comments on this change. Change subject: WIP: catalog manager audit/analysis ......................................................................
Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/2583/1/docs/design-docs/multi-master-1.0-supplement.md File docs/design-docs/multi-master-1.0-supplement.md: Line 3: UNKNOWN > What is this state transition? do we store, or do something about UNKNOWN UNKNOWN is just the default state in the PB. So when you create the PB, you get UNKNOWN to begin with. Line 4: 2. create tablets in state UNKNOWN, begin mutation to state PREPARING > same question Same answer. :) Line 5: 4. update in-memory maps (table by id/name, tablet by id) > Hum this answers my question but, why is this useful? We use these maps for table lookups (client requests by ID or by table name) as well as the odd tablet lookup (i.e. client issues GetTabletLocations). Or did I misunderstand your question? Line 18: - The two replications can be safely combined > So can you combine them to see what they would look like? Or are you trying Everything in this document describes the status quo except for the "Thoughts" section, which, if they suggest changes, are future-thinking. Unfortunately we can't move all of the in-memory changes to come after the replicate. The main issue is step 4 (updating the maps); we use that step to both check that a table with name foo doesn't exist and to "claim" that name for this table. We have to do that before we replicate the write, because the write will NOT fail if we try to create a duplicate table (the table name is not part of the master tablet's primary key; only the table ID is). See http://gerrit.cloudera.org:8080/2577 for more details on this. However, before step 4, we could make state changes to the table and tablets without the lock, so they won't be subject to COW and the new states will be visible immediately in step 4. That should prevent the behavior in the aforementioned gerrit, though I think we should still keep that change for defense in depth. What do you think? Worth reordering, or should we just rely on checking at the call-sites? Line 27: a. create tablet t_new in state UNKNOWN, begin mutation to state PREPARING > in general what is this "begin mutation"? Since I wrote this in shorthand, I didn't fill in details like this. The table and tablet locking system is based on an underlying tri-state (read, write, commit) lock. The idea is that multiple readers can be concurrent with a single writer, and that to commit a write, the committer must wait for outstanding readers and writers to leave the lock. The implementation is COW-based. A read-only consumer reads directly from the underlying PB being protected (SysTablesEntryPB or SysTabletsEntryPB) in memory. A read-write consumer reads from and writes to a "mutable" copy of the in-memory state; that's how we ensure that RO consumers can operate concurrently with an RW consumer. Lastly, there's a multi-step "commit" operation for RW consumers: 1. Upgrade the lock from write to commit, waiting for all read consumers to exit the lock. 2. Reset the immutable copy of the data using the mutable copy. 3. Relinquish the lock. In this context, "begin mutation" means "acquire the lock for writing so that subsequent mutations are written to the write-only copy." -- To view, visit http://gerrit.cloudera.org:8080/2583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia3dd75552400b89718a0f789e96895a64d1fb91b Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: David Ribeiro Alves <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-HasComments: Yes
