[ https://issues.apache.org/jira/browse/CASSANDRA-11559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414721#comment-16414721 ]
Alex Lourie commented on CASSANDRA-11559: ----------------------------------------- I've done some work on this ticket and it's available at [https://github.com/apache/cassandra/compare/trunk...alourie:CASSANDRA-11559#files_bucket]. Few pointers: * I've refactored InetAddressAndPort class into VirtualEndpoint class across all the codebase (this will be responsible for the majority of the code changes). * I've added a UUID field to hold the hostID value of the endpoint and added additional methods for working with it. * I've reworked the TokenMetadata to hold structures other than maps for UUID-host references and they would no longer be needed, i.e. keeping just a set of endpoints is enough to hold both address data and the hostID data and to also look up hosts by IDs or the vice versa. * I've reworked the SystemKeyspace to also acknowledge the hostIDs where significant (in local data/peer data storing/fetching), and also only create new local id if requested (in most cases only when the node is created for the first time, but also useful for tests that require initiating multiple "nodes" on the same machine) I've added a field in DatabaseDescriptor to mark that SystemKeyspace is ready to be read. This is required for many unit tests that set up clusters "on the fly" and for further endpoint information discovery during the test run. * I've updated required unit tests to properly utilise the new object and initialise and clean others as required. * I've updated the code in some other locations to incorporate this change, which does make it simpler on many occasions. The current state is everything _seems_ to be working and the unit tests pass ([https://circleci.com/gh/alourie/cassandra/97]) The complication that comes out of this work is with building unit tests - the host ID would now be kept in multiple structures: * a VirtualEndpoint object when instantiated. * SystemKeyspace.localHost (queries the DB) * SystemKeyspace.peersInfo (queries the DB) * TokenMetadata lists (such as allEndpoints, tokenMap, etc) * Gossip.instance.endpointState maps (the specific endpoint is added including the uuid) * FBUtilities also keeps local reference once fetched As a result, when creating tests, one needs to update or clear the hostID-related information in all relevant places, otherwise, tests would fail with really confusing messages (in most cases because in some thread an endpoint comparison will happen and UUIDs won't match), such as "no seeds found", "host cannot be contacted" or various kinds of timeouts and NPEs. Additionally, when SystemKeyspace is ready to be read within a test flow, a DatabaseDescriptor.canReadSystemKeyspace field will need to be set to true so that the UUID would be fetched from SystemKeyspace. Additionally, at the moment we are keeping EndpointState separately from this object (in Gossip). Considering that now this VirtualEndpoint can include basically any information about the endpoint, it may as well incorporate its own state, and then all handling of the network/state information about an endpoint will be in one place. Supposedly this should simplify things further and allow clearing a lot of code. [~aweisberg] - you have done the previous move away from InetAddress representation to InetAddressAndPort, which this current patch changes considerably. I'd love your feedback on this. Any and all feedback is very welcome. > Enhance node representation > --------------------------- > > Key: CASSANDRA-11559 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11559 > Project: Cassandra > Issue Type: Sub-task > Components: Distributed Metadata > Reporter: Paulo Motta > Assignee: Alex Lourie > Priority: Minor > > We currently represent nodes as {{InetAddress}} objects on {{TokenMetadata}}, > what causes difficulties when replacing a node with the same address (see > CASSANDRA-8523 and CASSANDRA-9244). > Since CASSANDRA-4120 we index hosts by {{UUID}} in gossip, so I think it's > time to move that representation to {{TokenMetadata}}. > I propose representing nodes as {{InetAddress, UUID}} pairs on > {{TokenMetadata}}, encapsulated in a {{VirtualNode}} interface, so it will > backward compatible with the current representation, while still allowing us > to enhance it in the future with additional metadata (and improved vnode > handling) if needed. > This change will probably affect interfaces of internal classes like > {{TokenMetadata}} and {{AbstractReplicationStrategy}}, so I'd like to hear > from integrators and other developers if it's possible to change these > without major hassle or if we need to wait until 4.0. > Besides updating {{TokenMetadata}} and {{AbstractReplicationStrategy}} (and > subclasses), we will also need to replace all {{InetAddress}} uses with > {{VirtualNode.getEndpoint()}} calls on {{StorageService}} and related classes > and tests. We would probably already be able to replace some > {{TokenMetadata.getHostId(InetAddress endpoint)}} calls with > {{VirtualNode.getHostId()}}. > While we will still be dealing with {{InetAddress}} on {{StorageService}} in > this initial stage, in the future I think we should pass {{VirtualNode}} > instances around and only translate from {{VirtualNode}} to {{InetAddress}} > in the network layer. > Public interfaces like {{IEndpointSnitch}} will not be affected by this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org