Hello Eunsu, I am also using PasswordAuthenticator in my cassandra cluster. I didn't come across this issue while doing the exercise on preprod.
Are you sure that you changed the configuration of system_auth keyspace before adding the new datacenter using this: ALTER KEYSPACE system_auth WITH REPLICATION = {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}; Regards, Pradeep On Tue, Sep 18, 2018 at 7:23 AM, Eunsu Kim <eunsu.bil...@gmail.com> wrote: > > In my case, there were authentication issues when adding data centers. > > I was using a PasswordAuthenticator. > > As soon as the datacenter was added, the following authentication error > log was recorded on the client log file. > > com.datastax.driver.core.exceptions.AuthenticationException: > Authentication error on host /xxx.xxx.xxx.xx:9042: Provided username apm > and/or password are incorrect > > I was using DCAwareRoundRobinPolicy, but I guess it's probably because of > the withUsedHostsPerRemoteDc option. > > I took several steps and the error log disappeared. It is probably > ’nodetool rebuild' after altering the system_auth table. > > However, the procedure was not clearly defined. > > > On 18 Sep 2018, at 2:40 AM, Pradeep Chhetri <prad...@stashaway.com> wrote: > > Hello Alain, > > Thank you very much for reviewing it. You answer on seed nodes cleared my > doubts. I will update it as per your suggestion. > > I have few followup questions on decommissioning of datacenter: > > - Do i need to run nodetool repair -full on each of the nodes (old + new > dc nodes) before starting the decommissioning process of old dc. > - We have around 15 apps using cassandra cluster. I want to make sure that > all queries before starting the new datacenter are going with right > consistency level i.e LOCAL_QUORUM instead of QUORUM. Is there a way i can > log the consistency level of each query somehow in some log file. > > Regards, > Pradeep > > On Mon, Sep 17, 2018 at 9:26 PM, Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> Hello Pradeep, >> >> It looks good to me and it's a cool runbook for you to follow and for >> others to reuse. >> >> To make sure that cassandra nodes in one datacenter can see the nodes of >>> the other datacenter, add the seed node of the new datacenter in any of the >>> old datacenter’s nodes and restart that node. >> >> >> Nodes seeing each other from the distinct rack is not related to seeds. >> It's indeed recommended to use seeds from all the datacenter (a couple or >> 3). I guess it's to increase availability on seeds node and/or maybe to >> make sure local seeds are available. >> >> You can perfectly (and even have to) add your second datacenter nodes >> using seeds from the first data center. A bootstrapping node should never >> be in the list of seeds unless it's the first node of the cluster. Add >> nodes, then make them seeds. >> >> >> Le lun. 17 sept. 2018 à 11:25, Pradeep Chhetri <prad...@stashaway.com> a >> écrit : >> >>> Hello everyone, >>> >>> Can someone please help me in validating the steps i am following to >>> migrate cassandra snitch. >>> >>> Regards, >>> Pradeep >>> >>> On Wed, Sep 12, 2018 at 1:38 PM, Pradeep Chhetri <prad...@stashaway.com> >>> wrote: >>> >>>> Hello >>>> >>>> I am running cassandra 3.11.3 5-node cluster on AWS with SimpleSnitch. >>>> I was testing the process to migrate to GPFS using AWS region as the >>>> datacenter name and AWS zone as the rack name in my preprod environment and >>>> was able to achieve it. >>>> >>>> But before decommissioning the older datacenter, I want to verify that >>>> the data in newer dc is in consistence with the one in older dc. Is there >>>> any easy way to do that. >>>> >>>> Do you suggest running a full repair before decommissioning the nodes >>>> of older datacenter ? >>>> >>>> I am using the steps documented here: https://medium.com/p/465e9bf28d99 >>>> I will be very happy if someone can confirm me that i am doing the right >>>> steps. >>>> >>>> Regards, >>>> Pradeep >>>> >>>> >>>> >>>> >>>> >>> > >