Re: RE : Solr Date Range filtering using datapointfield (solr 8.8.2)

2022-03-21 Thread Reej Nayagam
7:28 PM Thomas Corthals wrote: > Hi Reej, > > Have you tried with 2022-03-21T15:59:999Z (with a capital Z)? > > Regards, > > Thomas > > Op ma 21 mrt. 2022 om 12:20 schreef Reej Nayagam : > > > Hi All, > > > > In solr 4 , we had a date field and i

RE : Solr Date Range filtering using datapointfield (solr 8.8.2)

2022-03-21 Thread Reej Nayagam
Hi All, In solr 4 , we had a date field and in schema, fieldType name = date and class ="solr.*TrieDateField*" when moving to solr8, we changed the class to solr.*DatePointField*. Now while filtering using solr4, the effective_date_dt : [* TO 2022-03-21T15:59:999z] works but in solr8 the same th

Re: Help required in Solr Stop Port - resolved

2022-02-20 Thread Reej Nayagam
, and now different stop ports are assigned to the 2 different nodes. *Thanks,* *Reej* On Fri, Feb 18, 2022 at 9:26 AM Reej Nayagam wrote: > Hi All, > We have Solr cloud setup, version : 8.8.2 > > I was able to set the stop port and everything was working fine. > > Initial se

Help required in Solr Stop Port

2022-02-17 Thread Reej Nayagam
and did a fresh setup of the nodes *Thanks,* *Reej* On Mon, Jan 10, 2022 at 3:36 PM Reej Nayagam wrote: > Thanks Shawn, this worked. > Thank you for the explanation & the help. > > > *Thanks & Regards* > *Reej* > > > On Mon, Jan 10, 2022 at 7:45 AM Shawn Heis

Re: Solr Stop Port

2022-02-17 Thread Reej Nayagam
,* *Reej* On Mon, Jan 10, 2022 at 3:36 PM Reej Nayagam wrote: > Thanks Shawn, this worked. > Thank you for the explanation & the help. > > > *Thanks & Regards* > *Reej* > > > On Mon, Jan 10, 2022 at 7:45 AM Shawn Heisey wrote: > >> On 1/7/2022 4:55 A

Fwd: 90% of physical memory being used up

2022-02-11 Thread Reej Nayagam
Hi All, Need your help in this. The physical memory being used up in servers where solr nodes and zk are installed . At times the free space goes to 0 MB and slowly frees up some 100 MB Using the windows resource monitor when we monitor, if we bring down zk alone more than 40% of memory is relea

Re: Memory and thread leak using SolrJ

2022-02-09 Thread Reej Nayagam
Hi, But I remember there is a mention that it's not recommended to close the solrclient. We too have the same problem. The physical memory goes up to 20 GB and the application server which makes calls to solr also slows down with hogging threads. This issue is not until we upgraded from solr 4.1

Re: Zookeeper client Timeout - takes the default 30000 ms

2022-01-28 Thread Reej Nayagam
s, is this normal? what is the zk_node_count ZK_node_count 1852 zk_approximate_data_size 7853679 *Thanks,* *Reej* On Thu, Jan 27, 2022 at 4:22 PM Reej Nayagam wrote: > Hi Vinay, > > We are connecting using cloudsolrclient passing the zk host, so if zk is > down, the connection to solr

Re: Zookeeper client Timeout - takes the default 30000 ms

2022-01-27 Thread Reej Nayagam
he.org/doc/r3.4.8/zookeeperAdmin.html#sc_zkCommands > > > > This is how it works from a command line. > > > > $ echo ruok | nc zoo-shared-1.test.search.cheggnet.com 2181 > > imok > > > > wunder > > Walter Underwood > > wun...@wunderwood.org > &

Re: Zookeeper client Timeout - takes the default 30000 ms

2022-01-26 Thread Reej Nayagam
Thank you Shawn, I will try this way and see if it helps. *Thanks,* *Reej* On Wed, Jan 26, 2022 at 10:59 PM Shawn Heisey wrote: > On 1/26/22 06:53, Reej Nayagam wrote: > > The scenario is solr servers are up, but majority of the zk is down, > > so we need to tell the is

Re: Zookeeper client Timeout - takes the default 30000 ms

2022-01-26 Thread Reej Nayagam
7; > > On Wed, Jan 26, 2022 at 8:54 AM Reej Nayagam wrote: > > > > The scenario is solr servers are up, but majority of the zk is down, > > so we need to tell the issue is with the zookeeper. I don’t find a way on > > how to identify the zookeeper status without wai

Re: Zookeeper client Timeout - takes the default 30000 ms

2022-01-26 Thread Reej Nayagam
and? It's somewhere near there if not in CLUSTERSTATUS. > > For interacting with zookeeper itself I would probably just use zk > clients directly. > > > > On Wed, Jan 26, 2022 at 7:41 AM Reej Nayagam wrote: > > > > Hi All, > > > > I need to ha

Zookeeper client Timeout - takes the default 30000 ms

2022-01-26 Thread Reej Nayagam
Hi All, I need to handle zk failure and so monitoring the zk ensemble, and if the majority of the zk fails we'll activate the HA to point to a DB search. So to check if each of the zk is alive , we are connecting as below, *zkClient = solrZkClient(zkaddress,1),* *return zkclient.getSolrZooke

Re: How to handle zk failure

2022-01-25 Thread Reej Nayagam
r can't be guaranteed to be partition tolerant anymore in > all cases. > > Having said that, if you still want zookeeper to work properly even if 2 > nodes are down (somewhat unlike in most cases), then increase your > zookeeper cluster to have 5 nodes. > > Thanks, > Vina

How to handle zk failure

2022-01-25 Thread Reej Nayagam
Hi All, We are using solr 8.82 cloud setup with zk ensemble 3.6.3 (3 zk servers). We have a HA monitoring system which will ping every 5 mins to check if the solr URL or shards are available and ping is success. If there is a failure ( if solr servers are down) it=E2=80=99ll flag to switch the sea

Re: Already listening port is assigned to dsolr.stop port

2022-01-09 Thread Reej Nayagam
Thanks Shawn, this worked. Thank you for the explanation & the help. *Thanks & Regards* *Reej* On Mon, Jan 10, 2022 at 7:45 AM Shawn Heisey wrote: > On 1/7/2022 4:55 AM, Reej Nayagam wrote: > > I wonder how the existing port (8983) which is already being assigned to > &g

Fwd: Already listening port is assigned to dsolr.stop port

2022-01-09 Thread Reej Nayagam
Hi All, If anyone can explain me the solr and jetty port details, why there is a stop port for jetty and how to configure the stop ports as the stop port has taken the port in which another solr server is listening. Thanks Reej -- Forwarded message - From: Reej Nayagam Date: Fri

Fwd: Already listening port is assigned to dsolr.stop port

2022-01-07 Thread Reej Nayagam
Adding few more info Solr8 is https and solr4 is http. Please advise on how to proceed. Thanks Reej -- Forwarded message - From: Reej Nayagam Date: Fri, 7 Jan 2022 at 7:55 PM Subject: Already listening port is assigned to dsolr.stop port To: Solr Hi All, We have

Already listening port is assigned to dsolr.stop port

2022-01-07 Thread Reej Nayagam
Hi All, We have configured solr8 cloud with ports 3983 and 9983 in server 1 and server 2. So we have 2 solr servers 3 zookeepers running. We have an existing solr4 version with ports 6983 and 8983 running in the same server and different zk ports also. Now the issue is we didn’t do any config ch

Solr.log is not getting created

2022-01-04 Thread Reej Nayagam
Hi All, We have created a powershell script to run solr service in windows. When running the service from the command line, the solr.log is created at the configured path - SOLR_LOGS_DIR in solr.in.cmd file But when starting as a service, even if there is a solr.log file, it gets renamed to solr.

Re: Zookeeper log4j version upgrade

2021-12-30 Thread Reej Nayagam
Thanks Shawn. Let me try to make my people understand. They want to make it clear in the audit. Thanks for the information Regards Reej On Thu, 30 Dec 2021 at 10:45 PM, Shawn Heisey wrote: > On 12/30/2021 3:16 AM, Reej Nayagam wrote: > > The Latest zookeeper release also uses log4j1.

Zookeeper log4j version upgrade

2021-12-30 Thread Reej Nayagam
Hi All, The Latest zookeeper release also uses log4j1.x. Anyone has upgraded log4j version to 2 for zookeeper. We tried upgrading the log4j version, but the logs are not getting generated. If any of you have tried please let me know . Thank you *Regards,* *Reej*

Zookeeper Log4j 1.x to log4j 2.x

2021-12-27 Thread Reej Nayagam
Hi All, We are having Zookeeper 3.6.3. Though log4j1.x is not vulnerable we are advised to upgrade it to 2.17 in zookeeper. We removed the log4j1.x and replaced it with log4j-1.2-api-2.17.0.jar, log4j-api-2.17.0.jar and log4j-core-2.17.0.jar. Now the issue is we cannot see the zookeeper logs writt

Re: Log4j vulnerability- Solr4 - urgent pls

2021-12-11 Thread Reej Nayagam
rraguntla > wrote: > > > > >- -Dlog4j2.formatMsgNoLookups=true > > > > > > restart jvm with the above param and should work. > > > > > > > > > > > > On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam < > > reej...@

Re: Log4j vulnerability- Solr4 - urgent pls

2021-12-11 Thread Reej Nayagam
> wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Dec 11, 2021, at 8:28 PM, Reej Nayagam wrote: > > > > Thank you for your reply. > > > > It mentions Dlog4j2 but with solr4 it is log4j1.2.17 > > Can we use this command

Re: Log4j vulnerability- Solr4 - urgent pls

2021-12-11 Thread Reej Nayagam
above param and should work. > > > > > > On Saturday, December 11, 2021, 09:51:54 PM EST, Reej Nayagam < > reej...@gmail.com> wrote: > > Hi All, > > In production we are using solr4 which uses log4j-1.2.17.jar. > > Can someone say the mitigation

Log4j vulnerability- Solr4 - urgent pls

2021-12-11 Thread Reej Nayagam
Hi All, In production we are using solr4 which uses log4j-1.2.17.jar. Can someone say the mitigation option for solr4 Thanks Reej -- *Thanks,* *Reej*

Solr cloud DIH status command returning different status every time

2021-12-08 Thread Reej Nayagam
Hi All, We are using Solr 8.8.2 cloud setup. We have a monitoring app to monitor the solr indexing and alert if there is a failure. But the issue in SIT environment we have only around 10k to 15k documents in each repository and the indexing status and responses are always returned correctly, In D

Re: Bufferunderflowexception while indexing

2021-11-30 Thread Reej Nayagam
: > On 11/29/21 11:40 PM, Reej Nayagam wrote: > > We are using solr 8.8.2, upgraded oracle from 12c to 19c, so changed the > > driver from ojdbc6.jar to ojdbc8_g.jar > > Now while indexing, the logs shows the below. Sorry I cannot attach the > log > > file. > > &

Bufferunderflowexception while indexing

2021-11-29 Thread Reej Nayagam
Hi All, We are using solr 8.8.2, upgraded oracle from 12c to 19c, so changed the driver from ojdbc6.jar to ojdbc8_g.jar Now while indexing, the logs shows the below. Sorry I cannot attach the log file. A month back this environment was working fine, connecting to oracle 12C, and then the servers w

Re: Help Needed: Distributed update Async Exception solr 8.8.2 - Update

2021-08-26 Thread Reej Nayagam
Hi Shawn, Thanks a lot. Managed to stop the commits from client side and no more errors now. Appreciate your timely response. Thank you ! *Regards,* *Reej* On Wed, Aug 25, 2021 at 7:59 PM Shawn Heisey wrote: > On 8/24/2021 10:08 PM, Reej Nayagam wrote: > > Okay, Got your poin

Re: Help Needed: Distributed update Async Exception solr 8.8.2 - Update

2021-08-24 Thread Reej Nayagam
On Wed, Aug 25, 2021 at 12:03 AM Shawn Heisey wrote: > On 8/24/2021 1:51 AM, Reej M wrote: > > So my doubt now is, Do we need to do any changes or wire anything for > the optimisation and commit to be carried out effectively, as we are > ignoring commit here or does solr internally will handle it

Fwd: Help Needed: Distributed update Async Exception solr 8.8.2 - Update

2021-08-24 Thread Reej Nayagam
e let us know if any changes are required in the config. Also, I could see the Error in solrwriter in the postimportdeletequery "Exception while deleting query: " So not sure if this has completely fixed or it'll reoccur again, confused. *Thanks* *Reej* On 22 Aug 2021, at 7

Help Needed: Distributed update Async Exception solr 8.8.2

2021-08-22 Thread Reej Nayagam
Hi All, We are getting a distributed update async exception during indexing. Solr version v8.8.2 and we are using cloudsolrclient and passing 3 zk ip’s for connecting. This indexing happens every 15 mins and at times the entire indexing is getting wiped out with data being pulled from the DB. Earl

Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2

2021-08-20 Thread Reej Nayagam
her node fails, not sure why this > is happening. > Any advice is highly appreciated. > > *Thanks,* > *Reej* > > > On Tue, Aug 17, 2021 at 9:52 PM Shawn Heisey wrote: > >> On 8/16/2021 5:17 AM, Reej Nayagam wrote: >> > Hi Shawn , >> &

Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2

2021-08-19 Thread Reej Nayagam
Aug 17, 2021 at 9:52 PM Shawn Heisey wrote: > On 8/16/2021 5:17 AM, Reej Nayagam wrote: > > Hi Shawn , > > Please find the gc logs from our test server. Your help is much > appreciated. > > > > > https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8Cq

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-08-16 Thread Reej Nayagam
s.apache.org/jira/browse/SOLR-15587 > > Will get this fixed in 8.10 > > Cheers, > Tim > > On Thu, Jul 22, 2021 at 6:23 PM Reej Nayagam wrote: > > > > Thanks Vincenzo D'Amore & Shawn > > Hi Ours is a legacy system using EJB, no maven or graddle. And now

Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2

2021-08-16 Thread Reej Nayagam
Hi Shawn , Please find the gc logs from our test server. Your help is much appreciated. https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the 8.8.2 version should have this anyway right. I've incr

Re: JDBCDatasource was not closed - Possible resource leak

2021-08-07 Thread Reej Nayagam
> On 8 Aug 2021, at 11:05 AM, Reej Nayagam wrote: > > Hi All, > We are getting this error during one of our scheduled indexing. Solr 8.8.2. > We have configured the the dB connection in data-config.XML, dB is oracle. > > Error : JDBCDatasource was not closed prior to

JDBCDatasource was not closed - Possible resource leak

2021-08-07 Thread Reej Nayagam
Hi All, We are getting this error during one of our scheduled indexing. Solr 8.8.2. We have configured the the dB connection in data-config.XML, dB is oracle. Error : JDBCDatasource was not closed prior to finalise(), indicates a bug - POSSIBLE RESOURCE LEAK. Kindly advise how to fix this. Th

Re: Zookeeper Server Error - Zookeeper server not running

2021-07-29 Thread Reej Nayagam
g is still correct for the > current network setup. > On Jul 29, 2021, 5:50 AM -0500, Reej Nayagam , wrote: > > Hi All, > > > > Solr version - 8.8.2, Zookeeper - 3.6.3 > > we have 3 zk's configured and there were no changes. > > All of a sudden, we got a er

Re: Zookeeper Server Error - Zookeeper server not running [Help required]

2021-07-29 Thread Reej Nayagam
> Hi All, > > Solr version - 8.8.2, Zookeeper - 3.6.3 > > we have 3 zk's configured and there were no changes. > All of a sudden, we got a error, too many connection from 192.xx.xx. > maxconnection is 60. > > We stopped all the 3 zk's, cleared data and logs, and tried to restart, we > get the bel

Zookeeper Server Error - Zookeeper server not running

2021-07-29 Thread Reej Nayagam
Hi All, Solr version - 8.8.2, Zookeeper - 3.6.3 we have 3 zk's configured and there were no changes. All of a sudden, we got a error, too many connection from 192.xx.xx. maxconnection is 60. We stopped all the 3 zk's, cleared data and logs, and tried to restart, we get the below error myid:1 war

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-22 Thread Reej Nayagam
zookeeper > > > ensemble. Well, something like this: > > > > > >final List zkServers = new ArrayList(); > > >zkServers.add("zookeeper1:2181"); zkServers.add("zookeeper2:2181"); > > > zkServers.add("zookeeper3:2181"); > > >

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-22 Thread Reej Nayagam
Thanks Shawn In our application I saw noggit.jar added so after removing it now I could instantiate with the zk IP’s. thanks for the help On Thu, 22 Jul 2021 at 8:54 AM, Shawn Heisey wrote: > On 7/21/2021 6:37 PM, Reej Nayagam wrote: > > I tried earlier with zk ensemble, but when i t

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-21 Thread Reej Nayagam
l List zkServers = new ArrayList(); >zkServers.add("zookeeper1:2181"); zkServers.add("zookeeper2:2181"); > zkServers.add("zookeeper3:2181"); >final SolrClient client = new CloudSolrClient.Builder(zkServers, > Optional.empty()).build(); > > >

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-21 Thread Reej Nayagam
Yes the passed url is with https *Thanks,* *Reej* On Thu, Jul 22, 2021 at 3:40 AM Timothy Potter wrote: > Does your `solrURL` value you're passing to `CloudSolrClient.Builder` > start with `https`? > > On Wed, Jul 21, 2021 at 10:12 AM Reej Nayagam wrote: > > > >

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-21 Thread Reej Nayagam
-SolrCloud > >> > >> And double check the "clusterprop" urlScheme > >> > >> On Wed, Jul 21, 2021 at 11:01 AM Vincenzo D'Amore > >> wrote: > >> > >>> Sorry for being so rude, I was in a hurry and just a while after the >

Re: Cloudsolrclient.getclusterstateprovider - returns incorrect base_url [http instead of https] - Urgent pls help

2021-07-21 Thread Reej Nayagam
Hi Sorry, My offic system doesn't have internet for security reasons. The issue is we have enabled ssl in solr and we try to get the clusterstate as below cloudsolrclient.getclusterstateprovider.getclusterstate() then using the clusterstate doccollection DocCollection dc = clusterstate.getCollect

Re: Could not load collection from ZK (worked in solr4, failing in solr8) - Help Needed

2021-07-08 Thread Reej Nayagam
On Thu, 8 Jul 2021 at 8:10 PM, sachin gk wrote: > Have you upgraded the zookeeper too ? Sole 8.8.2 is compatible with > zookeeper 3.5.5 > zk version is 3.6.3 > > On Thu, Jul 8, 2021, 17:35 Yuval Paz wrote: > > > What error do you see? > > > > Also, what are you trying to do? If you are trying t

Re: HTTPSolrClient - help required [Singleton Recommended?]

2021-07-05 Thread Reej Nayagam
Thanks a lot Shawn. Regards Reej On Mon, 5 Jul 2021 at 1:27 PM, Shawn Heisey wrote: > On 7/4/2021 8:55 PM, Reej Nayagam wrote: > > Please find the versions as below > > Solr core - 8.8.2 > > Solrj - 8.8.2 > > Zookeeper - 3.6.3 > > Zookeeper jute - 3.6.3

Re: HTTPSolrClient - help required [Singleton Recommended?]

2021-07-04 Thread Reej Nayagam
Hi Shawn, Please find the versions as below Solr core - 8.8.2 Solrj - 8.8.2 Zookeeper - 3.6.3 Zookeeper jute - 3.6.3 Kindly do share any code samples if you have them. Thanks & Regards Reej On Fri, 2 Jul 2021 at 9:05 AM, Shawn Heisey wrote: > On 7/1/2021 6:36 PM, Reej M wrote: > > Hi Shawn /

Re: HTTPSolrClient - help required [Singleton Recommended?]

2021-07-04 Thread Reej Nayagam
ep the > > monitoring > > >> separate and to set different timeouts than for queries. > > >> > > >> This setup is simple and fast. With our biggest cluster, we index > about > > a > > >> half million documents per minute. > > >> >

Re: HTTPSolrClient - help required

2021-06-29 Thread Reej Nayagam
27;m used to reuse a CloudSolrClient instance for each > collection/index. > > On Tue, Jun 29, 2021 at 3:12 PM Shawn Heisey wrote: > > > On 6/29/2021 6:43 AM, Reej Nayagam wrote: > > > Hi Vincenzo Yes we are using cloud and initial solr version was 4.10.4 > > > and we upgr

Re: HTTPSolrClient - help required

2021-06-29 Thread Reej Nayagam
n, why are you looking for the leader? > >> On Tue, Jun 29, 2021 at 1:58 PM Vincenzo D'Amore wrote: >> >> not sure I got your answer... are you using solrcloud? >> >>> On Tue, Jun 29, 2021 at 12:18 PM Reej Nayagam wrote: >>> >>> Hi >&g

Re: HTTPSolrClient - help required

2021-06-29 Thread Reej Nayagam
indly help *Thanks,* *Reej* On Tue, Jun 29, 2021 at 6:17 PM Reej Nayagam wrote: > Hi > Thanks for the reply. > I'm trying to create a singleton, but my worry is my base url changes > based on the leader. > so we have 14 repositories, and if we need to trigger indexing for 1

Re: HTTPSolrClient - help required

2021-06-29 Thread Reej Nayagam
r speed. > > -Original Message- > From: Reej Nayagam > Sent: 29 June 2021 07:29 > To: Solr ; solr-user Lucene < > solr-u...@lucene.apache.org> > Subject: Fwd: HTTPSolrClient - help required > > Hi Team, > Any advise is highly appreciated. > Do we need

Fwd: HTTPSolrClient - help required

2021-06-28 Thread Reej Nayagam
Hi Team, Any advise is highly appreciated. Do we need to use a singleton connection of a connection pool. Please kindly advise Thanks Reej Sent from my iPhone Begin forwarded message: > From: Reej Nayagam > Date: 28 June 2021 at 11:53:58 AM SGT > To: Solr , solr-user Lucene >

HTTPSolrClient

2021-06-27 Thread Reej Nayagam
Hi Team, We recently upgraded to solr8.8.2 We have replaced all our httpsolrserver connections to httpsolrclient and haven't explicitly closed the connections. Even in solr4 we haven't closed the connections. But now our main application has become too slow and it was said there is TIME_WAIT from t

Re: Keeperexception

2021-06-06 Thread Reej Nayagam
> On 7 Jun 2021, at 11:58 AM, Shawn Heisey wrote: > > On 6/6/2021 8:35 AM, Reej Nayagam wrote: >>> This was working in both cluster and standalone setups earlier in >>> solr4.10.2 but in 8.8.2 was it not working. Kindly advise. >>> This code is to fetch th

Turn off Auto Add replicas and replica moving

2021-06-06 Thread Reej Nayagam
Hi Everyone, We are using solr 8.8.2. We have set autoaddReplicas=false during collection creation. We have a cluster setup with 2 servers and 4 nodes (2 nodes each). When one node of a server is shutdown replicas are not replicated, when another node of the server is also down, replicas were no

Fwd: Keeperexception

2021-06-06 Thread Reej Nayagam
Hi Everyone, > We are using solr 8.8.2. The solrjapi to fetch the clusterstate works fine > with the clustersetup, but throws keeperexception, keeper error=session > expired clusterstate.Jason or aliases.Json when trying in a standalone setup. > Tried restarting the zk but not working. > Code

Turn off Autoadd Replicas

2021-06-05 Thread Reej Nayagam
Hi Everyone, We are using solr 8.8.2. We have set autoaddReplicas=false during collection creation. We have a cluster setup with 2 servers and 4 nodes (2 nodes each). When one node of a server is shutdown replicas are not replicated, when another node of the server is also down, replicas were not a