RE: [EXTERNAL] Lot of hints piling up

2017-12-21 Thread Amit Singh 
Check if any warn sign stating “ Node is down” or “Node is coming up” in 
system.log and also see on that node using nodetool tpstats whether dropped 
mutations are there or not.

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] 
Sent: Wednesday, October 25, 2017 3:37 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Lot of hints piling up

 

No OOM or HEAP errors

 

On Tue, Oct 24, 2017 at 2:51 PM, Mohapatra, Kishore 
<kishore.mohapa...@nuance.com> wrote:

Check how many sstables are there for the table you are having issues with.

You might be having the Heap pressure. Check your system.log for any OOM error 
or HEAP error.

 

Thanks

 

Kishore Mohapatra

Principal Operations DBA

Seattle, WA

Email :  <mailto:kishore.mohapa...@nuance.com> kishore.mohapa...@nuance.com

 

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] 
Sent: Monday, October 23, 2017 11:54 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Lot of hints piling up

 

Do not see any errors in the logs or OS and compactions are happening in the 
regular interval and good too.

 

 

Issue here is, this causing replication lag across the datacenters.

 

On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore 
<kishore.mohapa...@nuance.com> wrote:

Do you see any error in the cassandra log ?

Check compactionstats ?

Also check the OS level log messages to see if you are getting hardware level 
error messages.

 

Thanks

 

Kishore Mohapatra

Principal Operations DBA

Seattle, WA

Ph : 425-691-6417 (cell)

Email :  <mailto:kishore.mohapa...@nuance.com> kishore.mohapa...@nuance.com

 

 

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] 
Sent: Friday, October 20, 2017 9:44 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Lot of hints piling up

 

Hello,

 

We have cassandra cluster in 3 regions with version 2.1.13, and all of a sudden 
we started seeing lot of hints accumulating on the nodes. We are pretty sure 
there is no issue with the network between the regions and all the nodes are up 
and running all the time.

 

Is there any other reason for the hints accumulation other than the n/w? eg: 
wide rows or bigger objects?

 

Any pointers here could be very helpful.

 

b/w the hints gets processed after some point of time.

Re: [EXTERNAL] Lot of hints piling up

2017-10-24 Thread Jai Bheemsen Rao Dhanwada 
No OOM or HEAP errors

On Tue, Oct 24, 2017 at 2:51 PM, Mohapatra, Kishore <
kishore.mohapa...@nuance.com> wrote:

> Check how many sstables are there for the table you are having issues with.
>
> You might be having the Heap pressure. Check your system.log for any OOM
> error or HEAP error.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Email : kishore.mohapa...@nuance.com
>
>
>
>
>
> *From:* Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
> *Sent:* Monday, October 23, 2017 11:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Lot of hints piling up
>
>
>
> Do not see any errors in the logs or OS and compactions are happening in
> the regular interval and good too.
>
>
>
>
>
> Issue here is, this causing replication lag across the datacenters.
>
>
>
> On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore <
> kishore.mohapa...@nuance.com> wrote:
>
> Do you see any error in the cassandra log ?
>
> Check compactionstats ?
>
> Also check the OS level log messages to see if you are getting hardware
> level error messages.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Ph : 425-691-6417 (cell)
>
> Email : kishore.mohapa...@nuance.com
>
>
>
>
>
> *From:* Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
> *Sent:* Friday, October 20, 2017 9:44 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Lot of hints piling up
>
>
>
> Hello,
>
>
>
> We have cassandra cluster in 3 regions with version 2.1.13, and all of a
> sudden we started seeing lot of hints accumulating on the nodes. We are
> pretty sure there is no issue with the network between the regions and all
> the nodes are up and running all the time.
>
>
>
> Is there any other reason for the hints accumulation other than the n/w?
> eg: wide rows or bigger objects?
>
>
>
> Any pointers here could be very helpful.
>
>
>
> b/w the hints gets processed after some point of time.
>
>
>

RE: [EXTERNAL] Lot of hints piling up

2017-10-24 Thread Mohapatra, Kishore 
Check how many sstables are there for the table you are having issues with.
You might be having the Heap pressure. Check your system.log for any OOM error 
or HEAP error.

Thanks

Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Email : kishore.mohapa...@nuance.com<mailto:kishore.mohapa...@nuance.com>


From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
Sent: Monday, October 23, 2017 11:54 AM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Lot of hints piling up

Do not see any errors in the logs or OS and compactions are happening in the 
regular interval and good too.


Issue here is, this causing replication lag across the datacenters.

On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore 
<kishore.mohapa...@nuance.com<mailto:kishore.mohapa...@nuance.com>> wrote:
Do you see any error in the cassandra log ?
Check compactionstats ?
Also check the OS level log messages to see if you are getting hardware level 
error messages.

Thanks

Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Ph : 425-691-6417 (cell)
Email : kishore.mohapa...@nuance.com<mailto:kishore.mohapa...@nuance.com>


From: Jai Bheemsen Rao Dhanwada 
[mailto:jaibheem...@gmail.com<mailto:jaibheem...@gmail.com>]
Sent: Friday, October 20, 2017 9:44 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Lot of hints piling up

Hello,

We have cassandra cluster in 3 regions with version 2.1.13, and all of a sudden 
we started seeing lot of hints accumulating on the nodes. We are pretty sure 
there is no issue with the network between the regions and all the nodes are up 
and running all the time.

Is there any other reason for the hints accumulation other than the n/w? eg: 
wide rows or bigger objects?

Any pointers here could be very helpful.

b/w the hints gets processed after some point of time.

Re: [EXTERNAL] Lot of hints piling up

2017-10-23 Thread Jai Bheemsen Rao Dhanwada 
Do not see any errors in the logs or OS and compactions are happening in
the regular interval and good too.


Issue here is, this causing replication lag across the datacenters.

On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore <
kishore.mohapa...@nuance.com> wrote:

> Do you see any error in the cassandra log ?
>
> Check compactionstats ?
>
> Also check the OS level log messages to see if you are getting hardware
> level error messages.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Ph : 425-691-6417 (cell)
>
> Email : kishore.mohapa...@nuance.com
>
>
>
>
>
> *From:* Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
> *Sent:* Friday, October 20, 2017 9:44 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Lot of hints piling up
>
>
>
> Hello,
>
>
>
> We have cassandra cluster in 3 regions with version 2.1.13, and all of a
> sudden we started seeing lot of hints accumulating on the nodes. We are
> pretty sure there is no issue with the network between the regions and all
> the nodes are up and running all the time.
>
>
>
> Is there any other reason for the hints accumulation other than the n/w?
> eg: wide rows or bigger objects?
>
>
>
> Any pointers here could be very helpful.
>
>
>
> b/w the hints gets processed after some point of time.
>

RE: [EXTERNAL] Lot of hints piling up

2017-10-23 Thread Mohapatra, Kishore 
Do you see any error in the cassandra log ?
Check compactionstats ?
Also check the OS level log messages to see if you are getting hardware level 
error messages.

Thanks

Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Ph : 425-691-6417 (cell)
Email : kishore.mohapa...@nuance.com


From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
Sent: Friday, October 20, 2017 9:44 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Lot of hints piling up

Hello,

We have cassandra cluster in 3 regions with version 2.1.13, and all of a sudden 
we started seeing lot of hints accumulating on the nodes. We are pretty sure 
there is no issue with the network between the regions and all the nodes are up 
and running all the time.

Is there any other reason for the hints accumulation other than the n/w? eg: 
wide rows or bigger objects?

Any pointers here could be very helpful.

b/w the hints gets processed after some point of time.