Leak detected during repair (v2.1.8)

[Frisch, Michael]

ERROR [Reference-Reaper:1] 2015-07-29 12:34:45,941 Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@52628658) to class 
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1113728425:/data/cassandra/data/KSName/CFName/snapshots/8179b820-35ec-11e5-a75f-ed9e19db78e9/KSName-CFName-ka-5544
 was not released before the reference was garbage collected

ERROR [Reference-Reaper:1] 2015-07-29 12:34:45,948 Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@4eef1d76) to class 
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@880236150:/data/cassandra/data/KSName/CFName/snapshots/8179b820-35ec-11e5-a75f-ed9e19db78e9/KSName-CFName-ka-5543
 was not released before the reference was garbage collected

I was just wondering if this is a known issue or if others are experiencing 
this with C* v2.1.8.  I couldn't find any open tickets in the Jira about these 
leaks occurring during repairs.  I've only seen this during repairs and the 
only commonality that I could find between the column families that these leaks 
have been repeated for is that they contain very little data.  Could there be a 
race condition that's only hit when repairing very small datasets?  Turning on 
debug logging didn't produce any more context like I had hoped for.

Regards,
- Mike

problem with write_survey

[aeljami.ext]

Hello,

I start a node with the write survey = true

-Dcassandra.write_survey=true

Log:

INFO [main] 2015-07-29 15:29:35,697 StorageService.java (line 853) Startup 
complete, but write survey mode is active, not becoming an active ring member. 
Use JMX (StorageService->joinRing()) to finalize ring joining.

but the node allows read:

cqlsh> select * from myTable  where id = 1745;

id   | fname | lname
--+---+---
1745 |  john | smith

(Key 1745 exists on node with the the write survey = true)
Then, when attempting a transition a node from write survey to normal mode, the 
"nodetool join" invocation fails as the node is already "joined".

Do you have any idea ?

Cassandra Version: 2.0.10



_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Re: Leak detected during repair (v2.1.8)

[Jeff Jirsa]

https://issues.apache.org/jira/browse/CASSANDRA-9382


From:  "Frisch, Michael"
Reply-To:  "user@cassandra.apache.org"
Date:  Wednesday, July 29, 2015 at 6:56 AM
To:  Cassandra List
Subject:  Leak detected during repair (v2.1.8)

ERROR [Reference-Reaper:1] 2015-07-29 12:34:45,941 Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@52628658) to class 
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1113728425:/data/cassandra/data/KSName/CFName/snapshots/8179b820-35ec-11e5-a75f-ed9e19db78e9/KSName-CFName-ka-5544
 was not released before the reference was garbage collected

ERROR [Reference-Reaper:1] 2015-07-29 12:34:45,948 Ref.java:179 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@4eef1d76) to class 
org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@880236150:/data/cassandra/data/KSName/CFName/snapshots/8179b820-35ec-11e5-a75f-ed9e19db78e9/KSName-CFName-ka-5543
 was not released before the reference was garbage collected


I was just wondering if this is a known issue or if others are experiencing 
this with C* v2.1.8.  I couldn't find any open tickets in the Jira about these 
leaks occurring during repairs.  I've only seen this during repairs and the 
only commonality that I could find between the column families that these leaks 
have been repeated for is that they contain very little data.  Could there be a 
race condition that's only hit when repairing very small datasets?  Turning on 
debug logging didn't produce any more context like I had hoped for.

Regards,
- Mike



smime.p7s
Description: S/MIME cryptographic signature

Re: problem with write_survey

[Robert Coli]

On Wed, Jul 29, 2015 at 8:04 AM,  wrote:

>  but the node allows read:
>
>
>
> cqlsh> select * from myTable  where id = 1745;
>
>
>
> id   | fname | lname
>
> --+---+---
>
> 1745 |  john | smith
>
>
This is unexpected. Does the node show up in eg nodetool ring? If so, you
are probably failing to properly use write_survey.

Do you see this log message?

logger.info("Startup complete, but write survey mode is active, not
becoming an active ring member. Use JMX (StorageService->joinRing()) to
finalize ring joining.");


> (Key 1745 exists on node with the the write survey = true)
>
> Then, when attempting a transition a node from write survey to normal
> mode, the "nodetool join" invocation fails as the node is already "joined".
>
>
>
> Do you have any idea ?
>
>
>
> Cassandra Version: 2.0.10
>

https://issues.apache.org/jira/browse/CASSANDRA-9740

Is fixed in most recent 2.0.x, 2.0.17.

=Rob

Re: Reduced write performance when reading

[Robert Coli]

On Tue, Jul 28, 2015 at 4:49 PM, Soerian Lieve  wrote:

> I did already set that to the number of cores of the machines (24), but it
> made no difference.
>

I continue to suggest that you file a JIRA ticket... I feel you have done
sufficient community based due dilligence to question whether this
performance is expected on the server side. You have a repro path to
illustrate the behavior. Why not file a JIRA?

(and respond on thread with its url...)

=Rob

Re: Reduced write performance when reading

[Soerian Lieve]

Found the problem, it turns out that what Bharatendra suggested was
correct.
I had set the memtable_flush_writers to equal the number of cores but
hadn't restarted the Cassandra process, so they didn't take the
configuration.

On Wed, Jul 29, 2015 at 12:59 PM, Robert Coli  wrote:

> On Tue, Jul 28, 2015 at 4:49 PM, Soerian Lieve 
> wrote:
>
>> I did already set that to the number of cores of the machines (24), but
>> it made no difference.
>>
>
> I continue to suggest that you file a JIRA ticket... I feel you have done
> sufficient community based due dilligence to question whether this
> performance is expected on the server side. You have a repro path to
> illustrate the behavior. Why not file a JIRA?
>
> (and respond on thread with its url...)
>
> =Rob
>
>

RE: problem with write_survey

[aeljami.ext]

Do you see this log message?

logger.info("Startup complete, but write survey mode is 
active, not becoming an active ring member. Use JMX 
(StorageService->joinRing()) to finalize ring joining.");



ð  Yes !


Log : INFO [main] 2015-07-29 15:29:35,697 StorageService.java (line 853) 
Startup complete, but write survey mode is active, not becoming an active ring 
member. Use JMX (StorageService->joinRing()) to finalize ring joining.


De : Robert Coli [mailto:rc...@eventbrite.com]
Envoyé : mercredi 29 juillet 2015 21:03
À : user@cassandra.apache.org
Objet : Re: problem with write_survey

On Wed, Jul 29, 2015 at 8:04 AM, 
mailto:aeljami@orange.com>> wrote:
but the node allows read:

cqlsh> select * from myTable  where id = 1745;

id   | fname | lname
--+---+---
1745 |  john | smith

This is unexpected. Does the node show up in eg nodetool ring? If so, you are 
probably failing to properly use write_survey.

Do you see this log message?

logger.info("Startup complete, but write survey mode is 
active, not becoming an active ring member. Use JMX 
(StorageService->joinRing()) to finalize ring joining.");

(Key 1745 exists on node with the the write survey = true)
Then, when attempting a transition a node from write survey to normal mode, the 
"nodetool join" invocation fails as the node is already "joined".

Do you have any idea ?

Cassandra Version: 2.0.10

https://issues.apache.org/jira/browse/CASSANDRA-9740

Is fixed in most recent 2.0.x, 2.0.17.

=Rob


_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.