New Release of ecAudit

2019-08-01 Thread Per Otterström 
We are pleased to announce the release of ecAudit 2.1.0.

The ecAudit plug-in enhance Cassandra 3.x with audit logging functionality. In 
this release we've added support for a high-performance logging backend based 
on Chronicle as well as added new logging parameters and configuration options.

The full list of improvements are:

  *   Make wrapped authorizer backend configurable - #104
  *   Optionally skip values when logging prepared statements - #92
  *   Add support for client port in log messages - #90
  *   Add support for post-logging - #24
  *   Add support for host address in log message - #28
  *   Add support for Chronicle-Queue backend - #62
  *   Add metrics for filtering and logging - #72
  *   Add support for system timestamp in log message - #27
  *   Fix typo of java property for custom audit.yaml path - #59
  *   Build with Cassandra 3.11.4 (only flavor ecaudit_c3.11)
  *   Build with Cassandra 3.0.18 (only flavor ecaudit_c3.0)
  *   Introduce configurable log message format - #55
  *   Make the audit whitelist table a protected resource in Cassandra

This release comes in three flavors (same features, but for different Cassandra 
versions).

ecAudit for Cassandra 3.11.X:
Doc: https://github.com/Ericsson/ecaudit/tree/master
SW: 
https://search.maven.org/search?q=g:com.ericsson.bss.cassandra.ecaudit%20AND%20a:ecaudit_c3.11

ecAudit for Cassandra 3.0.X:
Doc: https://github.com/Ericsson/ecaudit/tree/release/c3.0
SW: 
https://search.maven.org/search?q=g:com.ericsson.bss.cassandra.ecaudit%20AND%20a:ecaudit_c3.0

ecAudit for Cassandra 3.0.11:
Doc: https://github.com/Ericsson/ecaudit/tree/release/c3.0.11
SW: 
https://search.maven.org/search?q=g:com.ericsson.bss.cassandra.ecaudit%20AND%20a:ecaudit_c3.0.11

Feel free to raise a ticket on GitHub if you have a feature request or want to 
contribute. Enjoy!

/The Ericsson team

Re: Repair failed and crash the node, how to bring it back?

2019-08-01 Thread Martin Xue 
Thanks ASAD, I will look more into it.
Regards
Martin

On Thu, Aug 1, 2019 at 11:40 PM ZAIDI, ASAD A  wrote:

> I don’t think anyone can predict with certainty if instance won’t crash
> but there are good chances it will -  unless you take remedial actions.
>
> If you are not doing subrange repair, a lot of merkle tree data can
> potentially be scanned/streamed taking toll on memory resources – that ,
> taking  account of all other running operations , easily bust available
> memory.
>
>
>
> You can do few things like – as short term measure – increase allotted
> heap size along with running subrange repair with script
>  or by using
> reaper tool.
>
> You may also want to check partition sizes of tables (nodetool tablestats)
> if they’re bloated. See if table scans  are infested with lots of
> tombstones which in turn also tax on heap consumption. My $.002 cents for
> the moment.
>
>
>
>
>
>
>
> *From:* Martin Xue [mailto:martin...@gmail.com]
> *Sent:* Wednesday, July 31, 2019 5:05 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Repair failed and crash the node, how to bring it back?
>
>
>
> Hi Alex,
>
>
>
> Thanks for your reply. The disk space was around 80%. The crash happened
> during repair, primary range full repair on 1TB keyspace.
>
>
>
> Would that crash again?
>
>
>
> Thanks
>
> Regards
>
> Martin
>
>
>
> On Thu., 1 Aug. 2019, 12:04 am Alexander Dejanovski, <
> a...@thelastpickle.com> wrote:
>
> It looks like you have a corrupted hint file.
>
> Did the node run out of disk space while repair was running?
>
>
>
> You might want to move the hint files off their current directory and try
> to restart the node again.
>
> Since you'll have lost mutations then, you'll need... to run repair ¯\_(ツ
> )_/¯
>
>
>
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
>
> http://www.thelastpickle.com
> 
>
>
>
>
>
> On Wed, Jul 31, 2019 at 3:51 PM Martin Xue  wrote:
>
> Hi,
>
>
>
> I am running repair on production, started with one of 6 nodes in the
> cluster (3 nodes in each of two DC). Cassandra version 3.0.14.
>
>
>
> running: repair -pr --full keyspace on node 1, 1TB data, takes two days,
> and crash,
>
>
>
> error shows:
>
> 3202]] finished (progress: 3%)
> Exception occurred during clean-up.
> java.lang.reflect.UndeclaredThrowableException
> Cassandra has shutdown.
> error: [2019-07-31 20:19:20,797] JMX connection closed. You should check
> server log for repair status of keyspace keyspace_masked (Subsequent
> keyspaces are not going to be repaired).
> -- StackTrace --
> java.io.IOException: [2019-07-31 20:19:20,797] JMX connection closed. You
> should check server log for repair status of keyspace keyspace_masked
> keyspaces are not going to be repaired).
> at
> org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
> at
> org.apache.cassandra.tools.RepairRunner.handleConnectionClosed(RepairRunner.java:91)
> at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:90)
> at
> javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
> at
> javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
> at
> javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
> at
> javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
> at
> javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
> at
> javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:533)
> at
> javax.management.remote.rmi.RMIConnector.access$1300(RMIConnector.java:121)
> at
> javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1534)
> at
> javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
>
>
>
> system.log shows
>
>

Re: Repair failed and crash the node, how to bring it back?

2019-08-01 Thread Martin Xue 
Hi Alex,

Thanks, much appreciated.

Regards
Martin


On Thu, Aug 1, 2019 at 3:34 PM Alexander Dejanovski 
wrote:

> Hi Martin,
>
> apparently this is the bug you've been hit by on hints :
> https://issues.apache.org/jira/browse/CASSANDRA-14080
> It was fixed in 3.0.17.
>
> You didn't provide the logs from Cassandra at the time of the crash, only
> the output of nodetool, so it's hard to say what caused it. You may be hit
> by this bug: https://issues.apache.org/jira/browse/CASSANDRA-14096
> This is unlikely to happen with Reaper (as mentioned in the description of
> the ticket) since it will generate smaller Merkle trees as subrange covers
> less partitions for each repair session.
>
> So the advice is : upgrade to 3.0.19 (even 3.11.4 IMHO as 3.0 offers less
> performance than 3.11) and use Reaper  to
> handle/schedule repairs.
>
> Cheers,
>
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> On Thu, Aug 1, 2019 at 12:05 AM Martin Xue  wrote:
>
>> Hi Alex,
>>
>> Thanks for your reply. The disk space was around 80%. The crash happened
>> during repair, primary range full repair on 1TB keyspace.
>>
>> Would that crash again?
>>
>> Thanks
>> Regards
>> Martin
>>
>> On Thu., 1 Aug. 2019, 12:04 am Alexander Dejanovski, <
>> a...@thelastpickle.com> wrote:
>>
>>> It looks like you have a corrupted hint file.
>>> Did the node run out of disk space while repair was running?
>>>
>>> You might want to move the hint files off their current directory and
>>> try to restart the node again.
>>> Since you'll have lost mutations then, you'll need... to run repair
>>> ¯\_(ツ)_/¯
>>>
>>> -
>>> Alexander Dejanovski
>>> France
>>> @alexanderdeja
>>>
>>> Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>>
>>> On Wed, Jul 31, 2019 at 3:51 PM Martin Xue  wrote:
>>>
 Hi,

 I am running repair on production, started with one of 6 nodes in the
 cluster (3 nodes in each of two DC). Cassandra version 3.0.14.

 running: repair -pr --full keyspace on node 1, 1TB data, takes two
 days, and crash,

 error shows:
 3202]] finished (progress: 3%)
 Exception occurred during clean-up.
 java.lang.reflect.UndeclaredThrowableException
 Cassandra has shutdown.
 error: [2019-07-31 20:19:20,797] JMX connection closed. You should
 check server log for repair status of keyspace keyspace_masked (Subsequent
 keyspaces are not going to be repaired).
 -- StackTrace --
 java.io.IOException: [2019-07-31 20:19:20,797] JMX connection closed.
 You should check server log for repair status of keyspace keyspace_masked
 keyspaces are not going to be repaired).
 at
 org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
 at
 org.apache.cassandra.tools.RepairRunner.handleConnectionClosed(RepairRunner.java:91)
 at
 org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:90)
 at
 javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
 at
 javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
 at
 javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
 at
 javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
 at
 javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
 at
 javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:533)
 at
 javax.management.remote.rmi.RMIConnector.access$1300(RMIConnector.java:121)
 at
 javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1534)
 at
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
 at
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
 at
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
 at
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
 at
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
 at
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

 system.log shows
 INFO  [Service Thread] 2019-07-31 20:19:08,579 GCInspector.java:284 -
 G1 Young Generation GC in 2915ms.  G1 Eden Space: 914358272 -> 0; G1 Old
>

Logon trigger

2019-08-01 Thread Abdul Patel 
Hi All,

As per normal databases have logon trigger, do we have similar option in cassandra.
Would like to implement ip restrictions in cassandra, or any other method?

RE: Repair failed and crash the node, how to bring it back?

2019-08-01 Thread ZAIDI, ASAD A 
I don’t think anyone can predict with certainty if instance won’t crash but 
there are good chances it will -  unless you take remedial actions.
If you are not doing subrange repair, a lot of merkle tree data can potentially 
be scanned/streamed taking toll on memory resources – that , taking  account of 
all other running operations , easily bust available memory.

You can do few things like – as short term measure – increase allotted heap 
size along with running subrange repair with 
script or by using 
reaper tool.
You may also want to check partition sizes of tables (nodetool tablestats) if 
they’re bloated. See if table scans  are infested with lots of tombstones which 
in turn also tax on heap consumption. My $.002 cents for the moment.



From: Martin Xue [mailto:martin...@gmail.com]
Sent: Wednesday, July 31, 2019 5:05 PM
To: user@cassandra.apache.org
Subject: Re: Repair failed and crash the node, how to bring it back?

Hi Alex,

Thanks for your reply. The disk space was around 80%. The crash happened during 
repair, primary range full repair on 1TB keyspace.

Would that crash again?

Thanks
Regards
Martin

On Thu., 1 Aug. 2019, 12:04 am Alexander Dejanovski, 
mailto:a...@thelastpickle.com>> wrote:
It looks like you have a corrupted hint file.
Did the node run out of disk space while repair was running?

You might want to move the hint files off their current directory and try to 
restart the node again.
Since you'll have lost mutations then, you'll need... to run repair ¯\_(ツ)_/¯

-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Wed, Jul 31, 2019 at 3:51 PM Martin Xue 
mailto:martin...@gmail.com>> wrote:
Hi,

I am running repair on production, started with one of 6 nodes in the cluster 
(3 nodes in each of two DC). Cassandra version 3.0.14.

running: repair -pr --full keyspace on node 1, 1TB data, takes two days, and 
crash,

error shows:
3202]] finished (progress: 3%)
Exception occurred during clean-up. 
java.lang.reflect.UndeclaredThrowableException
Cassandra has shutdown.
error: [2019-07-31 20:19:20,797] JMX connection closed. You should check server 
log for repair status of keyspace keyspace_masked (Subsequent keyspaces are not 
going to be repaired).
-- StackTrace --
java.io.IOException: [2019-07-31 20:19:20,797] JMX connection closed. You 
should check server log for repair status of keyspace keyspace_masked keyspaces 
are not going to be repaired).
at 
org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at 
org.apache.cassandra.tools.RepairRunner.handleConnectionClosed(RepairRunner.java:91)
at 
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:90)
at 
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at 
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at 
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at 
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at 
javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.close(RMIConnector.java:533)
at 
javax.management.remote.rmi.RMIConnector.access$1300(RMIConnector.java:121)
at 
javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1534)
at 
javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:471)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
at 
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)

system.log shows
INFO  [Service Thread] 2019-07-31 20:19:08,579 GCInspector.java:284 - G1 Young 
Generation GC in 2915ms.  G1 Eden Space: 914358272 -> 0; G1 Old Gen: 
19043999248 -> 20219035248;
INFO  [Service Thread] 2019-07-31 20:19:08,579 StatusLogger.java:52 - Pool Name 
   Active   Pending  Completed   Blocked  All Time Blocked
INFO