By default user_function_timeout_policy is set to die i.e. warn and kill the
JVM. Please find below a source code snippet that outlines possible setting.
/**
* Defines what to do when a UDF ran longer than
user_defined_function_fail_timeout.
* Possible options are:
* - 'die' - i.e. it is able to emit a warning to the client before the
Cassandra Daemon will shut down.
* - 'die_immediate' - shut down C* daemon immediately (effectively prevent
the chance that the client will receive a warning).
* - 'ignore' - just log - the most dangerous option.
* (Only valid, if enable_user_defined_functions_threads==true)
*/
public UserFunctionTimeoutPolicy user_function_timeout_policy =
UserFunctionTimeoutPolicy.die;
To answer your question. Yes it is normal for Cassandra to shut down due to a
rogue UDF.
Warm Regards,
Akhil Mehra
> On 30/06/2017, at 11:17 AM, Gopal, Dhruva <dhruva.go...@aspect.com> wrote:
>
> Hi –
> Is it normal for cassand to be shutdown forcefully on timeout exceptions
> when using UDFs? We are admittedly trying some load tests on our dev
> environments which may be somewhat constrained, but didn’t expect to see
> forceful shutdowns such as these when we ran our tests. We’re running
> Cassandra 3.10. Dev environment is an MBP (Core i7/16GB RAM). Sample error
> below – feedback will be much appreciated.
>
> ERROR [NonPeriodicTasks:1] 2017-06-29 10:48:54,476
> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.util.concurrent.TimeoutException: User defined function reporting.latest
> : (map<text, frozen<tuple<bigint, text>>>, bigint, timestamp, boolean) ->
> map<text, frozen<tuple<bigint, text>>> ran longer than 1500ms - will stop
> Cassandra VM
> at
> org.apache.cassandra.cql3.functions.UDFunction.async(UDFunction.java:483)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.functions.UDFunction.executeAsync(UDFunction.java:398)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.functions.UDFunction.execute(UDFunction.java:298)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.selection.ScalarFunctionSelector.getOutput(ScalarFunctionSelector.java:61)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.selection.Selection$SelectionWithProcessing$1.getOutputRow(Selection.java:592)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:430)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:523)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:500)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_101]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> ~[apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
>
> Regards,
> DHRUVA GOPAL
> SR. MANAGER, ENGINEERING
> REPORTING, ANALYTICS AND BIG DATA
> +1 408.325.2011 WORK
> +1 408.219.1094 MOBILE
> UNITED STATES
> dhruva.go...@aspect.com <mailto:dhruva.go...@aspect.com>
> aspect.com <http://www.aspect.com/>
> <image001.png>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and destroy
> any copies. You may not further disclose or distribute this email or its
> attachments.
