[
https://issues.apache.org/jira/browse/STORM-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Schonfeld updated STORM-1375:
-------------------------------------
Description:
When using the new Pacemaker (which, you kind of have to, cos the alternative
of not using it is abusing ZK to the point of deadlocking it...), you cannot
submit topologies because _submitTopology_ is calling ClusterState's
_delete_node_blobstore_ which never got implemented in
_pacemaker_state_factory.clj_.
Here's a nice stack trace:
{noformat}
2015-12-05 07:07:21 b.s.d.nimbus [WARN] Topology submission exception.
(topology name='twitter-cron-ba161de') #error {
:cause
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
:via
[{:type java.lang.AbstractMethodError
:message
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
:at [sun.reflect.NativeMethodAccessorImpl invoke0
NativeMethodAccessorImpl.java -2]}]
:trace
[[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
-2]
[sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
[sun.reflect.DelegatingMethodAccessorImpl invoke
DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 497]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[backtype.storm.cluster$mk_storm_cluster_state$reify__3846
setup_blobstore_BANG_ cluster.clj 345]
[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
-2]
[sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
[sun.reflect.DelegatingMethodAccessorImpl invoke
DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 497]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[backtype.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 467]
[backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803
submitTopologyWithOpts nimbus.clj 1523]
[backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
Nimbus.java 2940]
[backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
Nimbus.java 2924]
[org.apache.thrift7.ProcessFunction process ProcessFunction.java 39]
[org.apache.thrift7.TBaseProcessor process TBaseProcessor.java 39]
[backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor
process SimpleTransportPlugin.java 158]
[org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer invoke
AbstractNonblockingServer.java 518]
[org.apache.thrift7.server.Invocation run Invocation.java 18]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java
617]
[java.lang.Thread run Thread.java 745]]}
2015-12-05 07:07:21 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR]
Unexpected throwable while invoking!
java.lang.AbstractMethodError:
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at
backtype.storm.cluster$mk_storm_cluster_state$reify__3846.setup_blobstore_BANG_(cluster.clj:345)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:467)
at
backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803.submitTopologyWithOpts(nimbus.clj:1523)
at
backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2940)
at
backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2924)
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39)
at
backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at
org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
was:
When using the new Pacemaker (which, you kind of have to, cos the alternative
of not using it is abusing ZK to the point of deadlocking it...), you cannot
submit topologies because _submitTopology_ is calling ClusterState's
_delete_node_blobstore_ which never got implemented in
_pacemaker_state_factory.clj_.
Here's a nice stack trace:
{noformat}
2015-12-05 07:07:21 b.s.d.nimbus [WARN] Topology submission exception.
(topology name='twitter-cron-ba161de') #error {
:cause
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
:via
[{:type java.lang.AbstractMethodError
:message
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
:at [sun.reflect.NativeMethodAccessorImpl invoke0
NativeMethodAccessorImpl.java -2]}]
:trace
[[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
-2]
[sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
[sun.reflect.DelegatingMethodAccessorImpl invoke
DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 497]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[backtype.storm.cluster$mk_storm_cluster_state$reify__3846
setup_blobstore_BANG_ cluster.clj 345]
[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
-2]
[sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
[sun.reflect.DelegatingMethodAccessorImpl invoke
DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 497]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[backtype.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 467]
[backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803
submitTopologyWithOpts nimbus.clj 1523]
[backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
Nimbus.java 2940]
[backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
Nimbus.java 2924]
[org.apache.thrift7.ProcessFunction process ProcessFunction.java 39]
[org.apache.thrift7.TBaseProcessor process TBaseProcessor.java 39]
[backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor
process SimpleTransportPlugin.java 158]
[org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer invoke
AbstractNonblockingServer.java 518]
[org.apache.thrift7.server.Invocation run Invocation.java 18]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java
617]
[java.lang.Thread run Thread.java 745]]}
2015-12-05 07:07:21 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR]
Unexpected throwable while invoking!
java.lang.AbstractMethodError:
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at
backtype.storm.cluster$mk_storm_cluster_state$reify__3846.setup_blobstore_BANG_(cluster.clj:345)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:467)
at
backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803.submitTopologyWithOpts(nimbus.clj:1523)
at
backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2940)
at
backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2924)
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39)
at
backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at
org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{/noformat}
> Blobstore broke Pacemaker
> -------------------------
>
> Key: STORM-1375
> URL: https://issues.apache.org/jira/browse/STORM-1375
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 0.11.0
> Reporter: Michael Schonfeld
> Assignee: Ilya Ostrovskiy
> Priority: Critical
> Fix For: 0.11.0
>
>
> When using the new Pacemaker (which, you kind of have to, cos the alternative
> of not using it is abusing ZK to the point of deadlocking it...), you cannot
> submit topologies because _submitTopology_ is calling ClusterState's
> _delete_node_blobstore_ which never got implemented in
> _pacemaker_state_factory.clj_.
> Here's a nice stack trace:
> {noformat}
> 2015-12-05 07:07:21 b.s.d.nimbus [WARN] Topology submission exception.
> (topology name='twitter-cron-ba161de') #error {
> :cause
> org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
> :via
> [{:type java.lang.AbstractMethodError
> :message
> org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
> :at [sun.reflect.NativeMethodAccessorImpl invoke0
> NativeMethodAccessorImpl.java -2]}]
> :trace
> [[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
> -2]
> [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java
> 62]
> [sun.reflect.DelegatingMethodAccessorImpl invoke
> DelegatingMethodAccessorImpl.java 43]
> [java.lang.reflect.Method invoke Method.java 497]
> [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
> [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
> [backtype.storm.cluster$mk_storm_cluster_state$reify__3846
> setup_blobstore_BANG_ cluster.clj 345]
> [sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
> -2]
> [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java
> 62]
> [sun.reflect.DelegatingMethodAccessorImpl invoke
> DelegatingMethodAccessorImpl.java 43]
> [java.lang.reflect.Method invoke Method.java 497]
> [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
> [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
> [backtype.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 467]
> [backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803
> submitTopologyWithOpts nimbus.clj 1523]
> [backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
> Nimbus.java 2940]
> [backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts getResult
> Nimbus.java 2924]
> [org.apache.thrift7.ProcessFunction process ProcessFunction.java 39]
> [org.apache.thrift7.TBaseProcessor process TBaseProcessor.java 39]
> [backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor
> process SimpleTransportPlugin.java 158]
> [org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer invoke
> AbstractNonblockingServer.java 518]
> [org.apache.thrift7.server.Invocation run Invocation.java 18]
> [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java
> 1142]
> [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java
> 617]
> [java.lang.Thread run Thread.java 745]]}
> 2015-12-05 07:07:21 o.a.t.s.AbstractNonblockingServer$FrameBuffer [ERROR]
> Unexpected throwable while invoking!
> java.lang.AbstractMethodError:
> org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__3956.delete_node_blobstore(Ljava/lang/String;Ljava/lang/String;)V
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at
> backtype.storm.cluster$mk_storm_cluster_state$reify__3846.setup_blobstore_BANG_(cluster.clj:345)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:467)
> at
> backtype.storm.daemon.nimbus$fn__7774$exec_fn__2579__auto__$reify__7803.submitTopologyWithOpts(nimbus.clj:1523)
> at
> backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2940)
> at
> backtype.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2924)
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> backtype.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> at
> org.apache.thrift7.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> at org.apache.thrift7.server.Invocation.run(Invocation.java:18)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
