[jira] [Resolved] (STORM-3197) Make StormMetricsRegistry a regular instance class rather than a static utility
[ https://issues.apache.org/jira/browse/STORM-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-3197. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Srdo], I merged this into master. > Make StormMetricsRegistry a regular instance class rather than a static > utility > --- > > Key: STORM-3197 > URL: https://issues.apache.org/jira/browse/STORM-3197 > Project: Apache Storm > Issue Type: Task > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Having the registry be a static utility makes fixing some issues harder than > it should be. Preventing daemons from accidentally registering metrics for > other daemons (STORM-3101) and flushing reporters on shutdown without > breaking the tests (STORM-3173) for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-3228) Supervisor blobstore ref counting not working properly
[
https://issues.apache.org/jira/browse/STORM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated STORM-3228:
--
Labels: pull-request-available (was: )
> Supervisor blobstore ref counting not working properly
> --
>
> Key: STORM-3228
> URL: https://issues.apache.org/jira/browse/STORM-3228
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aaron Gresch
>Assignee: Aaron Gresch
>Priority: Major
> Labels: pull-request-available
>
> When supervisors release slots, they can continue to indefinitely download
> the blob.
> {code:java}
> 2018-09-17 16:18:00.955 o.a.s.l.AsyncLocalizer SLOT_6703 [INFO] Releasing
> slot for logviewer-ui-groups-test-11-1537201071 6703
> 2018-09-17 16:18:00.956 o.a.s.l.LocallyCachedBlob SLOT_6703 [WARN]
> {logviewer-ui-groups-test-11-1537201071 on 6703} had no reservation for
> logviewer-ui-groups-test-11-1537201071 stormjar.jar
> 2018-09-17 16:22:50.198 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor - 2
> [WARN] Failed to download blob LOCAL TOPO BLOB TOPO_CONF
> logviewer-ui-groups-test-11-1537201071 will try again in 100 ms{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3205) Optimization in TuplImpl
[
https://issues.apache.org/jira/browse/STORM-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-3205.
Resolution: Fixed
Thanks [~roshan_naik],
I merged this to master.
> Optimization in TuplImpl
>
>
> Key: STORM-3205
> URL: https://issues.apache.org/jira/browse/STORM-3205
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
>Affects Versions: 2.0.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Wrapping {{TuplImpl.values}} with Collections.unmodifiableList() turns out be
> very expensive. Its intention is obviously to check and prevent accidental
> tweaking TuplImpl once created. Given the high cost, if needed, we can limit
> this extra checking mechanism in debug/dev mode. Being in the critical path
> it means several thousand/million additional allocations per second of the
> List wrapper object proportional to the number of bolt/spout instances.
> *TVL :*
> | |throughput (k/sec)|cores|mem (mb)|
> |*master (#f5a410ba3)*|412 |4.26|103|
> |*storm-3205*|547 (+33%)|4.09|132|
> {{+cmd:+ bin/storm jar topos/storm-loadgen-2.0.0-SNAPSHOT.jar
> org.apache.storm.loadgen.ThroughputVsLatency *--rate 55* --spouts 1
> --splitters 3 --counters 2 -c topology.acker.executors=0}}
> *+ConstSpoutIdentityBoltNullBolt :+*
> | |throughput|
> |*master*|4.25 mill/sec|
> |*storm-3205*|5.4 mill/sec (+27%)|
> +cmd+: {{bin/storm jar topos/storm-perf-2.0.0-SNAPSHOT.jar
> org.apache.storm.perf.ConstSpoutIdBoltNullBoltTopo -c
> topology.acker.executors=0 -c topology.producer.batch.size=1000 400}}
> *Note:* The perf gains are more evident when operating at high thoughputs w/o
> backpressure occurring (i.e. some bolts have not yet become a bottleneck)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-3227) Improve security of credentials push
[ https://issues.apache.org/jira/browse/STORM-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-3227: -- Labels: pull-request-available (was: ) > Improve security of credentials push > > > Key: STORM-3227 > URL: https://issues.apache.org/jira/browse/STORM-3227 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client, storm-server >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Major > Labels: pull-request-available > > When pushing credentials to a topology most of the checks we do right now are > to verify that the topology is allowing a given user to do the push, but we > also need to protect the user from pushing to the wrong topology. > > This is really only an issue if a user has the push setup on some kind of a > cron like job, and the topology is down (which should be rare), but to > eliminate any race conditions we should have nimbus either verify that the > topology is owned by the same user as the one doing the push, or have an > optional user that the client expects the topology to be owned by. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-3228) Supervisor blobstore ref counting not working properly
[
https://issues.apache.org/jira/browse/STORM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617867#comment-16617867
]
Aaron Gresch commented on STORM-3228:
-
There are two implementations of the PortAndAssignment interface. The two
implementations are not considered equal in the reference count map.
> Supervisor blobstore ref counting not working properly
> --
>
> Key: STORM-3228
> URL: https://issues.apache.org/jira/browse/STORM-3228
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Aaron Gresch
>Assignee: Aaron Gresch
>Priority: Major
>
> When supervisors release slots, they can continue to indefinitely download
> the blob.
> {code:java}
> 2018-09-17 16:18:00.955 o.a.s.l.AsyncLocalizer SLOT_6703 [INFO] Releasing
> slot for logviewer-ui-groups-test-11-1537201071 6703
> 2018-09-17 16:18:00.956 o.a.s.l.LocallyCachedBlob SLOT_6703 [WARN]
> {logviewer-ui-groups-test-11-1537201071 on 6703} had no reservation for
> logviewer-ui-groups-test-11-1537201071 stormjar.jar
> 2018-09-17 16:22:50.198 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor - 2
> [WARN] Failed to download blob LOCAL TOPO BLOB TOPO_CONF
> logviewer-ui-groups-test-11-1537201071 will try again in 100 ms{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-3228) Supervisor blobstore ref counting not working properly
Aaron Gresch created STORM-3228:
---
Summary: Supervisor blobstore ref counting not working properly
Key: STORM-3228
URL: https://issues.apache.org/jira/browse/STORM-3228
Project: Apache Storm
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Aaron Gresch
Assignee: Aaron Gresch
When supervisors release slots, they can continue to indefinitely download the
blob.
{code:java}
2018-09-17 16:18:00.955 o.a.s.l.AsyncLocalizer SLOT_6703 [INFO] Releasing slot
for logviewer-ui-groups-test-11-1537201071 6703
2018-09-17 16:18:00.956 o.a.s.l.LocallyCachedBlob SLOT_6703 [WARN]
{logviewer-ui-groups-test-11-1537201071 on 6703} had no reservation for
logviewer-ui-groups-test-11-1537201071 stormjar.jar
2018-09-17 16:22:50.198 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor - 2
[WARN] Failed to download blob LOCAL TOPO BLOB TOPO_CONF
logviewer-ui-groups-test-11-1537201071 will try again in 100 ms{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-3225) AuthorizedUserFilter should not convert the media type to a string
[ https://issues.apache.org/jira/browse/STORM-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-3225. Resolution: Fixed Fix Version/s: 2.0.0 > AuthorizedUserFilter should not convert the media type to a string > -- > > Key: STORM-3225 > URL: https://issues.apache.org/jira/browse/STORM-3225 > Project: Apache Storm > Issue Type: Bug > Components: storm-webapp >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The AuthorizedUserFilter is converting the media type to a string so it can > do a comparison check. There is a better way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-3226) Improve Supervisor authorization handler error message
[ https://issues.apache.org/jira/browse/STORM-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-3226. Resolution: Fixed Fix Version/s: 2.0.0 > Improve Supervisor authorization handler error message > -- > > Key: STORM-3226 > URL: https://issues.apache.org/jira/browse/STORM-3226 > Project: Apache Storm > Issue Type: Improvement > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The error in the supervisor about the authorization handler is confusing and > does not explain how to fix the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-3227) Improve security of credentials push
Robert Joseph Evans created STORM-3227: -- Summary: Improve security of credentials push Key: STORM-3227 URL: https://issues.apache.org/jira/browse/STORM-3227 Project: Apache Storm Issue Type: Improvement Components: storm-client, storm-server Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans When pushing credentials to a topology most of the checks we do right now are to verify that the topology is allowing a given user to do the push, but we also need to protect the user from pushing to the wrong topology. This is really only an issue if a user has the push setup on some kind of a cron like job, and the topology is down (which should be rare), but to eliminate any race conditions we should have nimbus either verify that the topology is owned by the same user as the one doing the push, or have an optional user that the client expects the topology to be owned by. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2138) java.io.FileNotFoundException: stormconf.ser does not exist
[
https://issues.apache.org/jira/browse/STORM-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617296#comment-16617296
]
michel hummel commented on STORM-2138:
--
This issue still exists in V1.2.2.
I think there is a race condition when a topology is stopping:
the supervisor is removing the local files:
{noformat}
2018-09-14 21:33:03.718 o.a.s.d.s.Container [INFO] Cleaning up
5485e983-34a9-46ad-8679-bbe79c110cba:32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.718 o.a.s.d.s.Container [INFO] GET worker-user for
32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.718 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers/32173038-42d8-40c7-94ce-9229e3177b6e/pids/69074
2018-09-14 21:33:03.718 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers/32173038-42d8-40c7-94ce-9229e3177b6e/heartbeats
2018-09-14 21:33:03.730 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers/32173038-42d8-40c7-94ce-9229e3177b6e/pids
2018-09-14 21:33:03.731 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers/32173038-42d8-40c7-94ce-9229e3177b6e/tmp
2018-09-14 21:33:03.731 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers/32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.732 o.a.s.d.s.Container [INFO] REMOVE worker-user
32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.732 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/workers-users/32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.734 o.a.s.d.s.BasicContainer [INFO] Removed Worker ID
32173038-42d8-40c7-94ce-9229e3177b6e
2018-09-14 21:33:03.734 o.a.s.l.AsyncLocalizer [INFO] Released blob reference
ipc-fci-nrt_Tsit-RCFD_2017100_0018-09-14T21-22-51-429Z-100-1536960172 6702
Cleaning up BLOB references...
2018-09-14 21:33:03.735 o.a.s.l.AsyncLocalizer [INFO] Released blob reference
ipc-fci-nrt_Tsit-RCFD_2017100_0018-09-14T21-22-51-429Z-100-1536960172 6702
Cleaning up basic files...
2018-09-14 21:33:03.735 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
/space/StormData/storm-data/supervisor/stormdist/ipc-fci-nrt_Tsit-RCFD_2017100_0018-09-14T21-22-51-429Z-100-1536960172
{noformat}
and if the blobUpdateTimer is trigged at the same time
[https://github.com/apache/storm/blob/v1.2.2/storm-core/src/jvm/org/apache/storm/daemon/supervisor/timer/UpdateBlobs.java]
it will search for the topology configuration which doesn't exists anymore:
{noformat}
2018-09-14 21:33:03.796 o.a.s.e.EventManagerImp [ERROR] {} Error when
processing event
java.lang.RuntimeException: java.io.FileNotFoundException: File
'/space/StormData/storm-data/supervisor/stormdist/ipc-fci-nrt_Tsit-RCFD_2017100_0018-09-14T21-22-51-429Z-100-1536960172/stormconf.ser'
does not exist
at org.apache.storm.utils.Utils.wrapInRuntime(Utils.java:1571)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.daemon.supervisor.timer.UpdateBlobs.run(UpdateBlobs.java:86)
~[storm-core-1.2.2.jar:1.2.2]
at org.apache.storm.event.EventManagerImp$1.run(EventManagerImp.java:54)
[storm-core-1.2.2.jar:1.2.2]
Caused by: java.io.FileNotFoundException: File
'/space/StormData/storm-data/supervisor/stormdist/ipc-fci-nrt_Tsit-RCFD_2017100_0018-09-14T21-22-51-429Z-100-1536960172/stormconf.ser'
does not exist
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:292)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1815)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:262)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:374)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:368)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.daemon.supervisor.timer.UpdateBlobs.updateBlobsForTopology(UpdateBlobs.java:100)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.daemon.supervisor.timer.UpdateBlobs.run(UpdateBlobs.java:76)
~[storm-core-1.2.2.jar:1.2.2]
... 1 more
{noformat}
I think that dealing with this race condition is hard without doing a lock
may-be adding a catch on FileNotFoundException could be an acceptable fix ?
> java.io.FileNotFoundException: stormconf.ser does not exist
> ---
>
> Key: STORM-2138
> URL: https://issues.apache.org/jira/browse/STORM-2138
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.2
>Reporter: Eddy
>Priority: Major
>
> We are seeing problems in our storm topology whereby all our workers crash.
> The errors we see are
>
