Subject: [ANNOUNCE] Apache Storm 2.6.4 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.6.4 Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the project

Re: Is Pacemaker deprecated?

I don't know details about the plan to deprecate Pacemaker, besides the link you have provided. Perhaps @Richard Zowallais able to shed some light on this. There are no active PRs to remove Pacemaker AFAIK. Storm still defaults to storm.cluster.state.store: "org.apache.storm.cluster.ZKStateStor

Re: Solving load distribution among the executors.

Murmur hash is an option. I believe it was used on Storm's v1 codebase On Sat, Aug 17, 2024, 13:57 Karthick wrote: > Thanks Sahil, Aaron for the reply > > Are you encountering a performance problem? > > Yes facing slowness on the executors with more load. > > Is the fields grouping absolutely re

Subject: [ANNOUNCE] Apache Storm 2.6.3 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.6.3 Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the project

Re: Nimbus HA setup not working

Hi Sahil, Which Storm version are you using? Do you have logs for Nimbus, Supervisors and Workers? If so, can you post the errors? Some extra documentation: https://storm.apache.org/releases/1.2.3/Daemon-Fault-Tolerance.html https://storm.apache.org/releases/1.2.3/nimbus-ha-design.html On Wed,

Re: The group id does not exist.

ote: > is not present. > > ~  kafka-consumer-groups.sh --command-config > ~/.kafka/kafka.production.properties --bootstrap-server > xxx.confluent.cloud:9092 --describe --grou Consumer group 'zzz' has no > active members. > > > On Mon, Mar 25, 2024 at 11:25 AM Rui

Re: The group id does not exist.

Can you query your Kafka broker using Kafka's binaries and check if the consumer group is present? (something like ./kafka-consumer-groups.sh --bootstrap-server $broker:9092 --describe --consumer-group $consumerGroup) And also check the broker side logs? On Mon, 25 Mar 2024 at 16:19, Juan Pablo

Re: STORM-4021: Add worker hooks via storm.yaml configuration file

I'm not familiar use the use of Hooks, but looking at the documentation and the Javadoc description , if you wanted to define hooks in storm.yaml you w

Re: FIPS compliance for Storm 2.4.0

Which FIPS modules are you referring to? To my knowledge, Storm (and a vast array of other Apache Foundation projects) aren't FIPS compliant. FIPS compliance entails a rigorous review process by NIST (OpenSSL is FIPS compliant because it deals with cryptography). Please correct me if I'm wrong. O

Re: Running Storm with Java 21 runtime... works !

Great to hear, Alexandre! Cheers On Wed, 29 Nov 2023 at 20:07, Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote: > Hello, > > Just for the Storm's users community information: since today this > morning (CET), I am running my pre-production Storm 2.6.0 cluster (10+ > topologies relying o

Re: Logging to stdout

It seems you have strict needs about the logging system. I reckon Storm will offer you the same guarantees as any other framework/library that uses Log4J. On Thu, 7 Sept 2023 at 16:46, Étienne Miret wrote: > Thanks, > > The issue I’m worried about is a race condition where: >1. Log4j writes

Re: Logging to stdout

I reckon that is very unlikely to occur. The RollingFileAppender by default flushes immediately the records to disk (immediateFlush) (But, of course, please have a read https://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender Cheers On Thu, 7 Sept 2023 at 15:45, Étienne Mire

Re: Logging to stdout

Hi Étienne, > Aside from the fact that collecting files is > harder than stdout, it create issues regarding file rotation. Also, I > need to know the actual file names to monitor. File rotation is entirely orchestrated by the Log4J framework. You just need to define a RollingFile appender with

Re: [DISCUSS] Call for PMC Members and Contributors

Hi! I'm open to contribute to the project as well. Cheers On Wed, 1 Mar 2023 at 05:09, P. Taylor Goetz wrote: > Thank you both for your willingness to help the project. > > Stay tuned. > > -Taylor > > > On Feb 25, 2023, at 10:15 AM, Alexandre Vermeerbergen < > avermeerber...@gmail.com> wrote: >

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

_BYTES" value. > client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES" > to 4MB. > > So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES" > like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value). > > Th

Re: Why can not change THRIFT_CHUNK_SIZE_BYTES in StormSubmitter?

How big the uber jar that you are submitting? Have you tried tweaking the Nimbus side property? https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67 On Mon, 6 Feb 2023 at 04:14, Minwoo Kang wrote: > Hello, Users. > > When I submit storm topology, Occasionally it takes a long time.

Re: [DISCUSSION] Apache Storm and moving to the Attic

Storm is a very successful project, even before the major re-write that happened on version 2.x. The community has been slowing down over the last couple of years, but perhaps that is a reflection of the maturity level of the project, where there are no major bugs or feature requests. So I like to

Re: [EXTERNAL] Request #665141: How would you rate the support you received?

Hi Ilya. You are sending questions through a 3rd party to Storm's mailing list, which is public and whose members are volunteers. You are using a quite old version of Storm, so you should upgrade if possible Storm 0.9 seems to had already built in guaranteed message processing through the use of a

Re: Notify spout thread while sleep in case of empty nextTuple

Can you elaborate on the sleep/backoff strategy part? On Fri, Jul 29, 2022, 06:31 Mohit Goyal wrote: > Hi Rui, > > I'm using storm 1.2.1. I've written my spouts on top of BaseRichSpout. > Passing default values. > > *Thanks & Regards* > *Mohit Goyal* > &

Re: Notify spout thread while sleep in case of empty nextTuple

Hi Mohit, Which version of Storm are you using? What are the values you are passing to the Spout on KafkaSpoutConfig ? On Thu, 21 Jul 2022 at 08:17, Mohit Goyal wrote: > Hi All, > > We are using apache storm to read msgs from kafka queues. Traffic usually > is high in these kafka queues so proc

Re: [External] Re: Storm 2 Topology Stopped Processing

so, to be clear, I didn’t think there was a way to disable backpressure. > You can only extend the backpressure check time, which is what I did. > > > > *From:* Rui Abreu > *Sent:* Wednesday, April 20, 2022 12:13 PM > *To:* user@storm.apache.org > *Subject:* Re: [Externa

Re: [External] Re: Storm 2 Topology Stopped Processing

As Bipin said, it could be a number of things. Have you tried disabling the backpressure mechanism and limiting the in flight messages with topology.max.spout.pending ? https://storm.apache.org/releases/current/Performance.html On Wed, 20 Apr 2022 at 17:04, Le, Binh T. wrote: > Hi Bipin, > > Th

Re: Shared cache between Cluster nodes

Difficult to say without knowing many details, but Storm + external Redis cluster is a solid design for that. You can also think of something along the lines of Hazelcast distributed cache, but Redis should be fast enough. On Wed, 16 Feb 2022 at 13:55, Nadav Glickman wrote: > Hello, > > I would

Re: How to address log4j vulnerability in Storm 1.2.2

Hi Carmen, I haven't tested that solution myself yet, but in theory it could work. You can try that in a dev machine. Try to replace the log4j JARs shipped with your version of storm with the latest one and try to spot any runtime errors. On Tue, Jan 11, 2022, 23:28 Carmen Molatch wrote: > Hell

Re: How to address log4j vulnerability in Storm 1.2.2

You can follow the instructions from the Apache Log4j project and just remove the JndiLookup class from the classpath: - zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class On Tue, 11 Jan 2022 at 20:42, Carm

Re: Mitigating CVE-2021-44228

Thanks for the heads up Keith. I can see Log4j's documentation has been updated to reflect that: https://logging.apache.org/log4j/2.x/security.html On Sat, 11 Dec 2021 at 18:37, Keith Bourgoin wrote: > Hey folks, > > I imagine a lot of people are dealing with log4Shell right now. I wanted > to

Re: Old state crashing nimbus? (v2.2.0)

Hi Andrew, Not sure how much this helps, but in version 1.x, state was on the following znodes: /$storm-znode/storms /$storm-znode/assignments /$storm-znode/blobstore Deleting all references (with rm or deleteall, depending on Zookeeper's version), followed by a Nimbus's rolling restart should

Re: Leader address when behind external DNS

Hi Tom, I recall using a docker container on my machine that was running Storm nimbus and Storm supervisors and through a port mapping between the container and my machine, I was able to upload topologies to the cluster through localhost:6627 What kind of client operations are you referring to? O

Re: Memory setting for each Storm topology

Hello Vitaly, There was a similar question a few days ago. It seems setting Config.TOPOLOGY_WORKER_CHILDOPTS in the topology configuration solves the problem. Thanks Rui, but my idea is to change it on a per topology basis. The > storm.yaml would mean changing it globally. > However, after quite

Re: Problem moving topology from 1.2.3 to 2.2.0 - tuple distribution across cluster

Hi Thomas! If you believe it's a bug, please submit it to https://issues.apache.org/jira/browse/STORM, so hopefully someone will be able to have a look. On Mon, 16 Nov 2020 at 15:16, Thomas L. Redman wrote: > I think somebody on the Dev team needs to look into this. The topology I > wrote is on

Re: Passing worker JVM options not working as expected

re testing I have figured that using > > conf.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-XX:+UseG1GC"); > > does the trick :) > > Am Mi., 18. Nov. 2020 um 18:42 Uhr schrieb Rui Abreu >: > >> Hello Jonas, >> >> It should be using the *worker.childopts *d

Re: Passing worker JVM options not working as expected

Hello Jonas, It should be using the *worker.childopts *defined in storm/conf/storm.yaml. On Wed, 18 Nov 2020 at 16:04, Jonas Krauss wrote: > Hi all, > > we are running Storm 2.2.0 on openjdk build > 1.8.0_272-8u272-b10-0+deb9u1-b10 in a cluster with five supervisors. We are > trying to set indi

Re: Nodes underutilized

e spout's parallelism. But it depends on how many Kafka partitions your input topic has. On Tue, 15 Sep 2020 at 18:39, Thomas L. Redman wrote: > 2.2.0, I just upgraded not long ago. > > On Sep 14, 2020, at 9:28 AM, Rui Abreu wrote: > > Hi Thomas, > > Which version of Storm are

Re: Scheme in Spouts

t; -- > *From:* Paul Jose > *Sent:* 25 September 2020 13:20 > *To:* user@storm.apache.org > *Subject:* Re: Scheme in Spouts > > Hi Rui, > > Thank you so much. Don't know how I overlooked that. This works well. > Thanks again. > > Be

Re: Scheme in Spouts

Hi Paul, The class still exists under a new package: https://storm.apache.org/releases/current/javadocs/org/apache/storm/spout/SchemeAsMultiScheme.html On Thu, 24 Sep 2020 at 12:14, Paul Jose wrote: > > > Hi, > > My name is Paul. I am tasked with upgrading a storm topology running on > the 0.1

Re: Nodes underutilized

Hi Thomas, Which version of Storm are you using? On Sun, 13 Sep 2020 at 20:23, Thomas L. Redman wrote: > Sorry, I had previously sent this from a different email address, not sure > how well that would work with this service, hence this re-send. > > I’m running storm on a 3 node cluster, 32 phy

Re: Avoiding bottleneck in fields grouping

If maintaining the order of the messages is a requirement, fields grouping seems to be the only strategy that ensures that all tuples of the same partition will be sent to the same task ID. https://storm.apache.org/releases/current/Concepts.html Stream groupings > Part of defining a topology i

Re: where is nextCommitOffset stored?

Old Kafka consumers stored the commit offset in Zookeeper. Which version are you using? On Tue, 21 Jul 2020 at 10:54, Rajat Gangwar wrote: > Hi, > > I am seeing following error in Storm logs: > > java.lang.IllegalStateException: The offset [2] is below the current > *nextCommitOffset* [*5270353*

Re: Updating log4j2 version for Storm topology

Hi Mitchel, As Sunil mentioned, Storm installation brings a set of libs under its lib folder. Example for a 1.1.3 installation: asm-5.0.3.jar disruptor-3.3.2.jar log4j-api-2.8.2.jar log4j-over-slf4j-1.6.6.jar minlog-1.3.0.jar reflectasm-1.10.1.jar servlet-api-2.5.jar storm-core-1.1.3.

Re: Backpressure with fieldsgrouping

Hi Abhishek, Have a look at KafkaSpoutConfig (and KafkaSpoutConfig.Builder), particularly the setMaxUncommittedOffsets and ConsumerConfig.MAX_POLL_RECORDS_CONFIG (kConfigBuilder.setProp(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, i))) https://storm.apache.org/releases/current/javadocs/org/apache/sto

Re: machine reboot

Hi Zainal, I guess some assumptions above about Storm continues to work is wrong, what > is the best practice or recommended way for our situation? I will need more details. What stops working? How many workers are you running per supervisor and how many workers is your topology configured to us

Re: machine reboot

The rebalance command through Storm UI does not guarantee you that tasks will go to another worker / machine. There is a command line rebalance command but it's used to increase the number of workers or executors: $ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10 Without tryin

Re: machine reboot

As long the workers and tasks in machine A are healthy and sending hearbeats to Nimbus, they will keep running there. A redeployment of the topologies or a a rebalance command (you can use Storm UI for this), may send tasks to be executed in machine B. On Fri, Feb 21, 2020, 22:14 Zainal Arifin (BL

Re: Reg: Store/Retrieve State across supervisor

Hello, Redis is a good option for your use case. You can also have a look at https://storm.apache.org/releases/1.2.3/distcache-blobstore.html, but it might not suit your needs (I've never used it). You could in theory use Zookeeper through something like Curator, but Zookeeper is meant for servic

Re: question on deactivate and kill

To change the number of executors to a particular bolt, you can execute this on a Nimbus: storm rebalance exclamation-topology -e even-digit-bolt=3 But again, this will have a practical effect if you have spare tasks that those executors might work on. On Wed, 15 Jan 2020 at 17:32, Zainal Arifin (

Re: question on deactivate and kill

Deactivating a topology means the spouts stop consuming, but everything else continues to work as expected, meaning your bolts will process all in flight tuples. You can change the number of workers and executors in runtime. In order to scale up the latter, in you need to have previously submitted

Re: How to upgrade from 1.x to 2.x ?

topology in old cluster and deploy to new cluster > - kill old cluster > > I know that doesn't directly answer your question, but at least for us we > had no expectation that the topologies would remain compatible in a major > version update. > > Good luck! > Andrew &g

How to upgrade from 1.x to 2.x ?

Is there a backwards compatible way to perform a 1.x to 2.x Storm upgrade? I've conducted a few tests and a cluster upgrade from 1.x to 2.x makes topologies compiled to 1.X stop working and uploads to Nimbus are not possible. Thank you in advance

Storm UI "Used Mem"

Hello, I've noticed that Storm UI "Used Mem (MB)" column is always 1088 or 832 for any topology, regardless of how much RAM is being used by the workers in the machines [image: Inline images 1] Is this behaviour expected at all? I'm running Storm 1.1.0 Thanks

Re: Re: open storm ui very slow(Internet mail)

n 25 June 2017 at 20:42, Erik Weathers wrote: > 300 is not that many supervisors from my perspective. Have any of you > experiencing this issue dug in to see what's slowing it down? > > - Erik > > On Fri, Jun 23, 2017 at 1:48 AM Rui Abreu wrote: > >> >> On

Re: Re: open storm ui very slow(Internet mail)

On 22 June 2017 at 19:35, Erik Weathers wrote: > Sounds like you've hit some scaling bottleneck with Storm. I've never > tried running nearly that number of topologies. > It might be an inefficiency with the number of API calls, or with the > interactions between the Nimbus and ZooKeeper. > A