Thanks Rui for timely help. On Fri, Sep 27, 2024 at 4:47 PM Rui Abreu <rui.ab...@gmail.com> wrote:
> new SubmitOptions(TopologyInitialStatus.INACTIVE); > > Those options can be passed to StormSubmitter > (StormSubmitter.submitTopologyWithProgressBar for example) > > On Fri, 27 Sept 2024 at 12:11, Karthick <ibmkarthickma...@gmail.com> > wrote: > > > Hi Rui, > > > > How to submit topology with inactive mode? Does the storm support this? > > Please advise > > > > On Sun, Sep 22, 2024 at 9:42 PM Karthick <ibmkarthickma...@gmail.com> > > wrote: > > > > > Thanks Rui for providing suggestions, I will try out and come back to > > you. > > > > > > On Sun, Sep 22, 2024 at 3:55 PM Rui Abreu <rui.ab...@gmail.com> wrote: > > > > > >> You can try to speed up the upload process by tweaking > > >> https://github.com/apache/storm/blob/master/conf%2Fdefaults.yaml#L66 > > >> > > >> But regardless of the speed of the upload, in your upload class there > > are > > >> some steps you can take: > > >> - submit the new topology alongside the old one (with a different > > version > > >> on the name) > > >> - submit it as inactive > > >> - using the nimbus client, wait for the new topology to have workers > > >> assigned, which means it's ready to be activated > > >> - deactivate the old topology > > >> - activate the new one > > >> - kill the old topology > > >> > > >> Using this strategy, you are looking at a few seconds worth of > > >> interruption > > >> of the flow. > > >> > > >> On Sun, Sep 22, 2024, 08:21 Karthick <ibmkarthickma...@gmail.com> > > wrote: > > >> > > >> > Yes slowness is my concern, my consumer process in the spout will be > > >> > stopped and start on after submitting the topology. So it is treated > > as > > >> a > > >> > downtime. > > >> > > > >> > On Sat, Sep 21, 2024 at 9:33 PM Rui Abreu <rui.ab...@gmail.com> > > wrote: > > >> > > > >> > > > > > >> > > > If I add all dependencies in the topology jar it's size is too > > large > > >> > > around > > >> > > > 350MB. > > >> > > > > > >> > > > > >> > > Do you get an error when uploading the uber jar? Or is it just > slow? > > >> > > > > >> > > I believe the most practical process is indeed the submission of > all > > >> > > dependencies on a single bundle. That's how I've always been using > > >> Storm. > > >> > > > > >> > > > > >> > > On Sat, 21 Sept 2024 at 06:41, Karthick < > ibmkarthickma...@gmail.com > > > > > >> > > wrote: > > >> > > > > >> > > > Thanks for the reply Aaron, Rui > > >> > > > > > >> > > > Why do you need to update dependencies on external-lib on both > > >> Nimbus > > >> > > > > and Supervisor? > > >> > > > > > >> > > > > > >> > > > Like we may use jackson, kafka client, apache commons, other > > >> internal > > >> > > > frameworks jars and other third parties jars, so we do have > those > > in > > >> > the > > >> > > > external-lib, on the jar version changes we do change those jars > > in > > >> > > > external-lib. > > >> > > > > > >> > > > > > >> > > > Are you not submitting an uber-jar with all your dependencies > plus > > >> > > > > your topology? > > >> > > > > > >> > > > If I add all dependencies in the topology jar it's size is too > > large > > >> > > around > > >> > > > 350MB. > > >> > > > > > >> > > > > > >> > > > Announce deployment well ahead of time - allowing users to plan > > >> > switches > > >> > > to > > >> > > > > other clusters if desired. > > >> > > > > > >> > > > > > >> > > > Aaron, i'm asking for the topology code changes and internal > jar > > >> > version > > >> > > > changes update. Not for upgrading the Storm version, sorry for > not > > >> > > setting > > >> > > > proper questions here. > > >> > > > > > >> > > > On Fri, Sep 20, 2024 at 8:57 PM Rui Abreu <rui.ab...@gmail.com> > > >> wrote: > > >> > > > > > >> > > > > Hello Karthick, > > >> > > > > Why do you need to update dependencies on external-lib on both > > >> Nimbus > > >> > > > > and Supervisor? > > >> > > > > Are you not submitting an uber-jar with all your dependencies > > plus > > >> > > > > your topology? > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On Fri, 20 Sept 2024 at 08:10, Karthick < > > >> ibmkarthickma...@gmail.com> > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > Dear Apache Storm Community, > > >> > > > > > > > >> > > > > > I am currently managing an Apache Storm cluster with 38 > > nodes: 3 > > >> > > > > dedicated > > >> > > > > > to ZooKeeper, 1 to Nimbus and the UI, and 34 nodes running > > >> > Supervisor > > >> > > > and > > >> > > > > > Logviewer processes. Each node has 2 Workers. > > >> > > > > > > > >> > > > > > At present, our topology update process involves the > following > > >> > steps: > > >> > > > > > > > >> > > > > > 1. Killing the existing topology. > > >> > > > > > 2. Changing dependency JARs under the external-lib dir > and > > >> > > > restarting > > >> > > > > > Nimbus. > > >> > > > > > 3. Changing dependency JARs under the external-lib dir > and > > >> > > > restarting > > >> > > > > > Supervisors. > > >> > > > > > 4. Submitting the new topology. > > >> > > > > > > > >> > > > > > Each operation takes about 2–3 minutes. As the number of > > >> Supervisor > > >> > > > nodes > > >> > > > > > increases, the overall time for topology updates is > becoming a > > >> > > concern. > > >> > > > > > > > >> > > > > > I am reaching out to seek advice on how to optimize this > > >> process, > > >> > as > > >> > > I > > >> > > > > > believe there are more efficient ways to handle topology > > >> updates in > > >> > > > > > large-scale Storm deployments. Specifically: > > >> > > > > > > > >> > > > > > - Is there a more efficient process to handle code > changes > > >> > without > > >> > > > > > having to manually restart Nimbus and Supervisors? > > >> > > > > > - How can I reduce the overall time for topology updates, > > >> > > especially > > >> > > > > as > > >> > > > > > our cluster continues to grow? > > >> > > > > > - Are there industry-standard practices for implementing > > >> rolling > > >> > > > > updates > > >> > > > > > or automating the deployment process? > > >> > > > > > > > >> > > > > > Any insights, recommendations, or best practices that could > > help > > >> > > > > streamline > > >> > > > > > our update process would be greatly appreciated. > > >> > > > > > > > >> > > > > > Thank you for your time, and I look forward to your > > suggestions! > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
