Yes slowness is my concern, my consumer process in the spout will be stopped and start on after submitting the topology. So it is treated as a downtime.
On Sat, Sep 21, 2024 at 9:33 PM Rui Abreu <rui.ab...@gmail.com> wrote: > > > > If I add all dependencies in the topology jar it's size is too large > around > > 350MB. > > > > Do you get an error when uploading the uber jar? Or is it just slow? > > I believe the most practical process is indeed the submission of all > dependencies on a single bundle. That's how I've always been using Storm. > > > On Sat, 21 Sept 2024 at 06:41, Karthick <ibmkarthickma...@gmail.com> > wrote: > > > Thanks for the reply Aaron, Rui > > > > Why do you need to update dependencies on external-lib on both Nimbus > > > and Supervisor? > > > > > > Like we may use jackson, kafka client, apache commons, other internal > > frameworks jars and other third parties jars, so we do have those in the > > external-lib, on the jar version changes we do change those jars in > > external-lib. > > > > > > Are you not submitting an uber-jar with all your dependencies plus > > > your topology? > > > > If I add all dependencies in the topology jar it's size is too large > around > > 350MB. > > > > > > Announce deployment well ahead of time - allowing users to plan switches > to > > > other clusters if desired. > > > > > > Aaron, i'm asking for the topology code changes and internal jar version > > changes update. Not for upgrading the Storm version, sorry for not > setting > > proper questions here. > > > > On Fri, Sep 20, 2024 at 8:57 PM Rui Abreu <rui.ab...@gmail.com> wrote: > > > > > Hello Karthick, > > > Why do you need to update dependencies on external-lib on both Nimbus > > > and Supervisor? > > > Are you not submitting an uber-jar with all your dependencies plus > > > your topology? > > > > > > > > > > > > On Fri, 20 Sept 2024 at 08:10, Karthick <ibmkarthickma...@gmail.com> > > > wrote: > > > > > > > > Dear Apache Storm Community, > > > > > > > > I am currently managing an Apache Storm cluster with 38 nodes: 3 > > > dedicated > > > > to ZooKeeper, 1 to Nimbus and the UI, and 34 nodes running Supervisor > > and > > > > Logviewer processes. Each node has 2 Workers. > > > > > > > > At present, our topology update process involves the following steps: > > > > > > > > 1. Killing the existing topology. > > > > 2. Changing dependency JARs under the external-lib dir and > > restarting > > > > Nimbus. > > > > 3. Changing dependency JARs under the external-lib dir and > > restarting > > > > Supervisors. > > > > 4. Submitting the new topology. > > > > > > > > Each operation takes about 2–3 minutes. As the number of Supervisor > > nodes > > > > increases, the overall time for topology updates is becoming a > concern. > > > > > > > > I am reaching out to seek advice on how to optimize this process, as > I > > > > believe there are more efficient ways to handle topology updates in > > > > large-scale Storm deployments. Specifically: > > > > > > > > - Is there a more efficient process to handle code changes without > > > > having to manually restart Nimbus and Supervisors? > > > > - How can I reduce the overall time for topology updates, > especially > > > as > > > > our cluster continues to grow? > > > > - Are there industry-standard practices for implementing rolling > > > updates > > > > or automating the deployment process? > > > > > > > > Any insights, recommendations, or best practices that could help > > > streamline > > > > our update process would be greatly appreciated. > > > > > > > > Thank you for your time, and I look forward to your suggestions! > > > > > >
