I am going to prepare an RC1 for VOTE today.
Am Mittwoch, dem 15.11.2023 um 19:33 +0100 schrieb Alexandre Vermeerbergen: > Great ! > On a side note, my preproduction has been running with 2.6.0 build > 20231114 at scale since the last 10 hours, using my usual set: > RedHat Linux 8 > IBM Java Semeru 17.0.8.1 (aka OpenJ9) - 6 big machines for > Supervisors > 14 topologies (with one having 5 different spouts and 9 bolts, all > with lots of executors, using various field grouping stratefies) > + all topologies migrated at source level to use Jakarta EE APIs > (like jakarta.json instead of javax.son) > > => all runs smoothly with no regression. On Supervisor machines, CPU > is even a bit smoother (less peaks) than before this upgrade to 2.6.0 > built 2023/11/14. > > So from my standpoint, we should be in good shape to have a 2.6.0 > release ASAP ! > > Note: I wish I had time to test all this with Java 21, but to be > realistic I don't think it will happen before next month... > > Many thanks Richard for pushing us to try snapshot releases before > entering a vote cycle! > > Alexandre > > > Le mer. 15 nov. 2023 à 16:49, Richard Zowalla <r...@apache.org> a > écrit : > > > > Thx. The related PR is merged. > > > > Am Mittwoch, dem 15.11.2023 um 15:13 +0100 schrieb Richard Zowalla: > > > Yes, feel free to provide a fix / pr + jira issue. > > > > > > It is just a matter of adding > > > > > > <dependency> > > > <groupId>jakarta.xml.bind</groupId> > > > <artifactId>jakarta.xml.bind-api</artifactId> > > > <!-- version will be inherited --> > > > </dependency> > > > > > > to the related pom in the /external module [1] > > > > > > Gruß > > > Richard > > > > > > [1] > > > https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/pom.xml > > > > > > > > > Am Mittwoch, dem 15.11.2023 um 15:02 +0100 schrieb Alexandre > > > Vermeerbergen: > > > > I had found yet another thing that must be fixed before > > > > starting > > > > 2.6.0 > > > > vote: while reviewing my shells to install Storm on my Nimbus > > > > hosts > > > > and Supervision hosts, I found this old by-pass which I used to > > > > put > > > > in > > > > place when switching from Java 11 to Java 17 at runtime: > > > > > > > > cp storm-stable/lib-worker/jakarta.xml.bind-api-2.3.2.jar > > > > storm-stable/lib-tools/storm-kafka-monitor > > > > > > > > (storm-stable is the name of the directory where I extract > > > > Storm's > > > > binaries) > > > > > > > > It turns out that this ugly by-pass is still required when > > > > running > > > > with Java 17 (which is my case since more than 1 year), > > > > otherwise > > > > the > > > > display of Kafka lags in Storm UI is broken, and we have the > > > > following > > > > exceptions in ui.log: > > > > > > > > 2023-11-15 10:54:27.473 o.e.j.s.AbstractConnector main [INFO] > > > > Started > > > > ServerConnector@e3c9c02f{HTTP/1.1, (http/1.1)}{0.0.0.0:8070} > > > > 2023-11-15 10:54:27.474 o.e.j.s.Server main [INFO] Started > > > > @4256ms > > > > 2023-11-15 10:55:02.781 o.a.s.u.NimbusClient qtp-1075191296-25 > > > > [INFO] > > > > Found leader nimbus : > > > > ec23-1-251-0-65.eu-west-1.compute.amazonaws.com:6627 > > > > 2023-11-15 10:59:31.128 o.a.s.u.ShellUtils qtp-1075191296-24 > > > > [INFO] > > > > Failed running command > > > > [/usr/local/Storm/storm-stable/bin/storm-kafka-monitor, -t, > > > > audit, > > > > - > > > > g, > > > > StormAuditPublisherTopology_SbxRealTimeSupervisionAVEEEZZeuw1, > > > > -b, > > > > ec23-1-251-0-66.eu-west-1.compute.amazonaws.com,ec23-1-251-0- > > > > 65.eu- > > > > west-1.compute.amazonaws.com,ec2-52-18-172-77, > > > > -s, SASL_SSL, -c, /tmp/kafka-consumer- > > > > extra6421423768198678110props] > > > > org.apache.storm.utils.ShellUtils$ExitCodeException: SLF4J: > > > > Failed > > > > to > > > > load class "org.slf4j.impl.StaticLoggerBinder". > > > > SLF4J: Defaulting to no-operation (NOP) logger implementation > > > > SLF4J: See > > > > http://www.slf4j.org/codes.html#StaticLoggerBinder for > > > > further details. > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > > javax.xml.bind.DatatypeConverter > > > > at > > > > org.apache.kafka.common.security.scram.ScramMessages$ServerFirs > > > > tMes > > > > sa > > > > ge.<init>(ScramMessages.java:143) > > > > at > > > > org.apache.kafka.common.security.scram.ScramSaslClient.evaluate > > > > Chal > > > > le > > > > nge(ScramSaslClient.java:112) > > > > at > > > > org.apache.kafka.common.security.authenticator.SaslClientAuthen > > > > tica > > > > to > > > > r$2.run(SaslClientAuthenticator.java:280) > > > > at > > > > org.apache.kafka.common.security.authenticator.SaslClientAuthen > > > > tica > > > > to > > > > r$2.run(SaslClientAuthenticator.java:278) > > > > at > > > > java.base/java.security.AccessController.doPrivileged(AccessCon > > > > trol > > > > le > > > > r.java:784) > > > > at > > > > java.base/javax.security.auth.Subject.doAs(Subject.java:439) > > > > at > > > > org.apache.kafka.common.security.authenticator.SaslClientAuthen > > > > tica > > > > to > > > > r.createSaslToken(SaslClientAuthenticator.java:278) > > > > at > > > > org.apache.kafka.common.security.authenticator.SaslClientAuthen > > > > tica > > > > to > > > > r.sendSaslToken(SaslClientAuthenticator.java:215) > > > > at > > > > org.apache.kafka.common.security.authenticator.SaslClientAuthen > > > > tica > > > > to > > > > r.authenticate(SaslClientAuthenticator.java:189) > > > > at > > > > org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChann > > > > el.j > > > > av > > > > a:76) > > > > at > > > > org.apache.kafka.common.network.Selector.pollSelectionKeys(Sele > > > > ctor > > > > .j > > > > ava:385) > > > > at > > > > org.apache.kafka.common.network.Selector.poll(Selector.java:334 > > > > ) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java: > > > > 433) > > > > at > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClie > > > > nt.p > > > > ol > > > > l(ConsumerNetworkClient.java:232) > > > > at > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClie > > > > nt.p > > > > ol > > > > l(ConsumerNetworkClient.java:208) > > > > at > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClie > > > > nt.p > > > > ol > > > > l(ConsumerNetworkClient.java:184) > > > > at > > > > org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMet > > > > adat > > > > a( > > > > Fetcher.java:314) > > > > at > > > > org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(K > > > > afka > > > > Co > > > > nsumer.java:1386) > > > > at > > > > org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags > > > > (Kaf > > > > ka > > > > OffsetLagUtil.java:165) > > > > at > > > > org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOff > > > > setL > > > > ag > > > > Util.java:74) > > > > Caused by: java.lang.ClassNotFoundException: > > > > javax.xml.bind.DatatypeConverter > > > > at > > > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Buil > > > > tinC > > > > la > > > > ssLoader.java:827) > > > > at > > > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadC > > > > lass > > > > (C > > > > lassLoaders.java:188) > > > > at > > > > java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1095 > > > > ) > > > > ... 20 more > > > > > > > > at > > > > org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:26 > > > > 4) > > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT] > > > > > > > > copying jakarta.xml.bind-api-2.3.2.jar to > > > > storm-stable/lib-tools/storm-kafka-monitor and restarting > > > > storm-ui > > > > solves the problem. > > > > > > > > => Would it be possible to include jakarta.xml.bind-api- > > > > 2.3.2.jar > > > > in > > > > lib-tools/storm-kafka-monitor/ directory of binary Storm > > > > archive > > > > out > > > > of the box? > > > > > > > > Thanks, > > > > Alexandre > > > > > > > > Le mar. 14 nov. 2023 à 17:50, Alexandre Vermeerbergen > > > > <avermeerber...@gmail.com> a écrit : > > > > > > > > > > Good news on my side: > > > > > * First test with 2.6.0 snapshot 2023/11/14 with a single > > > > > Supervisor host is OK > > > > > => I will soon (tomorrow) be able to upgrade my pre > > > > > production > > > > > cluster to have a wider scope & performance results "at > > > > > scale" > > > > > * My crazy attempt to convert all my topologies code from > > > > > Java EE > > > > > API > > > > > to Jakarta EE API is finally working *fine*, the issue was > > > > > lying > > > > > between my keyboard and my seat (I used a misconfigured Kafka > > > > > cluster > > > > > with my mostly Kafka-depending topologies). So it means that > > > > > regardless of the fact that Storm internally still relies on > > > > > Java > > > > > EE > > > > > artefacts, a bunch of code migrated to Jakarta EE runs anway > > > > > - > > > > > that's > > > > > what I like (smooth transitions) > > > > > > > > > > I update when I'll have results on my pre-production cluster > > > > > at > > > > > scale > > > > > (max Thursday) > > > > > > > > > > Alexandre > > > > > > > > > > Le mar. 14 nov. 2023 à 16:46, Julien Nioche > > > > > <lists.digitalpeb...@gmail.com> a écrit : > > > > > > > > > > > > Thanks Richard, > > > > > > > > > > > > I have tested the latest snapshot with a simple crawl > > > > > > topology > > > > > > in > > > > > > local and > > > > > > deployed mode + a more complex one which generates WARC > > > > > > files > > > > > > (and > > > > > > therefore the HDFS resouces), again both in local and > > > > > > remote > > > > > > mode. > > > > > > No issues found, everything is looking fine > > > > > > > > > > > > Julien > > > > > > > > > > > > On Tue, 14 Nov 2023 at 09:52, Richard Zowalla > > > > > > <r...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > as written in the other thread, I have prepared another > > > > > > > SNAPSHOT > > > > > > > artifact incorporting the feedbach received in the first > > > > > > > round. > > > > > > > > > > > > > > I would like to receive some feedback on the current > > > > > > > SNAPSHOT > > > > > > > build > > > > > > > again. > > > > > > > > > > > > > > I just uploaded a 2.6.0-SNAPSHOT of Storm build from > > > > > > > > > > > > > > https://github.com/apache/storm/commit/a7aac90fdc4b87eba8dce3b851b55c86c4a390cf > > > > > > > > > > > > > > > > > > > > > It is available via the ASF Snapshot repository: > > > > > > > https://repository.apache.org/content/repositories/snapshots/ > > > > > > > > > > > > > > You can consume by adding > > > > > > > > > > > > > > <repository> > > > > > > > <id>apache.snapshots</id> > > > > > > > <name>Apache Snapshot Repository</name> > > > > > > > <url>https://repository.apache.org/snapshots</url> > > > > > > > <releases> > > > > > > > <enabled>false</enabled> > > > > > > > </releases> > > > > > > > </repository> > > > > > > > > > > > > > > to your project pom. As we do not deploy SNAPSHOTS > > > > > > > automatically, it > > > > > > > should be easy to just consume the latest SNAPSHOT. > > > > > > > > > > > > > > The packaged binaries are available at > > > > > > > nightlies.apache.org: > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/storm/2.6.0-SNAPSHOT/a7aac90fdc4b87eba8dce3b851b55c86c4a390cf/ > > > > > > > > > > > > > > > > > > > > > If you have some minutes left: Please test and report any > > > > > > > issues with > > > > > > > this binaries, so we can fix before attempting to > > > > > > > release. > > > > > > > > > > > > > > The most significant changes are some fixes of ui-related > > > > > > > bugs > > > > > > > + > > > > > > > upgrades of dependencies with CVEs, etc. > > > > > > > > > > > > > > Here is a diff: > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/storm/compare/8f883086032669a8f04b09a3b312d60af5b44533..a7aac90fdc4b87eba8dce3b851b55c86c4a390cf > > > > > > > > > > > > > > A summary in Jira is here: > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314820&version=12353484 > > > > > > > > > > > > > > Gruß > > > > > > > Richard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Open Source Solutions for Text Engineering* > > > > > > > > > > > > http://www.digitalpebble.com > > > > > > http://digitalpebble.blogspot.com/ > > > > > > #digitalpebble <http://twitter.com/digitalpebble> > > > > >
signature.asc
Description: This is a digitally signed message part
