You can do as you like ;-)
Am 4. November 2023 08:54:48 MEZ schrieb Alexandre Vermeerbergen <avermeerber...@gmail.com>: >I mean : > >Can I keep using my >'STORM-3958-Capacity-to-set-St >orm-UI's-title-in-conf/storm.yaml' >branch to push this fix, or should I create a new one? > >Le sam. 4 nov. 2023 à 08:53, Alexandre Vermeerbergen ><avermeerber...@gmail.com> a écrit : >> >> Can you keep using my >> 'STORM-3958-Capacity-to-set-Storm-UI's-title-in-conf/storm.yaml' >> branch to push this fix, or should I create a new one? >> >> Le sam. 4 nov. 2023 à 08:14, Richard Zowalla <rich...@zowalla.com> a écrit : >> > >> > Just push a fix for it (referencing the original issue) ;-) - we hadn't a >> > release in between, so not a big deal. >> > >> > Am 4. November 2023 08:01:54 MEZ schrieb Alexandre Vermeerbergen >> > <avermeerber...@gmail.com>: >> > >Temporary workaround for the Storm UI title's regression: run this >> > >command on the title-template.html file (in >> > >storm-stable/public/templates directory) after having extracted the >> > >binary archive: >> > > >> > >sed -i 's/{{ui\.title}}/{{title}}/' title-template.html >> > > >> > >Again sorry for having introduced this regression. >> > > >> > >Alexandre >> > > >> > >Le sam. 4 nov. 2023 à 07:52, Alexandre Vermeerbergen >> > ><avermeerber...@gmail.com> a écrit : >> > >> >> > >> Updates regarding my test with Storm 2.6.0 Snapshot 2023/11/02: >> > >> 1. I have modified my KO bolts using this pattern in their prepare() >> > >> method: >> > >> >> > >> long cacheMaxSize = >> > >> Long.parseLong(stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE).toString()); >> > >> >> > >> instead of: >> > >> >> > >> long cacheMaxSize = (Long) >> > >> stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE); >> > >> >> > >> 2. I have found the cause of the regression in loss of 'Storm UI' >> > >> title at the top left-hand side of all Storm UI's pages. >> > >> This is a mistake of mine in this PR: >> > >> >> > >> https://github.com/apache/storm/pull/3580/commits/0320ee6f2d367086b973b504a976ae6aff853ccd >> > >> More precisely, at line 19 of this file: >> > >> https://github.com/apache/storm/pull/3580/commits/0320ee6f2d367086b973b504a976ae6aff853ccd#diff-6d3a4162a8af826a25a2b8cd6763c08e9bc5165b736b45890b84fa50597d7481 >> > >> we must have: >> > >> <h1><a href="/">{{title}}</a></h1> >> > >> >> > >> instead of: >> > >> >> > >> <h1><a href="/">{{ui.title}}</a></h1> >> > >> >> > >> I think I missed this during my tests because of browser's cache >> > >> (bad excuse, I know...): probably I tested by-patching my deployed >> > >> Storm cluster, and I finally forgot to checkout this latest change of >> > >> my delivery. >> > >> >> > >> I had to use "private navigation" to make sure this time it fixes >> > >> this issue >> > >> >> > >> What would be the quickest way for this very simple fix to be delivered >> > >> ? >> > >> >> > >> Thanks, >> > >> Alexandre >> > >> >> > >> Le ven. 3 nov. 2023 à 20:57, Alexandre Vermeerbergen >> > >> <avermeerber...@gmail.com> a écrit : >> > >> > >> > >> > Thanks Richard for the analysis. >> > >> > >> > >> > My thought on this are very simple: let me fix my own old code, after >> > >> > all this was impacting only one of my topologies, the other one were >> > >> > using more careful Integer.parseInt or Integer.parseLong to avoid >> > >> > making assumption on the way the values in the config Map were >> > >> > serialized. >> > >> > >> > >> > My other concern with 2.6.0 Snapshot is that in Storm UI, the "Storm >> > >> > UI" title at the top left hand side of each page is missing. This may >> > >> > be the result of one of my own PR, I'll have to check why... >> > >> > >> > >> > Alex >> > >> > >> > >> > Le ven. 3 nov. 2023 à 20:23, Richard Zowalla <r...@apache.org> a >> > >> > écrit : >> > >> > > >> > >> > > Hi, >> > >> > > >> > >> > > thanks for the feedback. >> > >> > > >> > >> > > The main difference to 2.5.0 is, that we switched the JSON >> > >> > > implementation in Utils, which relied on a >> > >> > > super old JSON library. What is happening here is, that the new JSON >> > >> > > library (net.minidev) behaves differently. >> > >> > > >> > >> > > We can see it by writing some simple code without Storm (The related >> > >> > > code line with the parsing is taken from Storm Utils) >> > >> > > >> > >> > > >> > >> > > Map<String, Long> conf = new HashMap<>(); >> > >> > > conf.put("test", 100000000L); >> > >> > > >> > >> > > Map<String, Object> deserTopoConf = (Map<String, Object>) >> > >> > > JSONValue.parseWithException(JSONValue.toJSONString(conf)); >> > >> > > >> > >> > > for(Map.Entry<String, Object> e : deserTopoConf.entrySet()) { >> > >> > > System.out.println(e.getValue().getClass().getSimpleName()); >> > >> > > } >> > >> > > >> > >> > > This will print "Integer". Using "10000000000000000L" as a value >> > >> > > will >> > >> > > result in "Long". >> > >> > > >> > >> > > The main issue derives from the fact, that we serialize the config >> > >> > > map >> > >> > > into a JSON String and parse the result. >> > >> > > Based on the String representation, "100000000L" will still fit >> > >> > > into an >> > >> > > Integer, so the parser will not use Long ;-) >> > >> > > >> > >> > > If the number is big enough, the parser will switch to a Long. >> > >> > > The old library (> 10 years old now): >> > >> > > >> > >> > > deserTopoConf = (Map<String, Object>) >> > >> > > org.json.simple.JSONValue.parseWithException(org.json.simple.JSONValue. >> > >> > > toJSONString(conf)); >> > >> > > >> > >> > > for(Map.Entry<String, Object> e : deserTopoConf.entrySet()) { >> > >> > > System.out.println(e.getValue().getClass().getSimpleName()); >> > >> > > } >> > >> > > >> > >> > > This will print "Long". >> > >> > > >> > >> > > The difference between these libraries is, that the super old >> > >> > > org.json.simple just always returns Long as the type. >> > >> > > >> > >> > > If can eventually tune the new JSON parser to something like: >> > >> > > >> > >> > > deserTopoConf = (Map<String, Object>) new >> > >> > > JSONParser(JSONParser.BIG_DIGIT_UNRESTRICTED).parse(JSONValue.toJSONStr >> > >> > > ing(conf), JSONValue.defaultReader.DEFAULT); >> > >> > > >> > >> > > This will result in "Long" but might have other side-effects. >> > >> > > >> > >> > > From my point of view, the newer JSON library works as expected >> > >> > > (although it might break backward compatibility). >> > >> > > Unchecked casts are always evil, imho. In your specific case, it >> > >> > > might >> > >> > > be more save to cast to "Number" and than invoke "longValue()" >> > >> > > instead. >> > >> > > >> > >> > > Nevertheless, we might being able to restore the old behaviour by >> > >> > > tuning the JSONParser configuration. >> > >> > > >> > >> > > Thoughts? >> > >> > > >> > >> > > Gruß >> > >> > > Richard >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > Am Freitag, dem 03.11.2023 um 18:32 +0100 schrieb Alexandre >> > >> > > Vermeerbergen: >> > >> > > > On a side note, the Long value that "MyKOBolt" is set by a >> > >> > > > DefaultTopologyConfig to a value with is a java.lang.Long >> > >> > > > instance... >> > >> > > > hence it's surprising to see it internally converted into a >> > >> > > > java.lang.Integer in the Map passed to prepare() method... >> > >> > > > >> > >> > > > >> > >> > > > Le ven. 3 nov. 2023 à 18:27, Alexandre Vermeerbergen >> > >> > > > <avermeerber...@gmail.com> a écrit : >> > >> > > > > >> > >> > > > > Hello, >> > >> > > > > >> > >> > > > > I found a breaking change using Apache Storm 2.6.0 snapshot >> > >> > > > > build >> > >> > > > > dated 20231103 and without changing my topologies' code (keeping >> > >> > > > > the >> > >> > > > > same big jars). >> > >> > > > > >> > >> > > > > This seems to be tied to a datatype change in Map argument of >> > >> > > > > prepare() method of Bolt. >> > >> > > > > >> > >> > > > > For example, this Bolt is KO at runtime: >> > >> > > > > >> > >> > > > > public class MyKOBolt extends BaseRichBolt { >> > >> > > > > >> > >> > > > > @Override >> > >> > > > > public void prepare(Map stormConf, TopologyContext context, >> > >> > > > > OutputCollector collector) { >> > >> > > > > cacheMaxSize = (Long) >> > >> > > > > stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE); >> > >> > > > > cacheTtlMin = (Long) >> > >> > > > > stormConf.get(ConfKeys.SVC_DEF_CACHE_TTL_MIN); >> > >> > > > > >> > >> > > > > as it throws this error: >> > >> > > > > 2023-11-03 16:42:20.230 o.a.s.e.b.BoltExecutor >> > >> > > > > Thread-14-__system-executor[-1, -1] [INFO] Preparing bolt >> > >> > > > > __system:[-1] >> > >> > > > > 2023-11-03 16:42:20.193 o.a.s.u.Utils >> > >> > > > > Thread-15-evaluateTriggers-executor[11, 11] [ERROR] Async loop >> > >> > > > > died! >> > >> > > > > java.lang.ClassCastException: java.lang.Integer incompatible >> > >> > > > > with >> > >> > > > > java.lang.Long >> > >> > > > > at com.acme.storm.alerting.MyKOBolt >> > >> > > > > .prepare(EvaluationBolt.java:71) >> > >> > > > > ~[stormjar.jar:?] >> > >> > > > > at >> > >> > > > > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java: >> > >> > > > > 128) >> > >> > > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT] >> > >> > > > > at >> > >> > > > > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java: >> > >> > > > > 138) >> > >> > > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT] >> > >> > > > > at >> > >> > > > > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java: >> > >> > > > > 54) >> > >> > > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT] >> > >> > > > > at org.apache.storm.utils.Utils$1.run(Utils.java:393) >> > >> > > > > [storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT] >> > >> > > > > at java.lang.Thread.run(Thread.java:857) [?:?] >> > >> > > > > >> > >> > > > > whereas this bolt is fine with Storm 2.6.0: >> > >> > > > > >> > >> > > > > public class MyOKBolt extends BaseBasicBolt { >> > >> > > > > >> > >> > > > > @Override >> > >> > > > > public void prepare(final Map stormConf, final >> > >> > > > > TopologyContext >> > >> > > > > context) { >> > >> > > > > super.prepare(stormConf, context); >> > >> > > > > >> > >> > > > > this.redisPort = >> > >> > > > > Integer.parseInt((String) >> > >> > > > > stormConf.get(ConfKeys.REDIS_PORT)); >> > >> > > > > >> > >> > > > > That said, I can modify the code of "MyKOBolt" to use same >> > >> > > > > Integer.parseInt or Integer.parseLong trick, but it's the first >> > >> > > > > time >> > >> > > > > in my long history of upgrades that I have seen such runtime >> > >> > > > > incompatibility. >> > >> > > > > >> > >> > > > > Maybe "MyKOBolt" was badly written since the beginning and I >> > >> > > > > have >> > >> > > > > just >> > >> > > > > hit the punishment for it: is there a documentation which >> > >> > > > > clarifies >> > >> > > > > datatypes of the Map argument of prepare() method? >> > >> > > > > >> > >> > > > > Also worth noting: "MyKOBolt" derives from BaseRichBolt , while >> > >> > > > > "MyOKBolt" derives from BaseBasicBolt => could this have any >> > >> > > > > impact >> > >> > > > > on >> > >> > > > > this finding ? >> > >> > > > > >> > >> > > > > Last, I'm running this on Redhat Linux 8 and IBM Semeru JDK >> > >> > > > > 17.0.8.1. >> > >> > > > > >> > >> > > > > Kind regards, >> > >> > > > > Alexandre >> > >> > > > > >> > >> > > > > Le ven. 3 nov. 2023 à 15:16, Julien Nioche >> > >> > > > > <lists.digitalpeb...@gmail.com> a écrit : >> > >> > > > > > >> > >> > > > > > Thanks Richard. >> > >> > > > > > >> > >> > > > > > Tried the latest snapshot with StormCrawler both in local and >> > >> > > > > > deployed mode >> > >> > > > > > and did not find any issues. >> > >> > > > > > Will try it on a topology generating WARC files next week to >> > >> > > > > > check that the >> > >> > > > > > dependency changes on Hadoop have not broken anything. >> > >> > > > > > >> > >> > > > > > Have a good week end >> > >> > > > > > >> > >> > > > > > Julien >> > >> > > > > > >> > >> > > > > > On Thu, 2 Nov 2023 at 19:25, Richard Zowalla <r...@apache.org> >> > >> > > > > > wrote: >> > >> > > > > > >> > >> > > > > > > Short update. Sorted out an issue with the tar.gz/zip with >> > >> > > > > > > Julien today >> > >> > > > > > > and re-uploaded them to the nightlies area. >> > >> > > > > > > >> > >> > > > > > > This new bundle works as expected in my deployment but >> > >> > > > > > > happy to >> > >> > > > > > > receive >> > >> > > > > > > additional feedback before getting up a first release >> > >> > > > > > > candidate >> > >> > > > > > > ;-) >> > >> > > > > > > >> > >> > > > > > > Am Montag, dem 30.10.2023 um 08:21 +0100 schrieb Richard >> > >> > > > > > > Zowalla: >> > >> > > > > > > > Hi Alexandre, >> > >> > > > > > > > >> > >> > > > > > > > we are not in a hurry here :) - take as much as time you >> > >> > > > > > > > need. >> > >> > > > > > > > >> > >> > > > > > > > Gruß >> > >> > > > > > > > Richard >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > Am Montag, dem 30.10.2023 um 07:50 +0100 schrieb Alexandre >> > >> > > > > > > > Vermeerbergen: >> > >> > > > > > > > > Hello Richard, >> > >> > > > > > > > > >> > >> > > > > > > > > Okay, I'm more than happy to do that with my >> > >> > > > > > > > > pre-production >> > >> > > > > > > > > cluster >> > >> > > > > > > > > (~10 topologies) using the binary artifacts. >> > >> > > > > > > > > Would it be OK if I can use up to end of this week so >> > >> > > > > > > > > that >> > >> > > > > > > > > I'll be >> > >> > > > > > > > > able to have enough time to check all potential issues >> > >> > > > > > > > > that >> > >> > > > > > > > > this >> > >> > > > > > > > > upgrade bring? >> > >> > > > > > > > > >> > >> > > > > > > > > Thanks, >> > >> > > > > > > > > Alexandre >> > >> > > > > > > > > >> > >> > > > > > > > > Le lun. 30 oct. 2023 à 07:35, Richard Zowalla >> > >> > > > > > > > > <r...@apache.org> a >> > >> > > > > > > > > écrit : >> > >> > > > > > > > > > >> > >> > > > > > > > > > Hi all, >> > >> > > > > > > > > > >> > >> > > > > > > > > > before starting a release of 2.6.0-SNAPSHOT, I would >> > >> > > > > > > > > > like >> > >> > > > > > > > > > to >> > >> > > > > > > > > > receive >> > >> > > > > > > > > > some feedback on the current SNAPSHOT build. >> > >> > > > > > > > > > >> > >> > > > > > > > > > I just uploaded a 2.6.0-SNAPSHOT of Storm build from >> > >> > > > > > > > > > >> > >> > > > > > > https://github.com/apache/storm/commit/8f883086032669a8f04b09a3b312d60af5b44533 >> > >> > > > > > > > > > >> > >> > > > > > > > > > It is available via the ASF Snapshot repository: >> > >> > > > > > > > > > https://repository.apache.org/content/repositories/snapshots/ >> > >> > > > > > > > > > >> > >> > > > > > > > > > You can consume by adding >> > >> > > > > > > > > > >> > >> > > > > > > > > > <repository> >> > >> > > > > > > > > > <id>apache.snapshots</id> >> > >> > > > > > > > > > <name>Apache Snapshot Repository</name> >> > >> > > > > > > > > > >> > >> > > > > > > > > > <url>https://repository.apache.org/snapshots</url> >> > >> > > > > > > > > > <releases> >> > >> > > > > > > > > > <enabled>false</enabled> >> > >> > > > > > > > > > </releases> >> > >> > > > > > > > > > </repository> >> > >> > > > > > > > > > >> > >> > > > > > > > > > to your project pom. As we do not deploy SNAPSHOTS >> > >> > > > > > > > > > automatically, >> > >> > > > > > > > > > it >> > >> > > > > > > > > > should be easy to just consume the latest SNAPSHOT. >> > >> > > > > > > > > > >> > >> > > > > > > > > > The packaged binaries are available at >> > >> > > > > > > > > > nightlies.apache.org: >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > https://nightlies.apache.org/storm/2.6.0-SNAPSHOT/8f883086032669a8f04b09a3b312d60af5b44533/ >> > >> > > > > > > > > > >> > >> > > > > > > > > > If you have some minutes left: Please test and report >> > >> > > > > > > > > > any >> > >> > > > > > > > > > issues >> > >> > > > > > > > > > with >> > >> > > > > > > > > > this binaries, so we can fix before attempting to >> > >> > > > > > > > > > release. >> > >> > > > > > > > > > >> > >> > > > > > > > > > The most significant changes are in the are of >> > >> > > > > > > > > > hadoop/hbase/hdfs >> > >> > > > > > > > > > as >> > >> > > > > > > > > > we >> > >> > > > > > > > > > upgraded from 2.x to 3.x - our own test coverage >> > >> > > > > > > > > > within >> > >> > > > > > > > > > the build >> > >> > > > > > > > > > looks >> > >> > > > > > > > > > good but would be nice to get some real world >> > >> > > > > > > > > > use-case >> > >> > > > > > > > > > feedback >> > >> > > > > > > > > > on >> > >> > > > > > > > > > this. In addition, we had quite a lof of 3rd party >> > >> > > > > > > > > > dependency >> > >> > > > > > > > > > upgrades. >> > >> > > > > > > > > > >> > >> > > > > > > > > > In addition, it contains the pruning of external >> > >> > > > > > > > > > modules >> > >> > > > > > > > > > as >> > >> > > > > > > > > > listed >> > >> > > > > > > > > > in >> > >> > > > > > > > > > >> > >> > > > > > > > > > https://issues.apache.org/jira/browse/STORM-3988 >> > >> > > > > > > > > > >> > >> > > > > > > > > > A summaryin Jira is here: >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314820&version=12353484 >> > >> > > > > > > > > > >> > >> > > > > > > > > > Gruß >> > >> > > > > > > > > > Richard >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > -- >> > >> > > > > > >> > >> > > > > > *Open Source Solutions for Text Engineering* >> > >> > > > > > >> > >> > > > > > http://www.digitalpebble.com >> > >> > > > > > http://digitalpebble.blogspot.com/ >> > >> > > > > > #digitalpebble <http://twitter.com/digitalpebble> >> > >> > >