Hello François, Thank you for your very interesting feedback. Please find my reponses below:
On Mon, Feb 17, 2025 at 9:13 AM Francois Gerthoffert < [email protected]> wrote: > Hello Serge, > > I'm sorry, I was on vacation and couldn't attend the meeting. > > I see that you mentioned moving everything into one single branch, > waiting for things to stabilize, and then splitting the change into > multiple PRs. > > This is indeed a good idea, it is always better to introduce changes > in small reviewable increments, but if I could make a recommendation, > it would be to not consider it an "Unomi v3" branch, but instead an > experimental branch, since some of the changes you will introduce will > not necessarily be breaking (for example, I believe Opensearch support > is not breaking). > I don't mind doing this, but the idea will also to be able to a version 3 beta release asap. > > Once you have this experimental branch in a state you are happy with, > you'll have a good idea of what is breaking or not, and in that case, > you could begin by submitting non-breaking changes in small individual > PRs, making it possible for these to be part of Unomi v2. > From that point, the decision could be made to either: > - release a v2 with these recent changes, bump master to v3, and > create a maintenance branch for Unomi v2 (preferable IMHO) > - or to create a Unomi v3 branch and handle it separately > This is where I am a bit worried because some of the changes required in the services for example have required a lot of changes in the same files, so separating them in terms of commit is going to take some work, which is something that will not be trivial to do, although I will do my best to do it in parts to make it easier for review. I have been looking into tooling such as Graphite and even developing a tool to build PRs from a common changeset. But as I explained, many of these changes were required during "big" developments such as OpenSearch or multi-tenancy support, just to be able to work. For example I added some Karaf shell commands because it was the only way to properly understand the state of some of the services. The unified caching system was also introduced because it was becoming a big problem to manage all the different caches in each system and issues where constantly coming back and having to be fixed multiple times manually in different "almost" similar classes. This is not a new issue and was already diagnosed by Damien a long time ago, he's the one who actually suggested unifiying all this code. > This approach helps in creating a v3.0 release containing mostly > breaking changes, by keeping all the non-breaking changes in the "last > v2" it can help in reducing migration overhead for the community. > Btw I have already started looking into the migration process and hopefully I will be able to contribute something for v3 beta. > > In all cases, I believe it is best to avoid having one v3 branch > containing many unreviewed changes, hoping to have all of these > reviewed at once in a very large PR towards the end of the process. > Yes I can understand that. I am trying to avoid this as I understand the difficulty, but my focus is first on making sure all the developments for V3 are completed, then stabilized, then tested (automatically), and then I'll look into preparing them for review, although I'll probably first share the code as one branch anyway so that people that are interested can already test it and start looking into the changes if they want. Last but not least, I'd just like to say that while I understand the concerns, I am also a little surprised that there doesn't seem to be a lot of interest/enthusiasm for these improvements by the current developers. This is a lot of work I think would benefit the community, some of them, like multi-tenancy, have been requested a lot by the community and this is a very large body of work that will be contributed that will surely help the project in many ways. However, if there is no/little interest for them I don't have to contribute them I could just keep them as separate code in my own personal repository but I would prefer not to have to do that because I truly believe the community could benefit from these. I have also put a LOT of work into automated testing, for example the OpenSearch contribution was about 50% time spent on making sure all the tests could work properly with the automated tests, because I do understand the concern for high-quality testable features. For v3 I have introduced a new unit testing framework that makes it a lot easier and faster to test code and I have put a big focus on having good coverage (you can now simply test a backend service from your IDE without having to launch Elasticsearch !). Some of the new tests I have been working on have more lines of code than the system being tested, and include concurrency testing and edge cases. Best regards, Serge... > Hi all, > > > > Here are the meeting notes from the latest monthly meeting. Please don’t > > hesitate to reply to this email if I forgot or misrepresented something. > > > > Best regards, > > > > Serge Huber. > > > > > > Unomi meeting notes : > > > > ElasticSearch support: > > > > - ElasticSearch 9 support is important to Jahia developers because ES > 7 > > support will be dropped 9 months after the release (until end 2025). > > - Are seeing concurrency issues with less of writes, some event and > > merges are lost. Have reproduction scenario. Profile properties are > > overwritten because of concurrency issues. > > > > > > Developer experience > > > > - Talked about Developer experience around event processing, schema > > issues, which actions are triggered. > > - We have Karaf shell commands but they require an SSH connection and > > are not accessible via HTTP request like the API calls > > - Created a JIRA ticket with a proposal for an explain query parameter > > on the eventcollector. Serge prototyped the idea and it seems like a > very > > good option. A ThreadLocal object could be used to do this. > > - Another idea would be to change the log level temporary but might > not > > be viable, and could generate lots of noise in the experience anyway. > > > > > > Unomi V3 discussions > > > > > > > > - Serge proposed Unomi V3 that will be initially unstable > > - He is targeting a first version by the end of February > > - Serge talked about removing the clustering support and endpoint. > > Question about the cluster removal and load distribution. We used to > want > > to know which IPs to distribute to from a Javascript script. We could > > replace this DNS level handling which is very common on lots of > > installations and a standard for production environment. We should > > - For the new scheduler, could we have configurable purge jobs. The > idea > > would be to have an API to do it. > > - Talked about new SchedulerService > > - Talked about new Multi-tenancy work > > - Serge will present a lot of the changes scheduled for V3. > > > > > > Code reviews: > > > > - Romain mentioned that the changes should be individually reviewable > > - Romain is concerned that for V3 there will be to have a lots of > > changes to be reviewed at the same time > > - Serge will look into splitting things into PRs after the fact once > the > > changes stabilize, it should be possible to some degree that will help > > reviewers. > > > > > > Migration: > > > > - Discussion about migration, have issues with processing large sets > of > > data > > - Looking for solutions to maybe duplicate queries to have both the > old > > and the new system up to date before switching to the new one. > > - At the API level, to have a proxy that would duplicate requests, > using > > journals as well. Does something already exist ? (Serge is pretty sure > > there must be existing solutions for this) >
