I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension. This tiny recipe-like class brings tons of dependency and has nothing common with ZookeeperDiscovery, except it name.
ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and zookeeper-jute-3.5.5.jar. чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev <ilya.kasnach...@gmail.com>: > Hello! > > The situation as follows: > libs/optional/ignite-kubernetes: > ignite-kubernetes-2.8.1.jar jackson-annotations-2.9.10.jar > jackson-core-2.9.10.jar jackson-databind-2.9.10.jar > > libs/optional/ignite-zookeeper: > curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar > jackson-core-asl-1.9.13.jar log4j-1.2.17.jar > slf4j-log4j12-1.7.7.jar > curator-framework-4.2.0.jar guava-16.0.1.jar > jackson-mapper-asl-1.9.13.jar zookeeper-3.5.5.jar > curator-recipes-4.2.0.jar ignite-zookeeper-2.8.1.jar > slf4j-api-1.7.7.jar > zookeeper-jute-3.5.5.jar > > ignite-zookeeper just has way more dependencies than ignite-kubernetes, and > their size is 5-fold. > > These dependencies are not just weight but also vulnerability surface. > > Regards, > -- > Ilya Kasnacheev > > > чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky <ivanda...@gmail.com>: > > > Hi! I recently found that in slim binary release (at least in nightly > build > > of 2.9.0) there is no ignite-zookeeper module. > > But, for example, ignite-kubernetes is in. > > Petr, could you please clarify why this decision was made? > > We currently have only two discovery implementation in project and leave > > only one alternative looks not obvious for me. > > > > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov < > > alexey.scherbak...@gmail.com>: > > > > > Downloading additional jars will not 100% work in private networks, so > we > > > must provide full distro anyway. > > > > > > Also, I do not see any problems in downloading all dependencies, > because > > > their summary size is not too big. > > > > > > I suggest, in addition to tgz archive for manual deployment, provide > > Ignite > > > distribution as RPM (or other package manager(s)) for Linux (MSI for > > > Windows) to automate distribution upgrades, instead of reinventing the > > > wheel. > > > > > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko < > > > valentin.kuliche...@gmail.com>: > > > > > > > Hi Nikolay, > > > > > > > > Thanks for your feedback! Absolutely, the IEP is currently in the > draft > > > > mode - we will be adding more details going forward. Also, Petr > > mentioned > > > > that he wants to create a prototype which I think will make things > > > clearer > > > > for all of us. > > > > > > > > As for "download jars from the Internet on a production server", > that's > > > not > > > > necessarily what we propose. In case of Maven, artifacts are > downloaded > > > > from a specified repository, which doesn't have to be on the > Internet. > > > > Companies that have security concerns typically have a mirror/proxy > > > > containing approved artifacts only, already checked for viruses, etc. > > > > Production servers use the proxy rather than public repositories. > > > > > > > > The concerns that you brought up are certainly valid. However, I > > believe > > > > that using Maven actually addresses them, because Maven (as any other > > > > package/dependency manager for that matter) provides the > corresponding > > > > functionality out of the box. > > > > > > > > On the contrary, users who currently download our ZIP package are > > > *forced* > > > > to download modules and dependencies that they do not even need and > > will > > > > not use. In my experience, this is actually a bigger source of > concern. > > > > > > > > Anyway, let us add more details to the IEP, and we will go from > there. > > > > > > > > P.S. I'm also not sure I agree with your assessment of the embedded > > mode > > > > and thick clients, but that's probably a different discussion. Please > > > feel > > > > free to create a new thread if you would like to discuss this. > > > > > > > > -Val > > > > > > > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <nizhi...@apache.org > > > > > > wrote: > > > > > > > > > Also, guys. > > > > > > > > > > Can we make one step back and add some requirements to the IEP. > > > > > How we want to install and upgrade Ignite? > > > > > What specific issues we want to solve? > > > > > > > > > > With this input we can be on the same page during solution > > discussion. > > > > > > > > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov < > nizhikov....@gmail.com> > > > > > написал(а): > > > > > > > > > > > > Hello, Val. > > > > > > > > > > > > Sorry, but I sill don’t understand how «download jars from the > > > internet > > > > > on production server» approach helps us to solve the issues you > > > describe. > > > > > > > > > > > > Moreover, with this enhancement you want to add more dependencies > > to > > > > > Ignite and create some security vulnerabilities: > > > > > > * Maven tool - new dependency. I doubt that and of the > > > production > > > > > server have it > > > > > > * Internet maven repository - many production servers > > restrict > > > > > outgoing internet connection to any addresses. > > > > > > It’s hard to understand why distributed DB should > > have > > > > the > > > > > ability to connect to some kind of address on the Internet. > > > > > > * All Ignite distributions can be compromised with the hack > > of > > > > > some random maven repo or with the malicious proxy. > > > > > > * Note, that Ignite installation or upgrade procedure can > > > become > > > > > dead just because of some Maven repo down. > > > > > > > > > > > > > > > > > > I have 2 concerns: > > > > > > > > > > > > 1. We shouldn’t download any jars from the internet during > > production > > > > > deployment or upgrade procedure. > > > > > > 2. The IEP description, for now, is quite small for such > important > > > > > things as a way we distribute and upgrade Ignite. > > > > > > Petr, can you, please, make IEP more specific. > > > > > > I think we should add the following details: > > > > > > * Description of the typical Ignite deployment > > > procedure > > > > > including folder structure(config, persistence, .m2 and other > > folders) > > > > > > * This also should include examples of the > > bash > > > > > commands if any required. > > > > > > * Description of the typical upgrade(downgrade?) > > > > procedure > > > > > > * This also should include examples of the > > bash > > > > > commands if any required. > > > > > > * Description of «single tool responsible for all > > > > > operations». > > > > > > * all commands > > > > > > * all operations > > > > > > * all parameters > > > > > > > > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which > > > effectively > > > > > forces (or at least motivates) users to put everything related to > > > Ignite > > > > > under IGNITE_HOME (configs, additional JARs, etc.). > > > > > > > > > > > > AFAIU this issue doesn’t relate to the way Ignite distributed. > > > > > > We can and should fix this outside of this IEP. > > > > > > > > > > > >> 2. Modules are enabled/disabled manually by moving folders > around… > > > > this > > > > > is error-prone, > > > > > > > > > > > > Makes sense. > > > > > > Let’s fix this without «download jar from the internet»? > > > > > > We can have some kind of config with the enabled modules. > > > > > > > > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. > However, > > > if > > > > > you look at how this ZIP is built, you will notice that we don't > > > actually > > > > > include all transitive dependencies, > > > > > > > > > > > > We need to fix this. > > > > > > Let’s include all the required JARs to the distribution. > > > > > > > > > > > >> which means that an embedded node that uses Maven can have a > > > different > > > > > classpath. > > > > > > > > > > > > Embedded Ignite node is an error-prone design decision in the > first > > > > > place. > > > > > > We can’t fix it with the «download jars from the internet» > > approach. > > > > > > We should eliminate the client node as a thing and use a thin > > clients > > > > > instead of it. > > > > > > > > > > > >> 4. ignite.sh allows manual modifications (there are several > > > "Uncomment > > > > > this to..." lines). This complicates upgrades as well. > > > > > > > > > > > > Makes sense. > > > > > > Let’s fix this. > > > > > > > > > > > >> The whole purpose of the IEP is to fix the above, automate and > > > > simplify > > > > > operations for the users. The general idea is to create a single > tool > > > > > responsible for all those operations and use Maven under the hood > to > > > > > include/exclude modules. > > > > > > > > > > > > For now, IEP lacks the description of this tool. > > > > > > > > > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko < > > > > > valentin.kuliche...@gmail.com> написал(а): > > > > > >> > > > > > >> Hi Nikolay, > > > > > >> > > > > > >> Here are some of the issues that I see with the current binary > ZIP > > > > > package > > > > > >> that we create for every release. > > > > > >> > > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which > > > effectively > > > > > >> forces (or at least motivates) users to put everything related > to > > > > Ignite > > > > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a > > user > > > > > >> downloads and unzips a new version, they have to somehow merge > the > > > > > content > > > > > >> together - that's poor upgradability. > > > > > >> 2. Modules are enabled/disabled manually by moving folders > around. > > > Not > > > > > only > > > > > >> this is error-prone, but it also adds more issues to > > upgradability - > > > > you > > > > > >> have to apply these steps for every version you download. > > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. > However, > > > if > > > > > you > > > > > >> look at how this ZIP is built, you will notice that we don't > > > actually > > > > > >> include all transitive dependencies, which means that an > embedded > > > node > > > > > that > > > > > >> uses Maven can have a different classpath. It seems that we've > > been > > > > > lucky > > > > > >> so far as this hasn't caused any issues, but I still think the > > > > approach > > > > > is > > > > > >> wrong. The behavior should be consistent. > > > > > >> 4. ignite.sh allows manual modifications (there are several > > > "Uncomment > > > > > this > > > > > >> to..." lines). This complicates upgrades as well. > > > > > >> > > > > > >> The whole purpose of the IEP is to fix the above, automate and > > > > > >> simplify operations for the users. The general idea is to > create a > > > > > single > > > > > >> tool responsible for all those operations and use Maven under > the > > > hood > > > > > to > > > > > >> include/exclude modules. > > > > > >> > > > > > >> Exact commands and steps are still under discussion though. It > > looks > > > > > like > > > > > >> Petr is going to create a prototype that can be used as a > starting > > > > > point, > > > > > >> but if you have any suggestions or ideas in the meantime, please > > > share > > > > > them > > > > > >> with us. > > > > > >> > > > > > >> -Val > > > > > >> > > > > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov < > > > nizhi...@apache.org > > > > > > > > > > >> wrote: > > > > > >> > > > > > >>> Hello, Igniters. > > > > > >>> > > > > > >>> Not sure if I understand the essence of the proposal at the > > moment. > > > > > >>> Can you, please, describe the advantages of the proposed way > from > > > the > > > > > user > > > > > >>> perspective? > > > > > >>> > > > > > >>> How the typical DevOps pipeline should look like with this > > > > enhancement? > > > > > >>> > > > > > >>> How I as a user can create a fully functional installation > > package > > > of > > > > > the > > > > > >>> Ignite? > > > > > >>> AFAIK downloading some artifacts from the internet straight to > > the > > > > > >>> production server is a security anti-pattern. > > > > > >>> > > > > > >>> > > > > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dma...@apache.org> > > > > написал(а): > > > > > >>>> > > > > > >>>> Petr, thanks, > > > > > >>>> > > > > > >>>> There is also a collection of modules located in our > extensions > > > > > >>> repository: > > > > > >>>> https://github.com/apache/ignite-extensions > > > > > >>>> > > > > > >>>> @Saikat Maitra <saikat.mai...@gmail.com> is migrating all our > > > > > existing > > > > > >>>> integrations to that repository and, once this is done, the > > > > extensions > > > > > >>> will > > > > > >>>> be released to Maven separately. Is it reasonable to expand > the > > > > scope > > > > > of > > > > > >>>> the IEP-52 and contemplate on how to install those extensions? > > > > > >>>> > > > > > >>>> - > > > > > >>>> Denis > > > > > >>>> > > > > > >>>> > > > > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko < > > > > > >>>> valentin.kuliche...@gmail.com> wrote: > > > > > >>>> > > > > > >>>>> Hi Petr, > > > > > >>>>> > > > > > >>>>> The proposal makes sense to me - thanks for starting the > > > > discussion. > > > > > >>>>> Current installation and upgrade procedures involve a lot of > > > manual > > > > > >>> steps > > > > > >>>>> and quite error-prone, we need to automate and simplify them > as > > > > much > > > > > as > > > > > >>>>> possible. > > > > > >>>>> > > > > > >>>>> I believe we eventually should end up with a unified > > command-line > > > > > tool > > > > > >>>>> (ignitectl?) that will incorporate all the operations > > > > (enable/disable > > > > > >>>>> modules, start/stop, configuration updates, activation, > > baseline, > > > > > etc.). > > > > > >>>>> This IEP is a step in this direction. > > > > > >>>>> > > > > > >>>>> Looking forward to testing a prototype :) > > > > > >>>>> > > > > > >>>>> -Val > > > > > >>>>> > > > > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov < > > mr.wei...@gmail.com > > > > > > > > > >>> wrote: > > > > > >>>>> > > > > > >>>>>> Hi, Igniters! > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of > > enhanced > > > > > >>> delivery > > > > > >>>>>> and upgrade processes [1]. > > > > > >>>>>> The key idea is to make main binary as slim as possible > > > > (delivering > > > > > >>> every > > > > > >>>>>> optional component by demand only) while providing way to > run > > > each > > > > > new > > > > > >>>>>> version seamlessly without much of the efforts migrating > data > > > and > > > > > >>>>>> configuration between upgrades. > > > > > >>>>>> > > > > > >>>>>> I plan to create prototype based on current release of > Apache > > > > Ignite > > > > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in > > September. > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> [1] > > > > > >>>>>> > > > > > >>>>> > > > > > >>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958 > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>> > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > -- Sincerely yours, Ivan Daschinskiy
