I mean, that we should fix this issue, not just throw away zookeeper discovery.
чт, 17 сент. 2020 г. в 16:44, Ivan Daschinsky <ivanda...@gmail.com>: > I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension. > This tiny recipe-like class brings tons of dependency and has nothing > common with ZookeeperDiscovery, except it name. > > ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and > zookeeper-jute-3.5.5.jar. > > > > > чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev <ilya.kasnach...@gmail.com>: > >> Hello! >> >> The situation as follows: >> libs/optional/ignite-kubernetes: >> ignite-kubernetes-2.8.1.jar jackson-annotations-2.9.10.jar >> jackson-core-2.9.10.jar jackson-databind-2.9.10.jar >> >> libs/optional/ignite-zookeeper: >> curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar >> jackson-core-asl-1.9.13.jar log4j-1.2.17.jar >> slf4j-log4j12-1.7.7.jar >> curator-framework-4.2.0.jar guava-16.0.1.jar >> jackson-mapper-asl-1.9.13.jar zookeeper-3.5.5.jar >> curator-recipes-4.2.0.jar ignite-zookeeper-2.8.1.jar >> slf4j-api-1.7.7.jar >> zookeeper-jute-3.5.5.jar >> >> ignite-zookeeper just has way more dependencies than ignite-kubernetes, >> and >> their size is 5-fold. >> >> These dependencies are not just weight but also vulnerability surface. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky <ivanda...@gmail.com>: >> >> > Hi! I recently found that in slim binary release (at least in nightly >> build >> > of 2.9.0) there is no ignite-zookeeper module. >> > But, for example, ignite-kubernetes is in. >> > Petr, could you please clarify why this decision was made? >> > We currently have only two discovery implementation in project and leave >> > only one alternative looks not obvious for me. >> > >> > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov < >> > alexey.scherbak...@gmail.com>: >> > >> > > Downloading additional jars will not 100% work in private networks, >> so we >> > > must provide full distro anyway. >> > > >> > > Also, I do not see any problems in downloading all dependencies, >> because >> > > their summary size is not too big. >> > > >> > > I suggest, in addition to tgz archive for manual deployment, provide >> > Ignite >> > > distribution as RPM (or other package manager(s)) for Linux (MSI for >> > > Windows) to automate distribution upgrades, instead of reinventing the >> > > wheel. >> > > >> > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko < >> > > valentin.kuliche...@gmail.com>: >> > > >> > > > Hi Nikolay, >> > > > >> > > > Thanks for your feedback! Absolutely, the IEP is currently in the >> draft >> > > > mode - we will be adding more details going forward. Also, Petr >> > mentioned >> > > > that he wants to create a prototype which I think will make things >> > > clearer >> > > > for all of us. >> > > > >> > > > As for "download jars from the Internet on a production server", >> that's >> > > not >> > > > necessarily what we propose. In case of Maven, artifacts are >> downloaded >> > > > from a specified repository, which doesn't have to be on the >> Internet. >> > > > Companies that have security concerns typically have a mirror/proxy >> > > > containing approved artifacts only, already checked for viruses, >> etc. >> > > > Production servers use the proxy rather than public repositories. >> > > > >> > > > The concerns that you brought up are certainly valid. However, I >> > believe >> > > > that using Maven actually addresses them, because Maven (as any >> other >> > > > package/dependency manager for that matter) provides the >> corresponding >> > > > functionality out of the box. >> > > > >> > > > On the contrary, users who currently download our ZIP package are >> > > *forced* >> > > > to download modules and dependencies that they do not even need and >> > will >> > > > not use. In my experience, this is actually a bigger source of >> concern. >> > > > >> > > > Anyway, let us add more details to the IEP, and we will go from >> there. >> > > > >> > > > P.S. I'm also not sure I agree with your assessment of the embedded >> > mode >> > > > and thick clients, but that's probably a different discussion. >> Please >> > > feel >> > > > free to create a new thread if you would like to discuss this. >> > > > >> > > > -Val >> > > > >> > > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov < >> nizhi...@apache.org> >> > > > wrote: >> > > > >> > > > > Also, guys. >> > > > > >> > > > > Can we make one step back and add some requirements to the IEP. >> > > > > How we want to install and upgrade Ignite? >> > > > > What specific issues we want to solve? >> > > > > >> > > > > With this input we can be on the same page during solution >> > discussion. >> > > > > >> > > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov < >> nizhikov....@gmail.com> >> > > > > написал(а): >> > > > > > >> > > > > > Hello, Val. >> > > > > > >> > > > > > Sorry, but I sill don’t understand how «download jars from the >> > > internet >> > > > > on production server» approach helps us to solve the issues you >> > > describe. >> > > > > > >> > > > > > Moreover, with this enhancement you want to add more >> dependencies >> > to >> > > > > Ignite and create some security vulnerabilities: >> > > > > > * Maven tool - new dependency. I doubt that and of the >> > > production >> > > > > server have it >> > > > > > * Internet maven repository - many production servers >> > restrict >> > > > > outgoing internet connection to any addresses. >> > > > > > It’s hard to understand why distributed DB should >> > have >> > > > the >> > > > > ability to connect to some kind of address on the Internet. >> > > > > > * All Ignite distributions can be compromised with the >> hack >> > of >> > > > > some random maven repo or with the malicious proxy. >> > > > > > * Note, that Ignite installation or upgrade procedure can >> > > become >> > > > > dead just because of some Maven repo down. >> > > > > > >> > > > > > >> > > > > > I have 2 concerns: >> > > > > > >> > > > > > 1. We shouldn’t download any jars from the internet during >> > production >> > > > > deployment or upgrade procedure. >> > > > > > 2. The IEP description, for now, is quite small for such >> important >> > > > > things as a way we distribute and upgrade Ignite. >> > > > > > Petr, can you, please, make IEP more specific. >> > > > > > I think we should add the following details: >> > > > > > * Description of the typical Ignite deployment >> > > procedure >> > > > > including folder structure(config, persistence, .m2 and other >> > folders) >> > > > > > * This also should include examples of the >> > bash >> > > > > commands if any required. >> > > > > > * Description of the typical upgrade(downgrade?) >> > > > procedure >> > > > > > * This also should include examples of the >> > bash >> > > > > commands if any required. >> > > > > > * Description of «single tool responsible for all >> > > > > operations». >> > > > > > * all commands >> > > > > > * all operations >> > > > > > * all parameters >> > > > > > >> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which >> > > effectively >> > > > > forces (or at least motivates) users to put everything related to >> > > Ignite >> > > > > under IGNITE_HOME (configs, additional JARs, etc.). >> > > > > > >> > > > > > AFAIU this issue doesn’t relate to the way Ignite distributed. >> > > > > > We can and should fix this outside of this IEP. >> > > > > > >> > > > > >> 2. Modules are enabled/disabled manually by moving folders >> around… >> > > > this >> > > > > is error-prone, >> > > > > > >> > > > > > Makes sense. >> > > > > > Let’s fix this without «download jar from the internet»? >> > > > > > We can have some kind of config with the enabled modules. >> > > > > > >> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. >> However, >> > > if >> > > > > you look at how this ZIP is built, you will notice that we don't >> > > actually >> > > > > include all transitive dependencies, >> > > > > > >> > > > > > We need to fix this. >> > > > > > Let’s include all the required JARs to the distribution. >> > > > > > >> > > > > >> which means that an embedded node that uses Maven can have a >> > > different >> > > > > classpath. >> > > > > > >> > > > > > Embedded Ignite node is an error-prone design decision in the >> first >> > > > > place. >> > > > > > We can’t fix it with the «download jars from the internet» >> > approach. >> > > > > > We should eliminate the client node as a thing and use a thin >> > clients >> > > > > instead of it. >> > > > > > >> > > > > >> 4. ignite.sh allows manual modifications (there are several >> > > "Uncomment >> > > > > this to..." lines). This complicates upgrades as well. >> > > > > > >> > > > > > Makes sense. >> > > > > > Let’s fix this. >> > > > > > >> > > > > >> The whole purpose of the IEP is to fix the above, automate and >> > > > simplify >> > > > > operations for the users. The general idea is to create a single >> tool >> > > > > responsible for all those operations and use Maven under the hood >> to >> > > > > include/exclude modules. >> > > > > > >> > > > > > For now, IEP lacks the description of this tool. >> > > > > > >> > > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko < >> > > > > valentin.kuliche...@gmail.com> написал(а): >> > > > > >> >> > > > > >> Hi Nikolay, >> > > > > >> >> > > > > >> Here are some of the issues that I see with the current binary >> ZIP >> > > > > package >> > > > > >> that we create for every release. >> > > > > >> >> > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which >> > > effectively >> > > > > >> forces (or at least motivates) users to put everything related >> to >> > > > Ignite >> > > > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a >> > user >> > > > > >> downloads and unzips a new version, they have to somehow merge >> the >> > > > > content >> > > > > >> together - that's poor upgradability. >> > > > > >> 2. Modules are enabled/disabled manually by moving folders >> around. >> > > Not >> > > > > only >> > > > > >> this is error-prone, but it also adds more issues to >> > upgradability - >> > > > you >> > > > > >> have to apply these steps for every version you download. >> > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. >> However, >> > > if >> > > > > you >> > > > > >> look at how this ZIP is built, you will notice that we don't >> > > actually >> > > > > >> include all transitive dependencies, which means that an >> embedded >> > > node >> > > > > that >> > > > > >> uses Maven can have a different classpath. It seems that we've >> > been >> > > > > lucky >> > > > > >> so far as this hasn't caused any issues, but I still think the >> > > > approach >> > > > > is >> > > > > >> wrong. The behavior should be consistent. >> > > > > >> 4. ignite.sh allows manual modifications (there are several >> > > "Uncomment >> > > > > this >> > > > > >> to..." lines). This complicates upgrades as well. >> > > > > >> >> > > > > >> The whole purpose of the IEP is to fix the above, automate and >> > > > > >> simplify operations for the users. The general idea is to >> create a >> > > > > single >> > > > > >> tool responsible for all those operations and use Maven under >> the >> > > hood >> > > > > to >> > > > > >> include/exclude modules. >> > > > > >> >> > > > > >> Exact commands and steps are still under discussion though. It >> > looks >> > > > > like >> > > > > >> Petr is going to create a prototype that can be used as a >> starting >> > > > > point, >> > > > > >> but if you have any suggestions or ideas in the meantime, >> please >> > > share >> > > > > them >> > > > > >> with us. >> > > > > >> >> > > > > >> -Val >> > > > > >> >> > > > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov < >> > > nizhi...@apache.org >> > > > > >> > > > > >> wrote: >> > > > > >> >> > > > > >>> Hello, Igniters. >> > > > > >>> >> > > > > >>> Not sure if I understand the essence of the proposal at the >> > moment. >> > > > > >>> Can you, please, describe the advantages of the proposed way >> from >> > > the >> > > > > user >> > > > > >>> perspective? >> > > > > >>> >> > > > > >>> How the typical DevOps pipeline should look like with this >> > > > enhancement? >> > > > > >>> >> > > > > >>> How I as a user can create a fully functional installation >> > package >> > > of >> > > > > the >> > > > > >>> Ignite? >> > > > > >>> AFAIK downloading some artifacts from the internet straight to >> > the >> > > > > >>> production server is a security anti-pattern. >> > > > > >>> >> > > > > >>> >> > > > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dma...@apache.org> >> > > > написал(а): >> > > > > >>>> >> > > > > >>>> Petr, thanks, >> > > > > >>>> >> > > > > >>>> There is also a collection of modules located in our >> extensions >> > > > > >>> repository: >> > > > > >>>> https://github.com/apache/ignite-extensions >> > > > > >>>> >> > > > > >>>> @Saikat Maitra <saikat.mai...@gmail.com> is migrating all >> our >> > > > > existing >> > > > > >>>> integrations to that repository and, once this is done, the >> > > > extensions >> > > > > >>> will >> > > > > >>>> be released to Maven separately. Is it reasonable to expand >> the >> > > > scope >> > > > > of >> > > > > >>>> the IEP-52 and contemplate on how to install those >> extensions? >> > > > > >>>> >> > > > > >>>> - >> > > > > >>>> Denis >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko < >> > > > > >>>> valentin.kuliche...@gmail.com> wrote: >> > > > > >>>> >> > > > > >>>>> Hi Petr, >> > > > > >>>>> >> > > > > >>>>> The proposal makes sense to me - thanks for starting the >> > > > discussion. >> > > > > >>>>> Current installation and upgrade procedures involve a lot of >> > > manual >> > > > > >>> steps >> > > > > >>>>> and quite error-prone, we need to automate and simplify >> them as >> > > > much >> > > > > as >> > > > > >>>>> possible. >> > > > > >>>>> >> > > > > >>>>> I believe we eventually should end up with a unified >> > command-line >> > > > > tool >> > > > > >>>>> (ignitectl?) that will incorporate all the operations >> > > > (enable/disable >> > > > > >>>>> modules, start/stop, configuration updates, activation, >> > baseline, >> > > > > etc.). >> > > > > >>>>> This IEP is a step in this direction. >> > > > > >>>>> >> > > > > >>>>> Looking forward to testing a prototype :) >> > > > > >>>>> >> > > > > >>>>> -Val >> > > > > >>>>> >> > > > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov < >> > mr.wei...@gmail.com >> > > > >> > > > > >>> wrote: >> > > > > >>>>> >> > > > > >>>>>> Hi, Igniters! >> > > > > >>>>>> >> > > > > >>>>>> >> > > > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of >> > enhanced >> > > > > >>> delivery >> > > > > >>>>>> and upgrade processes [1]. >> > > > > >>>>>> The key idea is to make main binary as slim as possible >> > > > (delivering >> > > > > >>> every >> > > > > >>>>>> optional component by demand only) while providing way to >> run >> > > each >> > > > > new >> > > > > >>>>>> version seamlessly without much of the efforts migrating >> data >> > > and >> > > > > >>>>>> configuration between upgrades. >> > > > > >>>>>> >> > > > > >>>>>> I plan to create prototype based on current release of >> Apache >> > > > Ignite >> > > > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in >> > September. >> > > > > >>>>>> >> > > > > >>>>>> >> > > > > >>>>>> >> > > > > >>>>>> [1] >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>> >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958 >> > > > > >>>>>> >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>> >> > > > > >>> >> > > > > > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > -- >> > > >> > > Best regards, >> > > Alexei Scherbakov >> > > >> > >> > >> > -- >> > Sincerely yours, Ivan Daschinskiy >> > >> > > > -- > Sincerely yours, Ivan Daschinskiy > -- Sincerely yours, Ivan Daschinskiy
