Hello! With existing binary archive, user can move directories from apache-ignite-fabric/libs/optional to apache-ignite-fabric/libs to activate them.
But with RPM, user should not contemplate moving directories from /usr/lib/apache-ignite/optional to /usr/lib/apache-ignite. User lacks permissions for that operation and it would defeat the purpose of having a RPM (imagine trying to upgrade Ignite RPM version with a setup like that). "Turning distribution into Slackware" should not be an offering. How it could work: Imagine we create /var/lib/ignite, use it as IGNITE_HOME, add symlinks from files in /usr/lib to /var/lib/ignite. This way, we can add and remove symlinks to modules in optional/, and thus enable and disable them as user sees fit. This also answers the problem of "what's IGNITE_HOME for visor launched from /usr/bin" But obviously this will require dedication and effort. -- Ilya Kasnacheev 2018-04-14 8:03 GMT+03:00 Peter Ivanov <mr.wei...@gmail.com>: > Current packages design (after installation) does not differ from binary > archive - everything works (except necessity to run service instead > ignite.sh) just the same way, including libs/optional. > > Also, there can be issues with system JDK version by default, but every > problem will be in journalctl and/or /var/log, and package has strong > dependency on any implementation of JDK 1.8. > > > I am lacking enough feedback about Apache Ignite from packages from real > users, don’t know real use cases so still "moving in the dark". > > > On Fri, 13 Apr 2018 at 22:18, Denis Magda <dma...@apache.org> wrote: > > > Ilya, > > > > Thanks for your inputs. The reason why we decided to split Ignite into > > several packages mimics the reason why Java community introduced modular > > subsystem for JDK. That's all about size. Ignite distribution is too big, > > and we're trying to separate it into several components so that people > can > > install only the features they need. > > > > The point of a package is to ship something into root file system that > can > > > be used from root file system. If cpp files require compilation we > should > > > not ship them, or ship them to 'examples'. Ditto with benchmarks. If > > > there's no mechanism to add optional libs to Ignite classpath, we > should > > > not ship optional libs. Moreover, some of 'optional' modules such as > yarn > > > don't make sense here because they're not supposed to be used with > > > standalone Ignite. > > > > > > Agree that we need to ship the code that is ready to be run. As for the > > classpath thing, if an optional package is installed into the root (core) > > package directory, then its jars have to be added to "ignite/libs" > folder. > > After that, the one needs to restart a cluster node, nd it will add the > > just installed optional libs to the classpath. *Petr*, does it work this > > way or can be implemented this way to address Ilya's concerns? > > > > -- > > Denis > > > > On Fri, Apr 13, 2018 at 7:00 AM, Ilya Kasnacheev < > > ilya.kasnach...@gmail.com> > > wrote: > > > > > 2018-04-13 7:42 GMT+03:00 Peter Ivanov <mr.wei...@gmail.com>: > > > > > > > On Thu, 12 Apr 2018 at 20:04, Ilya Kasnacheev < > > ilya.kasnach...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > Moreover I did not find a way to start service if default installed > > JVM > > > > is > > > > > Java 7 :( I understand it's EOL, still this is something that hit > me. > > > > > > > > > > > > Apache Ignite >=2.4 does not support Java 7 - it is said in > > > documentation, > > > > DEVNOTES and even in startup scripts. > > > > > > > > I have Java 8 too, but I could not get service from package to start > > > Ignite since there's nowhere to put JAVA_HOME (or JVM_ARGS for that > > > matter). Is it possible to specify it while running packaged Ignite? > > > > > > > > > > > > > > > > > > > > apache-ignite-libs is a totally unexpected package name. > > apache-ignite > > > > core > > > > > doesn't depend on it. It doesn't enable anything out of the box. > The > > > > > package is huge. > > > > > > > > ‘apache-ignite-libs’ is an aggregation package (for now) for all > > optional > > > > libs we are delivering. Possibly later they will be split more > granular > > > or > > > > even package per lib (like php, perl, python, etc. do for their > libs). > > > > This package dependency on ‘apache-ignite-core’ may seem confusing > > > though, > > > > I will try to explain it in IEP at least for current iteration. > > > > > > > > > > Okay, but how do you add optional libs to be included into Ignite > > classpath > > > while being launched by service? Is it even possible? If it isn't, I > > think > > > it doesn't make sense to ship apache-ignite-libs at all. > > > > > > > > > > > > > > Further naming may become clear when we’ll start initiative on > > including > > > > packages to popular Linux distributions and theirs community will > join > > > > naming discussions. > > > > > > > Renaming packages once they're deployed widely will be a pain point to > > out > > > users. Some things should probably be thought out in advance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Frankly speaking, I'm not sure that improvements over Stage I are > > > enough > > > > as > > > > > of now. For demo-like activity, we can probably go with one package > > > fits > > > > > all. > > > > > > > > > > > > > The process of finding the best package architecture is iterative, > but > > > > previously community agreed in split design proposed for 2.5 release. > > > > > > > > Also, split architecture is half of proposed improvements. The other > > > half - > > > > new process for deploying packages to Bintray (with virtually > > indefinite > > > > storage capabilities). > > > > > > > I think we could drop the split for now, or at least drop > > > apache-ignite-libs package at all. Probably also drop apache-ignite-cpp > > > package and maybe apache-ignite-benchmarks. > > > > > > The point of a package is to ship something into root file system that > > can > > > be used from root file system. If cpp files require compilation we > should > > > not ship them, or ship them to 'examples'. Ditto with benchmarks. If > > > there's no mechanism to add optional libs to Ignite classpath, we > should > > > not ship optional libs. Moreover, some of 'optional' modules such as > yarn > > > don't make sense here because they're not supposed to be used with > > > standalone Ignite. > > > > > > IMO it is not right to try and shove every file from Ignite > distribution > > > into some package. We should only put in packages things that can be > > used. > > > If something can't be used without copying it to a different FS > location, > > > it should be in examples or not packaged at all. > > > > > > In my opinion, it doesn't make sense to implement an underwhelming > > package > > > split right now just because we have agreed to have *some* package > split > > in > > > 2.5. Let's aim for happiness. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > > > > > > > > > 2018-04-12 19:10 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>: > > > > > > > > > > > If someone from PMCы or Committers still sees necessity about > > > including > > > > > > these tasks into Apache Ignite 2.5 release, this is the last > chance > > > to > > > > do > > > > > > so. > > > > > > Otherwise this task will be moved to at 2.6 release at least, or > > even > > > > > > moved to backlog indefinitely. > > > > > > > > > > > > > > > > > > > > > > > > > On 9 Apr 2018, at 19:08, Petr Ivanov <mr.wei...@gmail.com> > > wrote: > > > > > > > > > > > > > > To top new RPM architecture off, update to release process is > > > > > introduced > > > > > > — [1] [2]. > > > > > > > > > > > > > > Both tasks (this one and IGNITE-7647) are ready for review and > > > should > > > > > be > > > > > > merged simultaneously. > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8172 > > > > > > > [2] https://github.com/apache/ignite-release/pull/1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On 2 Apr 2018, at 18:22, Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com > > > > > > > > > > > wrote: > > > > > > >> > > > > > > >> Hello! > > > > > > >> > > > > > > >> Let me share my idea of how this shoud work. Splitting package > > > into > > > > > > >> sub-packages should be dependency-driven. > > > > > > >> > > > > > > >> It means that all Ignite modules without dependencies or with > > > small > > > > > > >> dependencies (such as ignite-log4j) should be included in > > > > ignite-core. > > > > > > It > > > > > > >> doesn't make sense to make a zillion RPM packages. > > > > > > >> > > > > > > >> Critical things like ignite-spring and ignite-indexing should > be > > > in > > > > > > >> ignite-core of course, even if they have dependencies. > > Ignite-core > > > > > > should > > > > > > >> be fully self-sufficient and feature-complete. > > > > > > >> > > > > > > >> However, e.g. .net API should probably be in a separate > package, > > > > > > because it > > > > > > >> should depend on mono | net-core. We may also have > ignite-devel > > > > > package > > > > > > >> which should include all modules which only make sense for > > > > developers > > > > > > who > > > > > > >> write code. Such as hibernate integration. > > > > > > >> > > > > > > >> I'm not sure about MR modules. The main question should be, > does > > > it > > > > > have > > > > > > >> dependencies? Can it run stand-alone without writing code? > > > > > > >> > > > > > > >> Hope this helps, > > > > > > >> > > > > > > >> -- > > > > > > >> Ilya Kasnacheev > > > > > > >> > > > > > > >> 2018-03-27 15:10 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>: > > > > > > >> > > > > > > >>> Hi, Igniters! > > > > > > >>> > > > > > > >>> > > > > > > >>> Here are some news on our RPM packages initiative. > > > > > > >>> > > > > > > >>> 1. I’ve finished preliminary developing of Stage II version > of > > > RPM > > > > > > >>> packages [1]. Main “new feature” is — split design. Also I’ve > > > added > > > > > > >>> package.sh script for automating package building process > which > > > > will > > > > > > help > > > > > > >>> organise corresponding builds in TC as well as simplify > process > > > for > > > > > > >>> developers who wishes to have custom packages. > > > > > > >>> PR#3703 [2] is ready for review. Denis, in order to catch up > > with > > > > > > Apache > > > > > > >>> Ignite 2.5 release, I’d greatly appreciate your help in > finding > > > > > > reviewer. > > > > > > >>> 2. With the help of ASF INFRA team, we now have RPM [3] and > DEB > > > [4] > > > > > > >>> repositories on Apache Bintray. Though they are already > > prepared > > > > for > > > > > > >>> hosting RPM and DEB packages respectively, and there is a way > > of > > > > > > linking > > > > > > >>> them to apache.org/dist/ignite page, there is possible > > > alternative > > > > > in > > > > > > >>> storing there only plain directory layout corresponding to > each > > > > > > repository > > > > > > >>> type (RPM and DEB) and manage this layout (repodata, > > > distributions, > > > > > > >>> versions, etc.) by ourselves, having more control over > > > repositories > > > > > but > > > > > > >>> lacking some simplicity of deploying new releases. WDYT? > Should > > > we > > > > > try > > > > > > >>> Cassandra approach? They are storing their DEB packages as I > > > > > described > > > > > > >>> above [5]. > > > > > > >>> > > > > > > >>> Also — a question arose while I was working on this issue: > > which > > > > OSes > > > > > > (and > > > > > > >>> which versions of each) are we going to support (if we are > > going) > > > > in > > > > > > terms > > > > > > >>> of step-by-step list? Currently RPM packages are tested only > > with > > > > > > latest > > > > > > >>> CentOS (and, respectively — RHEL), but there are a lot more > > > > RPM-based > > > > > > >>> distributives [6] some of which are more o less popular among > > OS > > > > > > community > > > > > > >>> (ALT, Fedora, openSUSE, etc.). > > > > > > >>> > > > > > > >>> > > > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647 > > > > > > >>> [2] https://github.com/apache/ignite/pull/3703 > > > > > > >>> [3] https://bintray.com/apache/ignite-rpm > > > > > > >>> [4] https://bintray.com/apache/ignite-deb > > > > > > >>> [5] https://bintray.com/apache/cassandra/debian#files/ > > > > > > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_ > > > > > > distributions > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov <mr.wei...@gmail.com> > > > > wrote: > > > > > > >>>> > > > > > > >>>> I suppose that most everything if not all from libs/options > > will > > > > go > > > > > to > > > > > > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs'). > > > > > > >>>> More precise lib selection (if something from optional would > > > > better > > > > > to > > > > > > >>> have in core package) will be discussed right after > preliminary > > > > split > > > > > > >>> architecture agreement. > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov < > > dpavlov....@gmail.com > > > > > > > > > > wrote: > > > > > > >>>>> > > > > > > >>>>> I like idea of keeping simple system of modules, so +1 from > > me. > > > > > > >>>>> > > > > > > >>>>> Where optional libs (e.g Direct IO plugin) would be > included, > > > > would > > > > > > it > > > > > > >>> be > > > > > > >>>>> core or optional? > > > > > > >>>>> > > > > > > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda < > dma...@apache.org > > >: > > > > > > >>>>> > > > > > > >>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> How big would be a final core module? > > > > > > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and > > > > create > > > > > > it’s > > > > > > >>> own > > > > > > >>>>>>> package. > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> Guys, 30 vs 280M is a huuuuge difference. I would agree > > with > > > > Petr > > > > > > and > > > > > > >>>>>> propose the simplest modular system: > > > > > > >>>>>> > > > > > > >>>>>> - core module that includes basic Ignite capabilities > > > including > > > > > SQL, > > > > > > >>>>>> compute grid, service grid, k/v > > > > > > >>>>>> - optional module hosts the rest - ML, streamers > integration > > > > > (kafka, > > > > > > >>>>>> flink), kubernetes, etc. > > > > > > >>>>>> > > > > > > >>>>>> What do you think? > > > > > > >>>>>> > > > > > > >>>>>> -- > > > > > > >>>>>> Denis > > > > > > >>>>>> > > > > > > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov < > > > > > mr.wei...@gmail.com> > > > > > > >>> wrote: > > > > > > >>>>>> > > > > > > >>>>>>> *DEB package > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov < > > mr.wei...@gmail.com> > > > > > > wrote: > > > > > > >>>>>>>> > > > > > > >>>>>>>> Considering that DEV package for now is almost platform > > > > > > independent > > > > > > >>>>>> (its > > > > > > >>>>>>> a java application more or less), that package will work > > > almost > > > > > on > > > > > > any > > > > > > >>>>>>> DEB-based linux, including but not limited to Ubuntu, > > Debian, > > > > > etc. > > > > > > >>>>>>>> The only restriction is existence of systemctl (systemd) > > > > service > > > > > > >>>>>> manager > > > > > > >>>>>>> — we are dependent on it. > > > > > > >>>>>>>> > > > > > > >>>>>>>> Thats why, for instance, our RPM repository is called > > simply > > > > > ‘rpm’ > > > > > > >>> and > > > > > > >>>>>>> package has no arch or dist suffix — it will work on > > CentOS, > > > > > RHEL, > > > > > > >>>>>> Fedora, > > > > > > >>>>>>> etc. with presence of aforementioned systemd. > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>>>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan < > > > > > > dsetrak...@apache.org> > > > > > > >>>>>>> wrote: > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> Will Debian package work for Ubuntu? > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> D. > > > > > > >>>>>>>>> > > > > > > >>>>>>>>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov < > > > > > > mr.wei...@gmail.com> > > > > > > >>>>>>> wrote: > > > > > > >>>>>>>>> > > > > > > >>>>>>>>>> Not a problem, rather nuisance. Also, when we will > move > > to > > > > > > official > > > > > > >>>>>>>>>> repositories, there can be a problem from OS > community. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> Concerning DEB packages — I plan to use RPM as base > for > > > DEB > > > > > > package > > > > > > >>>>>>> build > > > > > > >>>>>>>>>> (package layout / install scripts) for speeding up > > things > > > > and > > > > > > >>>>>> excluding > > > > > > >>>>>>>>>> possible duplication and desynchronisation, so its a > > > matter > > > > of > > > > > > ’sit > > > > > > >>>>>>> and do’ > > > > > > >>>>>>>>>> rather then some technical research. Thats why I rose > > > > > discussion > > > > > > >>>>>> about > > > > > > >>>>>>>>>> future package architecture, so that after agreement > I'm > > > be > > > > > > able to > > > > > > >>>>>>> pack > > > > > > >>>>>>>>>> both RPM and DEB identically. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> Yet, if you insist, I can create DEB package according > > to > > > > > > current > > > > > > >>> RPM > > > > > > >>>>>>>>>> layout in no time. > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>>> On 15 Mar 2018, at 04:53, Dmitriy Setrakyan < > > > > > > >>> dsetrak...@apache.org> > > > > > > >>>>>>>>>> wrote: > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> Peter, > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> I don't think the package size of 280M is going to > be a > > > > > > problem at > > > > > > >>>>>>> all, > > > > > > >>>>>>>>>> but > > > > > > >>>>>>>>>>> what you are suggesting can be an improvement down > the > > > > road. > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> In the mean time, I think our top priority should be > to > > > > > provide > > > > > > >>>>>>> packages > > > > > > >>>>>>>>>>> for Debian and Ubuntu. Having only RPMs is not nearly > > > > enough. > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> Agree? > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> D. > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>> On Wed, Mar 14, 2018 at 5:36 AM, vveider < > > > > > mr.wei...@gmail.com> > > > > > > >>>>>> wrote: > > > > > > >>>>>>>>>>> > > > > > > >>>>>>>>>>>> Hi, Igniters! > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> Release 2.4 is almost there, at least binary part of > > it, > > > > so > > > > > > I'd > > > > > > >>>>>> like > > > > > > >>>>>>> to > > > > > > >>>>>>>>>>>> move > > > > > > >>>>>>>>>>>> forward to further improve and widen AI delivery > > through > > > > > > >>> packages. > > > > > > >>>>>>>>>>>> As of now, Apache Ignite ships in RPM package > weighing > > > > about > > > > > > >>> 280M+ > > > > > > >>>>>>> and, > > > > > > >>>>>>>>>> to > > > > > > >>>>>>>>>>>> improve usability and significantly reduce required > > > > download > > > > > > >>>>>> sizes, I > > > > > > >>>>>>>>>>>> purpose that in 2.5 release we introduce splitted > > > delivery > > > > > as > > > > > > >>>>>>> follows: > > > > > > >>>>>>>>>>>> - CORE > > > > > > >>>>>>>>>>>> - bin > > > > > > >>>>>>>>>>>> - config > > > > > > >>>>>>>>>>>> - libs (!optional) > > > > > > >>>>>>>>>>>> - OPTIONAL LIBS > > > > > > >>>>>>>>>>>> - BENCHMARKS > > > > > > >>>>>>>>>>>> - DOCS (?) > > > > > > >>>>>>>>>>>> - EXAMPLES > > > > > > >>>>>>>>>>>> - .NET PLATFORM FILES > > > > > > >>>>>>>>>>>> - C++ PLATFORM FILES > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> This architecture, as I assume, will add flexibility > > (no > > > > > > reason > > > > > > >>> to > > > > > > >>>>>>>>>> download > > > > > > >>>>>>>>>>>> all 280M+ of binaries where you are to run only core > > > node > > > > > > >>>>>>> functionality) > > > > > > >>>>>>>>>>>> and > > > > > > >>>>>>>>>>>> maintainability (you are in full control of what is > > > > > installed > > > > > > on > > > > > > >>>>>> your > > > > > > >>>>>>>>>>>> system). > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> After successful architecture choice, same scheme > are > > > > > planned > > > > > > to > > > > > > >>> be > > > > > > >>>>>>>>>> used in > > > > > > >>>>>>>>>>>> DEB packages as well. > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> WDYT? > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>>>> -- > > > > > > >>>>>>>>>>>> Sent from: > > > > > http://apache-ignite-developers.2346864.n4.nabble. > > > > > > >>> com/ > > > > > > >>>>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>>>> > > > > > > >>>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > > >>>>>> > > > > > > >>>> > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
