To give a complete picture. I will talk a bit about my experience on using the all-in-one file approach and a few examples in the tvm project.
Currently, tvm uses all-in-one build for its lightweight runtime in cases where we want a separate build system and do not want to bother to use CMake, this include - Standalone deployment: https://github.com/dmlc/tvm/tree/master/apps/howto_deploy - Android projects where we need to use NDK for cross compilation. - iOS where we need to use XCode. In all these cases, all-in-one build provide the benefit of easy to incorporate into the target project. However, we only use all-in-one build for the "minimum runtime" that is necessary to run a model. But will not include components that compiles and optimizes the model as things will get out of hand pretty quickly and in those cases, having a build system out-weights the gains of the single file. So if there is a minimum runtime that MXNet wants to incorporate into the users' env and the developers do not want to code up a CMake recipe for that, it is a possible route. And in that case, I would still suggest to move away from the current approach that create a single file but follow the approach in the above link. In both cases, the current amalgamation is not fulfilling its function so I think it is fine to remove and add new ones with thoughts if necessary TQ On Sat, Sep 28, 2019 at 9:49 AM Tianqi Chen <tqc...@cs.washington.edu> wrote: > The main use of amalgamation(aka all in one file build) for cases where it > is hard to setup a Make system. Most user knows how to include a single > file into their project, but it is relatively harder to incorporate an > entire build system. > > As a result, all-in-one file runtime is still being quite widely used and > I personally liked the approach, I just suggested that the current approach > may not be the best way to go and creates some maintenance burden. > > See the link of an example project that uses new all-in-one approach that > i mentioned(which illustrates the usecase of all-in-one file as well) > https://github.com/dmlc/tvm/tree/master/apps/howto_deploy > > TQ > > On Sat, Sep 28, 2019 at 3:46 AM Marco de Abreu <marco.g.ab...@gmail.com> > wrote: > >> Do we have a good knowledge and overview over all the use cases that use >> Amalgamation? At least from my perspective I don't feel well informed >> about >> the blast radius. >> >> -Marco >> >> Junru Shao <junrushao1...@gmail.com> schrieb am Sa., 28. Sep. 2019, >> 09:14: >> >> > As Tianqi and Sheng mentioned, given the fact that we are able to do >> > deployment in a possibly better way (correct me if I was wrong), I would >> > love to +1 to Pedro’s proposal. >> > >> > In the meantime, as a healthy open source community, I also agree with >> > Naveen’s point that we should do more homework for both our developers >> and >> > customers. IMHO, for example, it would be super helpful if Pedro may >> bring >> > up some documentation describing what is the “best practice” of using >> the >> > alternative of amalgamation, if our community agree to deprecate it. >> > >> > Thank you guys so much for the discussion! >> > >> > Junru >> > >> > On Fri, Sep 27, 2019 at 17:00 Tianqi Chen <tqc...@cs.washington.edu> >> > wrote: >> > >> > > The original amalgamation tries to generate a single file for >> compilation >> > > via a script. This is largely un-necessary, having a file that >> include >> > the >> > > dependent files and compile that one is relatively easy and sometimes >> > more >> > > robust(without expanding everything into a single file). >> > > >> > > I think it might makes sense to deprecate and remove the current one >> > given >> > > that it is no longer properly functioning. If someone is interested, >> > create >> > > another deployment example that is more standalone without the file >> > > expansion. Here is an reference of the "new style" used in the tvm >> > project >> > > https://github.com/dmlc/tvm/tree/master/apps/howto_deploy >> > > >> > > TQ >> > > >> > > On Fri, Sep 27, 2019 at 4:49 PM Sheng Zha <zhash...@apache.org> >> wrote: >> > > >> > > > As an alternative to amalgamation, could we simply ask users to >> > > statically >> > > > link to libmxnet.a, and then prune the symbol table to get rid of >> the >> > > > binary of unused functions? Though I don't know the full context of >> > > > amalgamation, based on my knowledge on this feature I'm not aware of >> > any >> > > > difference in the end result, compared to the code-inlining approach >> > that >> > > > amalgamation takes. >> > > > >> > > > -sz >> > > > >> > > > On 2019/09/12 17:29:02, Naveen Swamy <mnnav...@gmail.com> wrote: >> > > > > so the original email suggesting to remove was after all >> self-serving >> > > :) >> > > > > >> > > > > let's encourage if someone wants to maintain and make use of the >> > > original >> > > > > work and make it better. >> > > > > >> > > > > -1 to remove at this point >> > > > > >> > > > > P.S: I suggest to do some due diligence before bringing topics up >> for >> > > > > discussion. >> > > > > >> > > > > On Wed, Sep 11, 2019 at 8:10 AM Lv, Tao A <tao.a...@intel.com> >> > wrote: >> > > > > >> > > > > > Sorry to chime in. >> > > > > > >> > > > > > There is a PR to fix amalgamation. I was pinged several times to >> > > merge >> > > > it >> > > > > > but I don't think I have enough knowledge to do that. So it >> would >> > be >> > > > great >> > > > > > if someone from this thread can help to review. >> > > > > > >> > > > > > https://github.com/apache/incubator-mxnet/pull/15303 >> > > > > > >> > > > > > thanks, >> > > > > > -tao >> > > > > > >> > > > > > -----Original Message----- >> > > > > > From: Marco de Abreu <marco.g.ab...@gmail.com> >> > > > > > Sent: Wednesday, September 11, 2019 9:38 PM >> > > > > > To: dev@mxnet.incubator.apache.org >> > > > > > Subject: Re: [DISCUSS] Remove amalgamation >> > > > > > >> > > > > > Is Amalgamation only used on Android though? Are there any other >> > use >> > > > cases? >> > > > > > >> > > > > > -Marco >> > > > > > >> > > > > > Pedro Larroy <pedro.larroy.li...@gmail.com> schrieb am Mi., 11. >> > Sep. >> > > > 2019, >> > > > > > 11:57: >> > > > > > >> > > > > > > Hi Anirudh >> > > > > > > >> > > > > > > Appreciate your feedback and sorry if my email came across >> that >> > way >> > > > to >> > > > > > > you, I think you might miss some context. I don't think >> calling >> > > > > > > something hacky is anything bad and isn't supposed to be the >> > topic >> > > of >> > > > > > > the discussion. It was reported as not working by users, hence >> > the >> > > > > > > original thread. It was a request for opinions from people who >> > > might >> > > > > > > actually have tried to work in Mxnet on Android. >> > > > > > > >> > > > > > > Thanks. >> > > > > > > >> > > > > > > Pedro. >> > > > > > > >> > > > > > > >> > > > > > > On Tuesday, September 10, 2019, Anirudh Subramanian >> > > > > > > <anirudh2...@gmail.com >> > > > > > > > >> > > > > > > wrote: >> > > > > > > > Hi Pedro, >> > > > > > > > >> > > > > > > > I don't see anything "destructive" with Chris asking for >> > > > > > > > justification >> > > > > > > for >> > > > > > > > you calling something "hacky". The only email in this thread >> > > where >> > > > I >> > > > > > > > see >> > > > > > > ad >> > > > > > > > hominems and disrespectful comments is your email. >> > > > > > > > >> > > > > > > > On Sat, Sep 7, 2019, 10:18 PM Pedro Larroy >> > > > > > > > <pedro.larroy.li...@gmail.com >> > > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > >> Apache mentors should have a look at these reincident >> > harassment >> > > > > > > >> and destructive behaviors which demotivate contributions >> and >> > > take >> > > > > > > >> action. It takes only one bad apple to ruin a community. >> > > > > > > >> >> > > > > > > >> The mobile solution that is known to work as of know is >> cross >> > > > > > > >> compiling with "ci/build.py -p build.android_armv8" or >> > > > > > > >> "build.android_armv7". The only advantage of amalgamation >> is >> > to >> > > > > > > >> provide a smaller binary that we >> > > > > > > could >> > > > > > > >> accomplish with the C preprocessor. >> > > > > > > >> >> > > > > > > >> My technical contributions speak for themselves, including >> > > porting >> > > > > > > >> MXNet >> > > > > > > to >> > > > > > > >> Android and ARM and helping many users run MXNet in Jetson, >> > > > > > > >> Raspberry Pi and Android amongst many other topics. I have >> > never >> > > > > > > >> been disrespectful >> > > > > > > to >> > > > > > > >> anyone. I'm entitled to my own technical opinions about >> > > > > > > >> amalgamation or >> > > > > > > any >> > > > > > > >> other piece of code whatsoever, that's no personal >> disrespect >> > to >> > > > > > > >> anyone >> > > > > > > and >> > > > > > > >> perfectly valid. If you are not interested in this project >> > > > anymore, >> > > > > > > >> do >> > > > > > > us >> > > > > > > >> all a favor and stop trolling and being toxic. If you want >> my >> > > > > > > >> respect, >> > > > > > > step >> > > > > > > >> up your technical contributions, be positive and encourage >> > > others, >> > > > > > > >> this including commits, I haven't seen for many months, >> please >> > > be >> > > > > > > >> positive >> > > > > > > and >> > > > > > > >> constructive. This scorched-earth attitude is only >> reflecting >> > > bad >> > > > > > > >> on >> > > > > > > you. >> > > > > > > >> I'm certainly not interested in your ad-hominems or unasked >> > for >> > > > > > > technical >> > > > > > > >> advice, which to be honest, showing poor judgment and >> > > ignorance. >> > > > > > > >> Myself and others have come up with numbers, graphs, >> metrics >> > and >> > > > > > > >> arguments and have been met with dismissal, trolling and >> > > > > > > >> sea-lioning. I have recieved your insults via public and >> > private >> > > > > > > >> channels (such as linkedin) as have others. This is not ok >> and >> > > has >> > > > > > > >> to stop. If you have something personal against me or >> against >> > > your >> > > > > > > >> former employer, this is not the right place >> > > > > > > or >> > > > > > > >> forum. >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> On Fri, Sep 6, 2019 at 3:56 PM Chris Olivier >> > > > > > > >> <cjolivie...@gmail.com> >> > > > > > > >> wrote: >> > > > > > > >> >> > > > > > > >> > Hi Pedro, >> > > > > > > >> > >> > > > > > > >> > While I was not involved with amalgamation or its >> > development >> > > in >> > > > > > > >> > any >> > > > > > > way, >> > > > > > > >> > can you please refrain from referring to the work of >> others >> > > as a >> > > > > > > "hacky >> > > > > > > >> > solution"? This is derogatory slang and the statement >> was >> > not >> > > > > > > supported >> > > > > > > >> > with any justification for such name-calling. Someone >> > spent a >> > > > > > > >> > good >> > > > > > > deal >> > > > > > > >> of >> > > > > > > >> > time on this solution at some point in time and I am >> sure it >> > > > > > > >> > worked >> > > > > > > for >> > > > > > > >> its >> > > > > > > >> > purpose at that time -- I think it was used in the >> original >> > > > > > > >> > javascript >> > > > > > > >> port >> > > > > > > >> > as well, actually -- and it is disrespectful to call >> their >> > > > > > > >> > efforts "hacky". Please respect what came before. >> > > > > > > >> > >> > > > > > > >> > Thanks for understanding, >> > > > > > > >> > >> > > > > > > >> > -Chris >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > On Fri, Sep 6, 2019 at 3:07 PM Pedro Larroy < >> > > > > > > >> pedro.larroy.li...@gmail.com> >> > > > > > > >> > wrote: >> > > > > > > >> > >> > > > > > > >> > > Hi >> > > > > > > >> > > >> > > > > > > >> > > I would like to propose to remove amalgamation from >> MXNet >> > > and >> > > > > > > >> > > CI, >> > > > > > > users >> > > > > > > >> > > have reported that they couldn't use it successfully in >> > > > > > > >> > > Android, and instead they were able to use the cross >> > > compiled >> > > > > > > >> > > docker build >> > > > > > > >> > successfully. >> > > > > > > >> > > >> > > > > > > >> > > Any reason why we shouldn't remove this hacky solution? >> > > > > > > >> > > >> > > > > > > >> > > Pedro. >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > >> >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
