This seems similar to sqlite which offers (maybe now by default?) a single c file that is the whole DB engine and makes it quite attractive as a local db option.
I have no strong opinion on whether or not to keep amalgamation, but I wasn’t actually aware of this use-case for mxnet and I recall how much I liked sqlite for this. On Sat, Sep 28, 2019 at 10:17 AM Tianqi Chen <tqc...@cs.washington.edu> wrote: > To give a complete picture. I will talk a bit about my experience on using > the all-in-one file approach and a few examples in the tvm project. > > Currently, tvm uses all-in-one build for its lightweight runtime in cases > where we want a separate build system and do not want to bother to use > CMake, this include > > - Standalone deployment: > https://github.com/dmlc/tvm/tree/master/apps/howto_deploy > - Android projects where we need to use NDK for cross compilation. > - iOS where we need to use XCode. > > In all these cases, all-in-one build provide the benefit of easy to > incorporate into the target project. However, we only use all-in-one build > for the "minimum runtime" that is necessary to run a model. But will not > include components that compiles and optimizes the model as things will get > out of hand pretty quickly and in those cases, having a build system > out-weights the gains of the single file. > > So if there is a minimum runtime that MXNet wants to incorporate into the > users' env and the developers do not want to code up a CMake recipe for > that, it is a possible route. And in that case, I would still suggest to > move away from the current approach that create a single file but follow > the approach in the above link. In both cases, the current amalgamation is > not fulfilling its function so I think it is fine to remove and add new > ones with thoughts if necessary > > TQ > > On Sat, Sep 28, 2019 at 9:49 AM Tianqi Chen <tqc...@cs.washington.edu> > wrote: > > > The main use of amalgamation(aka all in one file build) for cases where > it > > is hard to setup a Make system. Most user knows how to include a single > > file into their project, but it is relatively harder to incorporate an > > entire build system. > > > > As a result, all-in-one file runtime is still being quite widely used and > > I personally liked the approach, I just suggested that the current > approach > > may not be the best way to go and creates some maintenance burden. > > > > See the link of an example project that uses new all-in-one approach that > > i mentioned(which illustrates the usecase of all-in-one file as well) > > https://github.com/dmlc/tvm/tree/master/apps/howto_deploy > > > > TQ > > > > On Sat, Sep 28, 2019 at 3:46 AM Marco de Abreu <marco.g.ab...@gmail.com> > > wrote: > > > >> Do we have a good knowledge and overview over all the use cases that use > >> Amalgamation? At least from my perspective I don't feel well informed > >> about > >> the blast radius. > >> > >> -Marco > >> > >> Junru Shao <junrushao1...@gmail.com> schrieb am Sa., 28. Sep. 2019, > >> 09:14: > >> > >> > As Tianqi and Sheng mentioned, given the fact that we are able to do > >> > deployment in a possibly better way (correct me if I was wrong), I > would > >> > love to +1 to Pedro’s proposal. > >> > > >> > In the meantime, as a healthy open source community, I also agree with > >> > Naveen’s point that we should do more homework for both our developers > >> and > >> > customers. IMHO, for example, it would be super helpful if Pedro may > >> bring > >> > up some documentation describing what is the “best practice” of using > >> the > >> > alternative of amalgamation, if our community agree to deprecate it. > >> > > >> > Thank you guys so much for the discussion! > >> > > >> > Junru > >> > > >> > On Fri, Sep 27, 2019 at 17:00 Tianqi Chen <tqc...@cs.washington.edu> > >> > wrote: > >> > > >> > > The original amalgamation tries to generate a single file for > >> compilation > >> > > via a script. This is largely un-necessary, having a file that > >> include > >> > the > >> > > dependent files and compile that one is relatively easy and > sometimes > >> > more > >> > > robust(without expanding everything into a single file). > >> > > > >> > > I think it might makes sense to deprecate and remove the current one > >> > given > >> > > that it is no longer properly functioning. If someone is interested, > >> > create > >> > > another deployment example that is more standalone without the file > >> > > expansion. Here is an reference of the "new style" used in the tvm > >> > project > >> > > https://github.com/dmlc/tvm/tree/master/apps/howto_deploy > >> > > > >> > > TQ > >> > > > >> > > On Fri, Sep 27, 2019 at 4:49 PM Sheng Zha <zhash...@apache.org> > >> wrote: > >> > > > >> > > > As an alternative to amalgamation, could we simply ask users to > >> > > statically > >> > > > link to libmxnet.a, and then prune the symbol table to get rid of > >> the > >> > > > binary of unused functions? Though I don't know the full context > of > >> > > > amalgamation, based on my knowledge on this feature I'm not aware > of > >> > any > >> > > > difference in the end result, compared to the code-inlining > approach > >> > that > >> > > > amalgamation takes. > >> > > > > >> > > > -sz > >> > > > > >> > > > On 2019/09/12 17:29:02, Naveen Swamy <mnnav...@gmail.com> wrote: > >> > > > > so the original email suggesting to remove was after all > >> self-serving > >> > > :) > >> > > > > > >> > > > > let's encourage if someone wants to maintain and make use of the > >> > > original > >> > > > > work and make it better. > >> > > > > > >> > > > > -1 to remove at this point > >> > > > > > >> > > > > P.S: I suggest to do some due diligence before bringing topics > up > >> for > >> > > > > discussion. > >> > > > > > >> > > > > On Wed, Sep 11, 2019 at 8:10 AM Lv, Tao A <tao.a...@intel.com> > >> > wrote: > >> > > > > > >> > > > > > Sorry to chime in. > >> > > > > > > >> > > > > > There is a PR to fix amalgamation. I was pinged several times > to > >> > > merge > >> > > > it > >> > > > > > but I don't think I have enough knowledge to do that. So it > >> would > >> > be > >> > > > great > >> > > > > > if someone from this thread can help to review. > >> > > > > > > >> > > > > > https://github.com/apache/incubator-mxnet/pull/15303 > >> > > > > > > >> > > > > > thanks, > >> > > > > > -tao > >> > > > > > > >> > > > > > -----Original Message----- > >> > > > > > From: Marco de Abreu <marco.g.ab...@gmail.com> > >> > > > > > Sent: Wednesday, September 11, 2019 9:38 PM > >> > > > > > To: dev@mxnet.incubator.apache.org > >> > > > > > Subject: Re: [DISCUSS] Remove amalgamation > >> > > > > > > >> > > > > > Is Amalgamation only used on Android though? Are there any > other > >> > use > >> > > > cases? > >> > > > > > > >> > > > > > -Marco > >> > > > > > > >> > > > > > Pedro Larroy <pedro.larroy.li...@gmail.com> schrieb am Mi., > 11. > >> > Sep. > >> > > > 2019, > >> > > > > > 11:57: > >> > > > > > > >> > > > > > > Hi Anirudh > >> > > > > > > > >> > > > > > > Appreciate your feedback and sorry if my email came across > >> that > >> > way > >> > > > to > >> > > > > > > you, I think you might miss some context. I don't think > >> calling > >> > > > > > > something hacky is anything bad and isn't supposed to be the > >> > topic > >> > > of > >> > > > > > > the discussion. It was reported as not working by users, > hence > >> > the > >> > > > > > > original thread. It was a request for opinions from people > who > >> > > might > >> > > > > > > actually have tried to work in Mxnet on Android. > >> > > > > > > > >> > > > > > > Thanks. > >> > > > > > > > >> > > > > > > Pedro. > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tuesday, September 10, 2019, Anirudh Subramanian > >> > > > > > > <anirudh2...@gmail.com > >> > > > > > > > > >> > > > > > > wrote: > >> > > > > > > > Hi Pedro, > >> > > > > > > > > >> > > > > > > > I don't see anything "destructive" with Chris asking for > >> > > > > > > > justification > >> > > > > > > for > >> > > > > > > > you calling something "hacky". The only email in this > thread > >> > > where > >> > > > I > >> > > > > > > > see > >> > > > > > > ad > >> > > > > > > > hominems and disrespectful comments is your email. > >> > > > > > > > > >> > > > > > > > On Sat, Sep 7, 2019, 10:18 PM Pedro Larroy > >> > > > > > > > <pedro.larroy.li...@gmail.com > >> > > > > > > > > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > >> Apache mentors should have a look at these reincident > >> > harassment > >> > > > > > > >> and destructive behaviors which demotivate contributions > >> and > >> > > take > >> > > > > > > >> action. It takes only one bad apple to ruin a community. > >> > > > > > > >> > >> > > > > > > >> The mobile solution that is known to work as of know is > >> cross > >> > > > > > > >> compiling with "ci/build.py -p build.android_armv8" or > >> > > > > > > >> "build.android_armv7". The only advantage of amalgamation > >> is > >> > to > >> > > > > > > >> provide a smaller binary that we > >> > > > > > > could > >> > > > > > > >> accomplish with the C preprocessor. > >> > > > > > > >> > >> > > > > > > >> My technical contributions speak for themselves, > including > >> > > porting > >> > > > > > > >> MXNet > >> > > > > > > to > >> > > > > > > >> Android and ARM and helping many users run MXNet in > Jetson, > >> > > > > > > >> Raspberry Pi and Android amongst many other topics. I > have > >> > never > >> > > > > > > >> been disrespectful > >> > > > > > > to > >> > > > > > > >> anyone. I'm entitled to my own technical opinions about > >> > > > > > > >> amalgamation or > >> > > > > > > any > >> > > > > > > >> other piece of code whatsoever, that's no personal > >> disrespect > >> > to > >> > > > > > > >> anyone > >> > > > > > > and > >> > > > > > > >> perfectly valid. If you are not interested in this > project > >> > > > anymore, > >> > > > > > > >> do > >> > > > > > > us > >> > > > > > > >> all a favor and stop trolling and being toxic. If you > want > >> my > >> > > > > > > >> respect, > >> > > > > > > step > >> > > > > > > >> up your technical contributions, be positive and > encourage > >> > > others, > >> > > > > > > >> this including commits, I haven't seen for many months, > >> please > >> > > be > >> > > > > > > >> positive > >> > > > > > > and > >> > > > > > > >> constructive. This scorched-earth attitude is only > >> reflecting > >> > > bad > >> > > > > > > >> on > >> > > > > > > you. > >> > > > > > > >> I'm certainly not interested in your ad-hominems or > unasked > >> > for > >> > > > > > > technical > >> > > > > > > >> advice, which to be honest, showing poor judgment and > >> > > ignorance. > >> > > > > > > >> Myself and others have come up with numbers, graphs, > >> metrics > >> > and > >> > > > > > > >> arguments and have been met with dismissal, trolling and > >> > > > > > > >> sea-lioning. I have recieved your insults via public and > >> > private > >> > > > > > > >> channels (such as linkedin) as have others. This is not > ok > >> and > >> > > has > >> > > > > > > >> to stop. If you have something personal against me or > >> against > >> > > your > >> > > > > > > >> former employer, this is not the right place > >> > > > > > > or > >> > > > > > > >> forum. > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> On Fri, Sep 6, 2019 at 3:56 PM Chris Olivier > >> > > > > > > >> <cjolivie...@gmail.com> > >> > > > > > > >> wrote: > >> > > > > > > >> > >> > > > > > > >> > Hi Pedro, > >> > > > > > > >> > > >> > > > > > > >> > While I was not involved with amalgamation or its > >> > development > >> > > in > >> > > > > > > >> > any > >> > > > > > > way, > >> > > > > > > >> > can you please refrain from referring to the work of > >> others > >> > > as a > >> > > > > > > "hacky > >> > > > > > > >> > solution"? This is derogatory slang and the statement > >> was > >> > not > >> > > > > > > supported > >> > > > > > > >> > with any justification for such name-calling. Someone > >> > spent a > >> > > > > > > >> > good > >> > > > > > > deal > >> > > > > > > >> of > >> > > > > > > >> > time on this solution at some point in time and I am > >> sure it > >> > > > > > > >> > worked > >> > > > > > > for > >> > > > > > > >> its > >> > > > > > > >> > purpose at that time -- I think it was used in the > >> original > >> > > > > > > >> > javascript > >> > > > > > > >> port > >> > > > > > > >> > as well, actually -- and it is disrespectful to call > >> their > >> > > > > > > >> > efforts "hacky". Please respect what came before. > >> > > > > > > >> > > >> > > > > > > >> > Thanks for understanding, > >> > > > > > > >> > > >> > > > > > > >> > -Chris > >> > > > > > > >> > > >> > > > > > > >> > > >> > > > > > > >> > On Fri, Sep 6, 2019 at 3:07 PM Pedro Larroy < > >> > > > > > > >> pedro.larroy.li...@gmail.com> > >> > > > > > > >> > wrote: > >> > > > > > > >> > > >> > > > > > > >> > > Hi > >> > > > > > > >> > > > >> > > > > > > >> > > I would like to propose to remove amalgamation from > >> MXNet > >> > > and > >> > > > > > > >> > > CI, > >> > > > > > > users > >> > > > > > > >> > > have reported that they couldn't use it successfully > in > >> > > > > > > >> > > Android, and instead they were able to use the cross > >> > > compiled > >> > > > > > > >> > > docker build > >> > > > > > > >> > successfully. > >> > > > > > > >> > > > >> > > > > > > >> > > Any reason why we shouldn't remove this hacky > solution? > >> > > > > > > >> > > > >> > > > > > > >> > > Pedro. > >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
