re the discussions about "doing it right" vs "cowboy method" (my words, please don't be offended):
I realized that I needed to put my money where my mouth is so I've now created PR-2700 [1] which incorporates Peter Hull's fix and adds a pipeline which builds the native lib for Linux, Windows and MacOS. I've also spelled out in the PR how we retain some type of (un)support for legacy platforms and how we move this fix into the NB distribution. Please have a look. If you want to know what the build result looks like prior to the PR being merged you can have a look in GitHub Actions UI on my repo [2]. Look in the "Artifacts" section which is the result of the build. I think doing it this way has more of a future in it. Example: Unless we trust Apple's Rosetta as a permanent solution we may soon have to support the Apple Silicon platform (aka macos-arm64) and with this approach it will be trivial to add. My point is: we cannot keep dodging and I hope I've hereby shown a path forward not just in this specific case (the lib.profiler) but also for those other crucial native binaries: the app launcher, the NBI launcher, the "cleaner.exe", etc. Basically GitHub Actions workflows come to the rescue. Hail to Microsoft for sponsoring this .. although I bet there's an agenda there by MS. :-) Btw: This is ASF software being build on GitHub Actions infra. I trust there's no ASF rule that says that ASF software *must* be build on ASF's Jenkins ? /Lars [1] https://github.com/apache/netbeans/pull/2700 [2] https://github.com/lbruun/netbeans/actions/runs/501551040 On Sat, Jan 16, 2021 at 5:26 PM Eric Bresie <ebre...@gmail.com> wrote: > > There is the resent distribution of the nb-javac distribution thread > leveraging compiled binaries there as well. Would the profiler components > fit in the same model? So make a project for the profiler, cause the > builds, have a binary release component then pull this in? > > On Sat, Jan 16, 2021 at 10:22 AM Eric Bresie <ebre...@gmail.com> wrote: > > > On Sat, Jan 16, 2021 at 9:58 AM Jan Lahoda <lah...@gmail.com> wrote: > > > >> On Sat, Jan 16, 2021 at 2:48 PM Lars Bruun-Hansen <lbruunhan...@gmail.com > >> > > >> wrote: > >> > >> > > What would it take for the CDDL code to be relicensed to AL2 and > >> > incorporated? > >> > > >> > The C code involved here [1] is indeed already AL2 licensed, meaning > >> > it has already been donated from Oracle and is now incorporated into > >> > the NetBeans Git repo. However, no binaries are ever build from this > >> > code because a proper build pipeline is missing. Instead the NetBeans > >> > distribution (still) uses binaries build long, long time ago from the > >> > Oracle-Hg repo and thus those binaries are CDDL licensed. > >> > > >> > > >> > > So is one possibility to break out the bin zip into multiple zips to > >> > allow more independence between bin files? > >> > Theoretically, yes. Say if one wanted to avoid re-building the binary > >> > for MacOS just because there's a need for a re-build of the Windows > >> > binary. But as I mention such separation would mean quite a lot of > >> > changes to the Java code which utilizes the binary package. I would > >> > > >> > >> I don't think it would be so bad. The zip with the binaries is unpacked by > >> the build process, and (AFAIK) only this unpacked version is used in the > >> product. It would be trivial to have another AL2-licensed zip, which would > >> be unpacked over the unpacked CDDL licensed zip, overwriting same-named > >> files. > >> > >> > >> > question if it is worth the effort compared to biting to bullet once > >> > and for all by making sure that these binaries can be auto-build from > >> > the code in the Apache NetBeans git. > >> > > >> > >> Well - if someone is up to this task, I surely don't have anything > >> against. > >> But please be prepared this means compiling the code on many platforms, > >> some of which might be considered marginal. > >> > >> Regarding "auto-build" - I am not sure what exactly is the idea there. The > >> standard NetBeans builds are generally meant to be platform neutral, not > >> sure what is the proposed way to achieve that. Is there a way to > >> cross-compiling a library for e.g. MacOS X on any other platform? > >> > > > > So recently while working a given PR, when checked in to github I noticed > > it performed some pulls, build, and sanity checks which seemed to have some > > platform specific checks. So is something similar to that in the build > > phased needed then? > > > > Where does that type of automated build activity get define? > > > > Are these platform (is) specific components something that would be better > > broken out all together to allow independent builds? Or is it still pretty > > Netbeans centric enough that it doesn’t make sense to separate it? > > > > Given the recent Netbeans Java editor VSCode integration is this another > > case of breaking it down and handled similar to that? > > > > Assume there maybe some flavor of platform specific build flag or platform > > specific build environment builds in use during build? > > > > So have platform specific build earlier in the mix and then higher level > > (Java based) build after that! > > > > Where are the platform specific aspects contained in the codebase? Assume > > this is the “native” code mentioned earlier? > > > > > >> > > >> > There is at least one positive example where setting up such build > >> > pipeline for C/C++ code has actually been done in Apache NetBeans > >> > universe, namely the Windows launcher. See [2]. Something similar > >> > needs to be done for the Profile Launcher binary. > >> > > >> > >> Note that this is optional, and requires non-trivial set-up. At very > >> least, > >> it needs MinGW, and I am not clear if this is tested outside of the Linux > >> environment. Overall, I suspect most NetBeans builds done by developers > >> don't actually build the launchers. Also, compared to the profiler > >> binaries, this is very simple: the only target is Windows. > >> > >> Overall, my suggestion would be to create a new zip for the AL2-licensed > >> profiler binaries, and unpack it over the unpacked content of the existing > >> CDDL-licensed archive. Should be reasonably easy to setup. > >> > > > > Where is the build for these defined? > > > > Jan > >> > >> > /Lars > >> > > >> > [1] > >> > > >> https://github.com/apache/netbeans/tree/master/profiler/lib.profiler/native > >> > [2] https://github.com/apache/netbeans/pull/1073 > >> > > >> > On Sat, Jan 16, 2021 at 2:03 PM Eric Bresie <ebre...@gmail.com> wrote: > >> > > > >> > > So is one possibility to break out the bin zip into multiple zips to > >> > allow more independence between bin files? > >> > > > >> > > What would it take for the CDDL code to be relicensed to AL2 and > >> > incorporated? > >> > > > >> > > Not knowing enough about said components but could the in some way be > >> > extracted out in some way an published someplace (maven central, etc.) > >> and > >> > made a compile time dependency in Netbeans which is pulled from > >> repository > >> > at build time (maybe runtime)? > >> > > > >> > > Eric Bresie > >> > > ebre...@gmail.com (mailto:ebre...@gmail.com) > >> > > > >> > > > On January 13, 2021 at 4:49:21 AM CST, Ernie Rael < > >> err...@raelity.com > >> > (mailto:err...@raelity.com)> wrote: > >> > > > Lars, maybe I'm just lucky in not knowing all the build > >> complexities. I > >> > > > have no direct experience with CI at all even in small projects. > >> > > > > >> > > > My simplistic view is that somewhere there is a binary and that it > >> > could > >> > > > be replaced with a new binary file. And. if needed, stick another > >> > > > license file (or reference to a license file) somewhere. I haven't > >> > heard > >> > > > it said that the same binary used for 32 and 64 bit archs; so I > >> guess > >> > > > that's not a problem. > >> > > > > >> > > > 1) and 2) are nice and, from what you say, easy. Considering that > >> the > >> > > > bits have been in use for over two years, it doesn't seem required; > >> but > >> > > > hey, it's easy so why not. > >> > > > > >> > > > The problem is back to your point 3). I'll never understand the > >> process > >> > > > well enough to follow the discussion in any meaningful way (and I'm > >> > > > quite content with that); and in my ignorance I will never believe > >> that > >> > > > some binary can't just be replaced without requiring all this other > >> > > > stuff to happen. Worst case, the dll could be replaced during > >> > > > installation, or even downloaded through the plugin manager couldn't > >> > it? > >> > > > > >> > > > I guess there's a legitimate argument that things can't be fixed > >> > > > piecemeal, because of support costs. Perhaps some investment in > >> > > > "piecemeal fix" tools, would have great dividends considering all > >> the > >> > > > things that can't be done because they don't fit into the right way > >> to > >> > > > do it. > >> > > > > >> > > > With the cut off for the next release at hand, I've brought the > >> issue > >> > > > up. I'll shut up about it now... At least until it's time for the > >> next > >> > > > release ;-) > >> > > > > >> > > > -ernie > >> > > > > >> > > > On 1/12/2021 11:44 PM, Lars Bruun-Hansen wrote: > >> > > > > Ernie, my comments were indeed very specific to the case at hand: > >> the > >> > > > > fix to the native binary for the Profiler Launch for Windows 64 > >> bit > >> > > > > and how to get that fix into the NetBeans distribution. > >> > > > > > >> > > > > It just happens to be exactly the same challenge in the other > >> cases > >> > I mention. > >> > > > > > >> > > > > I wrote my piece because I looked into if I could contribute for > >> this > >> > > > > specific problem (NETBEANS-1428, PR-2021). > >> > > > > Let's recap: > >> > > > > > >> > > > > 1. Pedro's PR-2021 needs to be further reviewed and then - if > >> > approved - merged. > >> > > > > 2. Then a new binary needs to be build for Windows 64 bit, the > >> file > >> > > > > profilerinterface.dll, based on the new Apache code, i.e. the code > >> > > > > which was just merged. It is not as such difficult to build the > >> new > >> > > > > DLL locally, I can do it. Pedro has already done it (#1). No > >> sweat. > >> > > > > 3. The new binary then needs to become part of > >> > > > > https://netbeans.osuosl.org/binaries/ > >> <hash>-profiler-external-binaries-8.2.zip > >> > ( > >> > > >> https://netbeans.osuosl.org/binaries/%3Chash%3E-profiler-external-binaries-8.2.zip > >> > ) > >> > > > > (#2) in order to be picked up by the NetBeans build pipeline. > >> > However, > >> > > > > we cannot just replace that single DLL in that ZIP because then > >> the > >> > > > > ZIP file would become a mix of CDDL licensed binaries and AL2 > >> > licensed > >> > > > > binaries and from two different source code repos. Ouch!. So > >> there's > >> > > > > some complexity there. As a quick fix we want to avoid rebuilding > >> for > >> > > > > all those other platforms and only re-build for Windows, perhaps > >> even > >> > > > > only for Windows 64bit, right? We can maybe change the Java code > >> for > >> > > > > lib.profiler so that there's a external binaries ZIP for Windows > >> and > >> > > > > another for the rest of the platforms but that is a somewhat > >> involved > >> > > > > task and it adds complexity. Again, this specific design, the > >> common > >> > > > > ZIP, is shared with the other use cases I mention and the issue > >> about > >> > > > > integrating a new binary is exactly similar. > >> > > > > > >> > > > > This is why I struggle to see an easy quick fix for the problem at > >> > > > > hand, integrating the new binary for the Profiler Launcher solely > >> for > >> > > > > Windows 64 bit. You very quickly end up in a situation where you > >> need > >> > > > > to rebuild binaries for MacOS and Linux too. > >> > > > > > >> > > > > I understand your frustration but I think this explains why people > >> > > > > keep "hijacking" or digressing to the bigger picture as you > >> mention > >> > > > > and then the specific issue stalls. How annoying. The reason is > >> it is > >> > > > > just not that easy to see a path to a quick fix. From my analysis > >> my > >> > > > > conclusion is that we might as well tackle the bigger picture, > >> i.e. > >> > > > > creating an ASF build pipeline for those binaries. And I'm > >> normally > >> > > > > all for quick fixes :-) > >> > > > > > >> > > > > > >> > > > > Peace. > >> > > > > > >> > > > > /Lars > >> > > > > > >> > > > > Ref #1: https://github.com/pedro-w/netbeans/releases/tag/v2 > >> > > > > Ref #2: Currently > >> > > > > > >> > > >> https://netbeans.osuosl.org/binaries/45225DCFC94A9782419E95C70957822A0C44612D-profiler-external-binaries-8.2.zip > >> > > > > as per the file > >> > > > > > >> > > >> https://github.com/apache/netbeans/blob/master/profiler/lib.profiler/external/binaries-list > >> > . > >> > > > > > >> > > > > On Wed, Jan 13, 2021 at 3:00 AM Ernie Rael <err...@raelity.com > >> > (mailto:err...@raelity.com)> wrote: > >> > > > > > This thread is about /Profiling on 64bit Windows/. With hopes to > >> > find a > >> > > > > > short term solution; quick and dirty if nothing more general is > >> > available. > >> > > > > > > >> > > > > > This subject has been avoided for years often time by hijacking > >> and > >> > > > > > starting to talk about the big picture issue. Which of course is > >> > > > > > important and includes important issues and information. Some of > >> > this > >> > > > > > may have been in various discussions over the past years. (I > >> don't > >> > know) > >> > > > > > Maybe there's an issue open about it somewhere, where you could > >> > > > > > contribute; or start a new thread. > >> > > > > > > >> > > > > > Where relevant, it would be great to reference those big picture > >> > here. > >> > > > > > But not avoid the issue at hand. > >> > > > > > > >> > > > > > Apologies (and thanks for the space) for blowing off steam. > >> > > > > > -ernie > >> > > > > > > >> > > > > > On 1/12/2021 12:13 AM, Lars Bruun-Hansen wrote: > >> > > > > > > I think we as a community need to fix the "native builds" > >> issue > >> > very > >> > > > > > > soon. PRs are piling up. > >> > > > > > > > >> > > > > > > This current email thread is not the only one. Over the past 2 > >> > years > >> > > > > > > fixes have been made (PRs merged) to native binaries code in > >> the > >> > areas > >> > > > > > > of > >> > > > > > > > >> > > > > > > * Native Launchers > >> > > > > > > * NBI (installers) > >> > > > > > > * Profiler launcher > >> > > > > > > > >> > > > > > > The problem is the same in all cases: The´source code is fixed > >> > but > >> > > > > > > there's never a new binary being build hence nothing really > >> > happens. > >> > > > > > > > >> > > > > > > These binaries were never re-build in an Apache NetBeans > >> > setting. They > >> > > > > > > were simply copied from their old Oracle-era location to > >> > > > > > > https://netbeans.osuosl.org/binaries from where they are > >> picked > >> > up by > >> > > > > > > the Apache NetBeans build process but labelled as something > >> > which has > >> > > > > > > a CDDL license. This is legally correct but is a limbo state: > >> The > >> > > > > > > source code is (now) owned by ASF but for reason of lack of > >> time > >> > we > >> > > > > > > are still using the pre-ASF binaries. I can 100% understand > >> why > >> > such > >> > > > > > > "shortcut" was made and salute those who made it happen. > >> > > > > > > > >> > > > > > > However, obviously it is sub-optimal that parts of the > >> NetBeans > >> > > > > > > Platform and IDE distribution effectively cannot be bug-fixed. > >> > > > > > > > >> > > > > > > The right solution IMHO is that those parts of the Apache > >> > NetBeans > >> > > > > > > universe would be a first-class citizen in the general Jenkins > >> > build > >> > > > > > > pipeline for the Platform and IDE distribution. By > >> first-class I > >> > mean > >> > > > > > > that the binary is automatically re-build whenever the source > >> > code > >> > > > > > > changes. This would require Jenkins build slaves for Linux, > >> > MacOS and > >> > > > > > > Windows (which I understand that ASF indeed has) and it would > >> > probably > >> > > > > > > require quite a bit of setup work to get right. Apache > >> NetBeans > >> > is a > >> > > > > > > Java-oriented community so knowledge about building C/C++ > >> > binaries is > >> > > > > > > limited which is probably why nobody so far is stepping > >> forward. > >> > As > >> > > > > > > Ernie suggests there may be a short term solution too which is > >> > more > >> > > > > > > quick-and-dirty i.e. someone would re-build locally, verify > >> the > >> > > > > > > binaries and then upload as one-shot fix. However it is not > >> > always > >> > > > > > > that simple. The binaries are distributed as one ZIP. If one > >> > decides > >> > > > > > > to only re-build the binary for the Windows platform then this > >> > current > >> > > > > > > packaging breaks down because then you have one binary in the > >> ZIP > >> > > > > > > which is ASF and the others which are CCDL and no longer build > >> > from > >> > > > > > > the same revision in the VCS. It would be a mess. Therefore, > >> in > >> > many > >> > > > > > > ways, I'm not sure the quick-and-dirty solution really exists. > >> > > > > > > > >> > > > > > > Is this a correct status ? > >> > > > > > > > >> > > > > > > Lars > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tue, Jan 12, 2021 at 2:07 AM Eirik Bakke < > >> eba...@ultorg.com > >> > (mailto:eba...@ultorg.com)> wrote: > >> > > > > > > > Build issues aside, I think the code in the PR actually > >> needs > >> > at least a cursory review from someone else than the original author. > >> > Others, like myself, have passed by and tested the fix or left other > >> > comments, but I'm not sure anyone sat down and reviewed it all in one > >> > sitting. > >> > > > > > > > > >> > > > > > > > ( https://github.com/apache/netbeans/pull/2021 ) > >> > > > > > > > > >> > > > > > > > -- Eirik > >> > > > > > > > > >> > > > > > > > -----Original Message----- > >> > > > > > > > From: Ernie Rael <err...@raelity.com (mailto: > >> > err...@raelity.com)> > >> > > > > > > > Sent: Monday, January 11, 2021 12:34 PM > >> > > > > > > > To: dev@netbeans.apache.org (mailto:dev@netbeans.apache.org > >> ) > >> > > > > > > > Subject: Re: Profiling on 64bit Windows > >> > > > > > > > > >> > > > > > > > On 1/11/2021 6:53 AM, Geertjan Wielenga wrote: > >> > > > > > > > > What needs to be done? Can you provide a PR? > >> > > > > > > > Sadly no. I suspect that someone who knows their way around > >> a > >> > distribution would have that knowledge. I could be wrong, but I suspect > >> > it's simple. > >> > > > > > > > > >> > > > > > > > If it is simple and it hasn't been done in two years, I'd > >> > guess it's more a lack of will or willingness to just check in the > >> binaries. > >> > > > > > > > > >> > > > > > > > -ernie > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Gj > >> > > > > > > > > > >> > > > > > > > > On Mon, 11 Jan 2021 at 15:33, Ernie Rael < > >> err...@raelity.com > >> > (mailto:err...@raelity.com)> wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi all, > >> > > > > > > > > > > >> > > > > > > > > > There's been a fix for the 64 bit windows profiling > >> > problem for over > >> > > > > > > > > > two years. > >> > > > > > > > > > > >> > > > > > > > > > > >> > https://github.com/pedro-w/netbeans/releases/tag/v0.1-alpha > >> > > > > > > > > > https://issues.apache.org/jira/browse/NETBEANS-1428 > >> > > > > > > > > > > >> > > > > > > > > > AFAICT, it hasn't been integrated. There's some > >> discussion > >> > every few > >> > > > > > > > > > months or so on how to get the binaries to be > >> > automatically built as > >> > > > > > > > > > part of the general build. An answer hasn't been found, > >> so > >> > the > >> > > > > > > > > > problem remains. These binaries have been around and in > >> > use for quite a while. > >> > > > > > > > > > > >> > > > > > > > > > This issue seems stuck in the /Perfectionism is the > >> enemy > >> > of > >> > > > > > > > > > progress/ trap. > >> > > > > > > > > > > >> > > > > > > > > > Can't the binaries be checked in and included, while the > >> > > > > > > > > > investigation for a solution proceeds? > >> > > > > > > > > > > >> > > > > > > > > > -ernie > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > --------------------------------------------------------------------- > >> > > > > > > > > > To unsubscribe, e-mail: > >> > dev-unsubscr...@netbeans.apache.org (mailto: > >> > dev-unsubscr...@netbeans.apache.org) > >> > > > > > > > > > For additional commands, e-mail: > >> > dev-h...@netbeans.apache.org (mailto:dev-h...@netbeans.apache.org) > >> > > > > > > > > > > >> > > > > > > > > > For further information about the NetBeans mailing > >> lists, > >> > visit: > >> > > > > > > > > > > >> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> --- > >> > > -- > > Eric Bresie > > ebre...@gmail.com > > > -- > Eric Bresie > ebre...@gmail.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org For additional commands, e-mail: dev-h...@netbeans.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
