fdo#63315: sign windows binaries during build
Hello See https://bugs.freedesktop.org/show_bug.cgi?id=63315 From the Cloph's comment it seems the issue is fixed but with room for improvement. Should we close it or not ? ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
fdo#63315: sign windows binaries during build
Hello See https://bugs.freedesktop.org/show_bug.cgi?id=63315 From the Cloph's comment it seems the issue is fixed but with room for improvement. Should we close it or not ? Kind regards Mat M ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi, On Tue, May 6, 2014 at 1:56 AM, Mathias MICHEL wrote: > Hello > > See https://bugs.freedesktop.org/show_bug.cgi?id=63315 > From the Cloph's comment it seems the issue is fixed but with room for > improvement. > > Should we close it or not ? I'm interested in stability in this area and the current functionality is good enough for me, so I'd say let's close it. Best regards, Andras ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi *, On Tue, May 6, 2014 at 11:23 AM, Andras Timar wrote: > Hi, > > On Tue, May 6, 2014 at 1:56 AM, Mathias MICHEL > wrote: >> >> See https://bugs.freedesktop.org/show_bug.cgi?id=63315 >> From the Cloph's comment it seems the issue is fixed but with room for >> improvement. The room for improvement is in the general build, not specific to signing (i.e. copying files individually instead of using for example tar, or at least copy with multiple files at once/recursive copy, touching lots and lots of files,...) calling cp x times (creating a process that often), instead of once for x files is slow as hell on windows/cygwin. LO's patched version of make did make some touch and cp builtin functionality, but since gbuildification those rules don't match the majority of cases. Signing works automatically when specified as configure option. >> Should we close it or not ? > > I'm interested in stability in this area and the current functionality > is good enough for me, so I'd say let's close it. Yes, close it. Signing needs exclusive lock on the file, so signing will fail when the file is opened by another process - so this is another problem that would likely occur if the signing was done after each creation. (see http://cgit.freedesktop.org/libreoffice/core/commit/?id=ea391abb3f2eb0ffaa892f9d7437dcf33bda6440 that made singing depend on slowchecks being done, as otherwise signing would fail) ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi, On Tue, May 06, 2014 at 02:24:15PM +0200, Christian Lohmaier wrote: > Hi *, > > On Tue, May 6, 2014 at 11:23 AM, Andras Timar wrote: > > Hi, > > > > On Tue, May 6, 2014 at 1:56 AM, Mathias MICHEL > > wrote: > >> > >> See https://bugs.freedesktop.org/show_bug.cgi?id=63315 > >> From the Cloph's comment it seems the issue is fixed but with room for > >> improvement. > > The room for improvement is in the general build, not specific to > signing (i.e. copying files individually instead of using for example > tar, or at least copy with multiple files at once/recursive copy, > touching lots and lots of files,...) calling cp x times (creating a > process that often), instead of once for x files is slow as hell on > windows/cygwin. I would strongly object to any change that improves performance of the release builds at the expense of incremental builds, e.g., by dropping precise dependencies in order to copy more files at once. I really do not want to go back to the world of "compatible" vs. "incompatible" builds... > > LO's patched version of make did make some touch and cp builtin > functionality, but since gbuildification those rules don't match the > majority of cases. I do not really understand that... Does that mean there is something that actively prevents make to use the builtin functions? > > >> Should we close it or not ? > > > > I'm interested in stability in this area and the current functionality > > is good enough for me, so I'd say let's close it. > > Yes, close it. > > Signing needs exclusive lock on the file, so signing will fail when > the file is opened by another process - so this is another problem > that would likely occur if the signing was done after each creation. No, it would not. It is the current state, which does the signing at arbitrary time, that has this inherent problem. The proposal in that easy hack is to sign every library during linking; _nothing_ can be using the library at that point, unless there are missing dependencies. > (see > http://cgit.freedesktop.org/libreoffice/core/commit/?id=ea391abb3f2eb0ffaa892f9d7437dcf33bda6440 > that made singing depend on slowchecks being done, as otherwise > signing would fail) And I strongly suspect the next stop will be gallery creation, as gengal is internal tool which needs a zillion libraries... If we really want to continue to use this script for signing, IMHO it should be done as a separate step after build. D. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi David, *, On Wed, May 7, 2014 at 10:17 AM, David Tardon wrote: > On Tue, May 06, 2014 at 02:24:15PM +0200, Christian Lohmaier wrote: >> On Tue, May 6, 2014 at 11:23 AM, Andras Timar wrote: >> > On Tue, May 6, 2014 at 1:56 AM, Mathias MICHEL >> > wrote: >> >> >> >> See https://bugs.freedesktop.org/show_bug.cgi?id=63315 >> >> From the Cloph's comment it seems the issue is fixed but with room for >> >> improvement. >> >> The room for improvement is in the general build, not specific to >> signing (i.e. copying files individually instead of using for example >> tar, or at least copy with multiple files at once/recursive copy, >> touching lots and lots of files,...) calling cp x times (creating a >> process that often), instead of once for x files is slow as hell on >> windows/cygwin. > > I would strongly object to any change that improves performance of the > release builds at the expense of incremental builds, e.g., by dropping > precise dependencies in order to copy more files at once. Building SDK or Helpcontent or translations is not affected by incremental builds usually. And of course you won't lose dependencies by copying multiple stuff at once. You might waste some time doing i/o by shoving around files that weren't changed, but for stuff like sdk documentation that is created by doxygen that injects timestamps into every file (i.e. stuff is always changed) this doesn't matter from my POV. Same for help. Changing help means regenerating index, so everything will be looked up anyway, so having it in the filesystem cache doesn't hurt for linux. But copying this in one go will help the build on linux. > I really do > not want to go back to the world of "compatible" vs. "incompatible" > builds... That won't be the case. Copying more stuff at once for those will not mess with dependendencies. It might result in more disk-activity, but the areas where it matters most (sdk, help, translations) aren't touched in everyday development, aren't even enabled in everyday development builds. >> LO's patched version of make did make some touch and cp builtin >> functionality, but since gbuildification those rules don't match the >> majority of cases. > > I do not really understand that... Does that mean there is something > that actively prevents make to use the builtin functions? No, not preventing. The code that identifies candidates that could be handled by patched make just need adjustments to match. https://gerrit.libreoffice.org/gitweb?p=gnu-make-lo.git;a=blob;f=runbuiltin.c;hb=refs/heads/gnu-make-lo-3.82#l454 But this is only a workaround to hide the problem with the many processes. Removing the tons of "touch" calls would be real improvement. And I don't see a reason why a stuff need to depend on a touch-pseudo-target the created object itself. >[...] >> (see >> http://cgit.freedesktop.org/libreoffice/core/commit/?id=ea391abb3f2eb0ffaa892f9d7437dcf33bda6440 >> that made singing depend on slowchecks being done, as otherwise >> signing would fail) > > And I strongly suspect the next stop will be gallery creation, as gengal > is internal tool which needs a zillion libraries... Well, even if rhetorical: If that turns out to be a problem: Yes. Still preferred over spawning zillions of processes on windows. > If we really want to > continue to use this script for signing, IMHO it should be done as a > separate step after build. Which in effect it is. After slowcheck it is more or less separate. Having it after the build as separate /manual step is pointless, as then you woudn't only need to sign, but also rebuild the installsets. And building installests on windows also is slow. So I object to making it an "after make finished" thing. ciao Christian ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
David Tardon wrote: > If we really want to continue to use this script for signing, IMHO > it should be done as a separate step after build. > Might be a good idea anyway. Defined point in time when keys are used, less code involved, etc etc. -- Thorsten signature.asc Description: Digital signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi, On Wed, May 07, 2014 at 10:41:59AM +0200, Christian Lohmaier wrote: > Hi David, *, > > On Wed, May 7, 2014 at 10:17 AM, David Tardon wrote: > > On Tue, May 06, 2014 at 02:24:15PM +0200, Christian Lohmaier wrote: > >> On Tue, May 6, 2014 at 11:23 AM, Andras Timar wrote: > >> > On Tue, May 6, 2014 at 1:56 AM, Mathias MICHEL > >> > wrote: > >> >> > >> >> See https://bugs.freedesktop.org/show_bug.cgi?id=63315 > >> >> From the Cloph's comment it seems the issue is fixed but with room for > >> >> improvement. > >> > >> The room for improvement is in the general build, not specific to > >> signing (i.e. copying files individually instead of using for example > >> tar, or at least copy with multiple files at once/recursive copy, > >> touching lots and lots of files,...) calling cp x times (creating a > >> process that often), instead of once for x files is slow as hell on > >> windows/cygwin. > > > > I would strongly object to any change that improves performance of the > > release builds at the expense of incremental builds, e.g., by dropping > > precise dependencies in order to copy more files at once. > > Building SDK or Helpcontent or translations is not affected by > incremental builds usually. And of course you won't lose dependencies > by copying multiple stuff at once. You might waste some time doing i/o > by shoving around files that weren't changed, but for stuff like sdk > documentation that is created by doxygen that injects timestamps into > every file (i.e. stuff is always changed) this doesn't matter from my > POV. SDK docs are already delivered using gb_GeneratedPackage, which copies whole directories, because the file names are random. > Same for help. Changing help means regenerating index, so everything > will be looked up anyway, so having it in the filesystem cache doesn't > hurt for linux. But copying this in one go will help the build on > linux. Actually copying all the generated help files at once would even make the gbuild code simpler... I will add this to my TODO list. > > >[...] > >> (see > >> http://cgit.freedesktop.org/libreoffice/core/commit/?id=ea391abb3f2eb0ffaa892f9d7437dcf33bda6440 > >> that made singing depend on slowchecks being done, as otherwise > >> signing would fail) > > > > And I strongly suspect the next stop will be gallery creation, as gengal > > is internal tool which needs a zillion libraries... > > Well, even if rhetorical: If that turns out to be a problem: Yes. > Still preferred over spawning zillions of processes on windows. > > > If we really want to > > continue to use this script for signing, IMHO it should be done as a > > separate step after build. > > Which in effect it is. After slowcheck it is more or less separate. Except that it isn't. There are still tasks that are not needed by unit tests, so they could end up running in parallel with the signing. I have already mentioned gallery creation, but there is generation of help, UI file translations and possibly others. All of these use internal tools, therefore they use some internal libraries. The only way to ensure there will be no problems in this area is to move the signing to instsetoo_native and let the target depend on $(call gb_Postprocess_get_target,AllModulesButInstsetNative) Which puts it outside of the "normal" build anyway. > Having it after the build as separate /manual step is pointless, as > then you woudn't only need to sign, but also rebuild the installsets. > > And building installests on windows also is slow. So I object to > making it an "after make finished" thing. Well, there is no reason why the installsets must be created directly as part of build. There could be a separate make target for that. D. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
On 07/05/14 14:17, David Tardon wrote: > Hi, > > On Wed, May 07, 2014 at 10:41:59AM +0200, Christian Lohmaier wrote: >> Hi David, *, >> >> On Wed, May 7, 2014 at 10:17 AM, David Tardon wrote: >>> If we really want to >>> continue to use this script for signing, IMHO it should be done as a >>> separate step after build. >> >> Which in effect it is. After slowcheck it is more or less separate. > > Except that it isn't. There are still tasks that are not needed by unit > tests, so they could end up running in parallel with the signing. I have > already mentioned gallery creation, but there is generation of help, UI > file translations and possibly others. All of these use internal tools, > therefore they use some internal libraries. > > The only way to ensure there will be no problems in this area is to move > the signing to instsetoo_native and let the target depend on > $(call gb_Postprocess_get_target,AllModulesButInstsetNative) > Which puts it outside of the "normal" build anyway. why don't we do the signing in the same rule that links the library? that would avoid all potential dependency problems. (and gbuild does something similar with "sn.exe" for CLI "libraries" already) is the signing "interactive" or is there some other reason why that wouldn't be possible? ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: fdo#63315: sign windows binaries during build
Hi, On Thu, May 08, 2014 at 01:08:06PM +0200, Michael Stahl wrote: > On 07/05/14 14:17, David Tardon wrote: > > Hi, > > > > On Wed, May 07, 2014 at 10:41:59AM +0200, Christian Lohmaier wrote: > >> Hi David, *, > >> > >> On Wed, May 7, 2014 at 10:17 AM, David Tardon wrote: > >>> If we really want to > >>> continue to use this script for signing, IMHO it should be done as a > >>> separate step after build. > >> > >> Which in effect it is. After slowcheck it is more or less separate. > > > > Except that it isn't. There are still tasks that are not needed by unit > > tests, so they could end up running in parallel with the signing. I have > > already mentioned gallery creation, but there is generation of help, UI > > file translations and possibly others. All of these use internal tools, > > therefore they use some internal libraries. > > > > The only way to ensure there will be no problems in this area is to move > > the signing to instsetoo_native and let the target depend on > > $(call gb_Postprocess_get_target,AllModulesButInstsetNative) > > Which puts it outside of the "normal" build anyway. > > why don't we do the signing in the same rule that links the library? > that would avoid all potential dependency problems. (and gbuild does > something similar with "sn.exe" for CLI "libraries" already) > That was the topic of fdo#63315 =) Apparently I chose a poor summary for it... D. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
