Re: [pacman-dev] [PATCH 3/4] makepkg: unify source file times for improved build reproducibility
On 04/21/2017 07:08 AM, Allan McRae wrote: > On 21/04/17 13:36, Eli Schwartz wrote: >> On 04/20/2017 11:01 PM, Allan McRae wrote: >>> I am probably moving this to after source extraction/prepare() running, >>> so it can be skipped with --noextract. >> >> But --noextract depends on your having at some point previously run >> --nobuild, in order to pull updated sources, re-patch any patches, etc. >> which I still don't want to do manually outside of makepkg... I don't >> see why makepkg should start breaking things for me. >> >> How about instead we guard it with >> BUILDENV+=(fix_everyone_elses_SOURCE_DATE_EPOCH_stuff) >> >> Or better yet, just file bugs against whatever upstream build >> system/programming language/source code is determined to sneakily embed >> source code modification times into generated files, and call it a day? >> > > Adding list back - any further off-list replies will be completely ignored. > > The reproducible builds people will provide details, but it appears > pyo/pyc do this. > > A > Unfortunately it won't be possible to fully avoid such uniform modifications times of input files to get reproducible builds. The most known dominator there is indeed python and its way to determine the produced artifacts state compared to the source code. Make does use the same information by setting the modification time of the produced artifacts to compare them. I agree make is in that detail a nicer way as that info in contained outside of the content of the produced artifacts itself, however there are external design decisions that we need to accept that they exist and won't change. So as both use cases (reproducible and incremental builds) seem to be valid and wanted features, let's be cooperative and see how we can make both separately work to make all of us happy. :) As I don't really see how both can work at the same time: one way that could do its job is an option like --incremental that would not do any timestamp unification. This could be used in the case of a incremental build of a VCS package in a non cleaned environment. I think that is not too much of a hassle to do in case someone wants to build an VCS package incrementally with previous object files? cheers, Levente
Re: [pacman-dev] [PATCH 3/4] makepkg: unify source file times for improved build reproducibility
On 05/02/17 at 09:30pm, Levente Polyak wrote: > On 04/21/2017 07:08 AM, Allan McRae wrote: > > On 21/04/17 13:36, Eli Schwartz wrote: > >> On 04/20/2017 11:01 PM, Allan McRae wrote: > >>> I am probably moving this to after source extraction/prepare() running, > >>> so it can be skipped with --noextract. > >> > >> But --noextract depends on your having at some point previously run > >> --nobuild, in order to pull updated sources, re-patch any patches, etc. > >> which I still don't want to do manually outside of makepkg... I don't > >> see why makepkg should start breaking things for me. > >> > >> How about instead we guard it with > >> BUILDENV+=(fix_everyone_elses_SOURCE_DATE_EPOCH_stuff) > >> > >> Or better yet, just file bugs against whatever upstream build > >> system/programming language/source code is determined to sneakily embed > >> source code modification times into generated files, and call it a day? > >> > > > > Adding list back - any further off-list replies will be completely ignored. > > > > The reproducible builds people will provide details, but it appears > > pyo/pyc do this. > > > > A > > Unfortunately it won't be possible to fully avoid such uniform > modifications times of input files to get reproducible builds. The most > known dominator there is indeed python and its way to determine the > produced artifacts state compared to the source code. Make does use the > same information by setting the modification time of the produced > artifacts to compare them. > I agree make is in that detail a nicer way as that info in contained > outside of the content of the produced artifacts itself, however there > are external design decisions that we need to accept that they exist and > won't change. > > So as both use cases (reproducible and incremental builds) seem to be > valid and wanted features, let's be cooperative and see how we can make > both separately work to make all of us happy. :) > > As I don't really see how both can work at the same time: one way that > could do its job is an option like --incremental that would not do any > timestamp unification. This could be used in the case of a incremental > build of a VCS package in a non cleaned environment. I think that is not > too much of a hassle to do in case someone wants to build an VCS package > incrementally with previous object files? Why is setting the modification timestamp necessary? makepkg should be preserving the modification timestamps of files when it extracts them from archives. So two builds using the same source tarballs should already have files with the same timestamps. When is this not sufficient? apg
Re: [pacman-dev] [PATCH 3/4] makepkg: unify source file times for improved build reproducibility
On 05/02/2017 10:09 PM, Andrew Gregory wrote: > On 05/02/17 at 09:30pm, Levente Polyak wrote: >> On 04/21/2017 07:08 AM, Allan McRae wrote: >>> On 21/04/17 13:36, Eli Schwartz wrote: On 04/20/2017 11:01 PM, Allan McRae wrote: > I am probably moving this to after source extraction/prepare() running, > so it can be skipped with --noextract. But --noextract depends on your having at some point previously run --nobuild, in order to pull updated sources, re-patch any patches, etc. which I still don't want to do manually outside of makepkg... I don't see why makepkg should start breaking things for me. How about instead we guard it with BUILDENV+=(fix_everyone_elses_SOURCE_DATE_EPOCH_stuff) Or better yet, just file bugs against whatever upstream build system/programming language/source code is determined to sneakily embed source code modification times into generated files, and call it a day? >>> >>> Adding list back - any further off-list replies will be completely ignored. >>> >>> The reproducible builds people will provide details, but it appears >>> pyo/pyc do this. >>> >>> A >> >> Unfortunately it won't be possible to fully avoid such uniform >> modifications times of input files to get reproducible builds. The most >> known dominator there is indeed python and its way to determine the >> produced artifacts state compared to the source code. Make does use the >> same information by setting the modification time of the produced >> artifacts to compare them. >> I agree make is in that detail a nicer way as that info in contained >> outside of the content of the produced artifacts itself, however there >> are external design decisions that we need to accept that they exist and >> won't change. >> >> So as both use cases (reproducible and incremental builds) seem to be >> valid and wanted features, let's be cooperative and see how we can make >> both separately work to make all of us happy. :) >> >> As I don't really see how both can work at the same time: one way that >> could do its job is an option like --incremental that would not do any >> timestamp unification. This could be used in the case of a incremental >> build of a VCS package in a non cleaned environment. I think that is not >> too much of a hassle to do in case someone wants to build an VCS package >> incrementally with previous object files? > > Why is setting the modification timestamp necessary? makepkg should > be preserving the modification timestamps of files when it extracts > them from archives. So two builds using the same source tarballs > should already have files with the same timestamps. When is this not > sufficient? > > apg > Reproducibility is not something that should be exclusive to archives providing such anyway. There is an increasing interest in builds from repository checkout. On top, prepare() modifies such so those should be adjusted too before the build starts. cheers, Levente
