Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
Source: ffmpeg Followup-For: Bug #985187 X-Debbugs-Cc: vagr...@reproducible-builds.org Control: fixed -1 7:6.1-1 Control: close -1 This package is not yet building reproducibly[1], but the build-path embed correctly identified by this bugreport as a contributing factor has been removed[2] from ffmpeg v6.1 onwards and no longer adds variance to built packages. [1] - https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html [2] - https://git.ffmpeg.org/gitweb/ffmpeg.git/blobdiff/7cfd7e4af4b1c0f280f0c64a8088d362a2917e79:/tools/cl2c..88e2cca3dbd1f509982778804ba2463058bb729a:/tools/source2c
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
On Tuesday, 3 January 2023 19:06:41 CET Vagrant Cascadian wrote: > Looks like some new source of non-determinism was added in 5.0.x, you > can see bookworm (which does not test build paths) was reproducible > until 5.0.x started getting tested in bookworm in June of 2022: > > https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffosco > pe-results/ffmpeg.html > > Fixing the build path issue mentioned in this patch may dramatically > reduce the size of the diffoscope output from ~71MB to ~29KB ... so > might be nice to apply still even if it does not fix all the > reproducibility issues. I'll take another stab at it... I just added "SALSA_CI_REPROTEST_ENABLE_DIFFOSCOPE: 1" to my salsa-ci.yml file which I assume means it'll now also produce a diffoscope output. Feel free to use it if it helps: https://salsa.debian.org/diederik/ffmpeg/-/pipelines https://salsa.debian.org/diederik/ffmpeg/-/commits/fix-reprotest-issue Cheers, Diederik signature.asc Description: This is a digitally signed message part.
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
On 2023-01-03, Diederik de Haas wrote: > On 14 Apr 2022 15:45:26 -0700 Vagrant Cascadian builds.org> wrote: >> On 2021-08-08, Sebastian Ramacher wrote: >> > On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote: >> >> Source: ffmpeg >> >> Severity: normal >> >> Tags: patch >> >> User: reproducible-bui...@lists.alioth.debian.org >> >> >> >> The build path is embedded in various files generated with tools/cl2c: >> >> >> >> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ >> >> diffoscope-results/ffmpeg.html > > FWIW, in https://salsa.debian.org/diederik/ffmpeg/-/pipelines/478274 I added > your patch, but the 'reprotest' job still failed. > I don't know how to read diffoscope's result (and currently don't have time > to > learn it), but it looks like more is needed to make it reproducible. > Happy to add more patches to my branch and if 'reprotest' then succeeds, > you're free to add a "Tested-By: " tag with my name+email Looks like some new source of non-determinism was added in 5.0.x, you can see bookworm (which does not test build paths) was reproducible until 5.0.x started getting tested in bookworm in June of 2022: https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/ffmpeg.html Fixing the build path issue mentioned in this patch may dramatically reduce the size of the diffoscope output from ~71MB to ~29KB ... so might be nice to apply still even if it does not fix all the reproducibility issues. I'll take another stab at it... >> >> The attached patch fixes this by patching tools/cl2c to use a basename >> >> in the generated file rather than the full path. >> > >> > As this patch touches upstream's build system, please submit it >> > upstream: >> >> Did so, haven't really heard anything back: >> >> https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/ >> >> Any suggestions? > > I have no particular knowledge wrt ffmpeg-devel, but with the kernel patches > sometimes fall through the cracks and the best way is just to resend them. > > I did notice that the patch itself looks better then the email I saw on the > 'Forwarded' link. I _think_ that with 'git send-email' it would look cleaner. > > I would suggest to slightly reword the commit message so that it doesn't > contain "Without this patch". Maybe it's just me, but if the secondary commit > message starts with "To make builds reproducible ..." that would heighten my > interest in the patch. 'But' I really like the reproducible builds effort :-) > > And possibly update the reference to ffmpeg-4.3.2 to point to 5.1 or > something. > And I'd replace "Originally submitted to Debian as:\n\n https://b.d.o/985187; > with a "Link:" tag. Thanks for the suggestions! live well, vagrant signature.asc Description: PGP signature
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
On 14 Apr 2022 15:45:26 -0700 Vagrant Cascadian wrote: > On 2021-08-08, Sebastian Ramacher wrote: > > On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote: > >> Source: ffmpeg > >> Severity: normal > >> Tags: patch > >> User: reproducible-bui...@lists.alioth.debian.org > >> > >> The build path is embedded in various files generated with tools/cl2c: > >> > >> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ > >> diffoscope-results/ffmpeg.html FWIW, in https://salsa.debian.org/diederik/ffmpeg/-/pipelines/478274 I added your patch, but the 'reprotest' job still failed. I don't know how to read diffoscope's result (and currently don't have time to learn it), but it looks like more is needed to make it reproducible. Happy to add more patches to my branch and if 'reprotest' then succeeds, you're free to add a "Tested-By: " tag with my name+email > >> The attached patch fixes this by patching tools/cl2c to use a basename > >> in the generated file rather than the full path. > > > > As this patch touches upstream's build system, please submit it > > upstream: > > Did so, haven't really heard anything back: > > https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/ > > Any suggestions? I have no particular knowledge wrt ffmpeg-devel, but with the kernel patches sometimes fall through the cracks and the best way is just to resend them. I did notice that the patch itself looks better then the email I saw on the 'Forwarded' link. I _think_ that with 'git send-email' it would look cleaner. I would suggest to slightly reword the commit message so that it doesn't contain "Without this patch". Maybe it's just me, but if the secondary commit message starts with "To make builds reproducible ..." that would heighten my interest in the patch. 'But' I really like the reproducible builds effort :-) And possibly update the reference to ffmpeg-4.3.2 to point to 5.1 or something. And I'd replace "Originally submitted to Debian as:\n\n https://b.d.o/985187; with a "Link:" tag. HTH, Diederik signature.asc Description: This is a digitally signed message part.
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
On 2021-08-08, Sebastian Ramacher wrote: > On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote: >> Source: ffmpeg >> Severity: normal >> Tags: patch >> User: reproducible-bui...@lists.alioth.debian.org >> Usertags: buildpath >> X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org >> >> The build path is embedded in various files generated with tools/cl2c: >> >> >> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html >> >> avgblur.o (shipped in libavfilter.a) >> >> #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl" >> vs. >> #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl" >> >> >> The attached patch fixes this by patching tools/cl2c to use a basename >> in the generated file rather than the full path. >> >> >> It builds fine with this patch applied, but someone more familiar with >> ffmpeg should ensure ffmpeg still functions correctly with the patch >> applied. > > As this patch touches upstream's build system, please submit it > upstream: > > https://ffmpeg.org/developer.html#Contributing Did so, haven't really heard anything back: https://patchwork.ffmpeg.org/project/ffmpeg/patch/87sfwyotmg.fsf@yucca/ Any suggestions? live well, vagrant signature.asc Description: PGP signature
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
Control: tags -1 upstream Hi Vagrant On 2021-03-13 20:05:47 -0800, Vagrant Cascadian wrote: > Source: ffmpeg > Severity: normal > Tags: patch > User: reproducible-bui...@lists.alioth.debian.org > Usertags: buildpath > X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org > > The build path is embedded in various files generated with tools/cl2c: > > > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html > > avgblur.o (shipped in libavfilter.a) > > #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl" > vs. > #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl" > > > The attached patch fixes this by patching tools/cl2c to use a basename > in the generated file rather than the full path. > > > It builds fine with this patch applied, but someone more familiar with > ffmpeg should ensure ffmpeg still functions correctly with the patch > applied. As this patch touches upstream's build system, please submit it upstream: https://ffmpeg.org/developer.html#Contributing Thanks Cheers > > > Thanks for maintaining ffmpeg! > > live well, > vagrant > From ec49e459797d6e112358566ae083c3bfae801023 Mon Sep 17 00:00:00 2001 > From: Vagrant Cascadian > Date: Sat, 13 Mar 2021 03:52:51 + > Subject: [PATCH] tools/cl2c: Strip full path from input file in embedded > output. > MIME-Version: 1.0 > Content-Type: text/plain; charset=UTF-8 > Content-Transfer-Encoding: 8bit > > Without this patch, the full build path gets embedded into various > binaries shipped in the package, for example, libavfilter.a contains > a references to: > > #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl" > > https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html > --- > tools/cl2c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/tools/cl2c b/tools/cl2c > index e3f92bab..48444e61 100755 > --- a/tools/cl2c > +++ b/tools/cl2c > @@ -23,11 +23,13 @@ input="$1" > output="$2" > > name=$(basename "$input" | sed 's/.cl$//') > +# Avoid embedded the build path, using only the basename of the input file. > +base_input=$(basename "$input") > > cat >$output < // Generated from $input > const char *ff_opencl_source_$name = > -"#line 1 \"$input\"\n" > +"#line 1 \"$base_input\"\n" > EOF > > # Convert \ to \\ and " to \", then add " to the start and end of the line. > -- > 2.20.1 > -- Sebastian Ramacher signature.asc Description: PGP signature
Bug#985187: ffmpeg: reproducible builds: Embeds build path in binaries generated with cl2c
Source: ffmpeg Severity: normal Tags: patch User: reproducible-bui...@lists.alioth.debian.org Usertags: buildpath X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org The build path is embedded in various files generated with tools/cl2c: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ffmpeg.html avgblur.o (shipped in libavfilter.a) #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl" vs. #line·1·"/build/2/ffmpeg-4.3.2/2nd/libavfilter/opencl/avgblur.cl" The attached patch fixes this by patching tools/cl2c to use a basename in the generated file rather than the full path. It builds fine with this patch applied, but someone more familiar with ffmpeg should ensure ffmpeg still functions correctly with the patch applied. Thanks for maintaining ffmpeg! live well, vagrant From ec49e459797d6e112358566ae083c3bfae801023 Mon Sep 17 00:00:00 2001 From: Vagrant Cascadian Date: Sat, 13 Mar 2021 03:52:51 + Subject: [PATCH] tools/cl2c: Strip full path from input file in embedded output. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Without this patch, the full build path gets embedded into various binaries shipped in the package, for example, libavfilter.a contains a references to: #line·1·"/build/1st/ffmpeg-4.3.2/libavfilter/opencl/avgblur.cl" https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html --- tools/cl2c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/cl2c b/tools/cl2c index e3f92bab..48444e61 100755 --- a/tools/cl2c +++ b/tools/cl2c @@ -23,11 +23,13 @@ input="$1" output="$2" name=$(basename "$input" | sed 's/.cl$//') +# Avoid embedded the build path, using only the basename of the input file. +base_input=$(basename "$input") cat >$output < signature.asc Description: PGP signature
