Bug#894476: RCC: provide option to encode EPOCH timestamp
Hi Thomas, > Maybe the solution is then not in rcc but in whatever generate the files > that the qrc that rcc processes refer to. For instance in the case of > ultracopier lrelease could have a mean if generating .qm files with the > same modified timestamp as the .ts file it processes That sounds workable.. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#894476: RCC: provide option to encode EPOCH timestamp
On April 3, 2018 10:15:42 PM GMT+01:00, "Lisandro Damián Nicanor Pérez Meyer" wrote: >El mar., 3 de abr. de 2018 16:42, Sune Vuorela >escribió: > >> On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote: >> > I'm not /entirely/ sure what the difference is as I'm not in the >> > Qt/RCC world too often these days (alas !), but why just not use >> > SOURCE_DATE_EPOCH *iff* it is exported? >> > >> > Normal systems simply do not have this envvar set, so there is >really >> > no danger of it leaking elsewhere or causing unintended >side-effects. >> >> We don't know how application users are using this. >> >> The following is some theoretical example, that I'm quite sure could >be >> used >> in some real world applications: >> >> QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in >the >> ancient past >> QFileInfo systemfile("/usr/share/foo/data.xml"); >> >> if (systemfile.lastModified() > embeddedFile.lastModified()) >> { >>data = readFromFile(systemFile); >> } >> else >> { >>data = readFromFile(embeddedFile); >> } >> >> If S_D_E gets used, rather than the data.xml modified date in the >source, >> this >> will end up using the wrong file if S_D_E is newer than the system >copy of >> the >> file. >> >> This is not a unused databit, but a fully available piece of metadata >for >> the >> files. > > >I'm afraid Sune is right here, it might be used that way. Questionable? >Sure thing, but still valid code. > >But after all the same can be said from embedding translations into the >binary itself. > >So yes, we will need help from upstream with this one. > >> >> Maybe the solution is then not in rcc but in whatever generate the files that the qrc that rcc processes refer to. For instance in the case of ultracopier lrelease could have a mean if generating .qm files with the same modified timestamp as the .ts file it processes. This would guarantee stable output for a stable source and this later on rcc would also generate stable output. Thoughts? Best regards, Thomas
Bug#894476: RCC: provide option to encode EPOCH timestamp
El mar., 3 de abr. de 2018 16:42, Sune Vuorela escribió: > On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote: > > I'm not /entirely/ sure what the difference is as I'm not in the > > Qt/RCC world too often these days (alas !), but why just not use > > SOURCE_DATE_EPOCH *iff* it is exported? > > > > Normal systems simply do not have this envvar set, so there is really > > no danger of it leaking elsewhere or causing unintended side-effects. > > We don't know how application users are using this. > > The following is some theoretical example, that I'm quite sure could be > used > in some real world applications: > > QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the > ancient past > QFileInfo systemfile("/usr/share/foo/data.xml"); > > if (systemfile.lastModified() > embeddedFile.lastModified()) > { >data = readFromFile(systemFile); > } > else > { >data = readFromFile(embeddedFile); > } > > If S_D_E gets used, rather than the data.xml modified date in the source, > this > will end up using the wrong file if S_D_E is newer than the system copy of > the > file. > > This is not a unused databit, but a fully available piece of metadata for > the > files. I'm afraid Sune is right here, it might be used that way. Questionable? Sure thing, but still valid code. But after all the same can be said from embedding translations into the binary itself. So yes, we will need help from upstream with this one. > >
Bug#894476: RCC: provide option to encode EPOCH timestamp
On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote: > I'm not /entirely/ sure what the difference is as I'm not in the > Qt/RCC world too often these days (alas !), but why just not use > SOURCE_DATE_EPOCH *iff* it is exported? > > Normal systems simply do not have this envvar set, so there is really > no danger of it leaking elsewhere or causing unintended side-effects. We don't know how application users are using this. The following is some theoretical example, that I'm quite sure could be used in some real world applications: QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the ancient past QFileInfo systemfile("/usr/share/foo/data.xml"); if (systemfile.lastModified() > embeddedFile.lastModified()) { data = readFromFile(systemFile); } else { data = readFromFile(embeddedFile); } If S_D_E gets used, rather than the data.xml modified date in the source, this will end up using the wrong file if S_D_E is newer than the system copy of the file. This is not a unused databit, but a fully available piece of metadata for the files. /Sune -- I didn’t stop pretending when I became an adult, it’s just that when I was a kid I was pretending that I fit into the rules and structures of this world. And now that I’m an adult, I pretend that those rules and structures exist. - zefrank
Bug#894476: RCC: provide option to encode EPOCH timestamp
On April 3, 2018 8:20:22 PM GMT+01:00, Sune Vuorela wrote: >On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote: >> Hi Sune! >> >> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under >normal >> > circumstances >> >> Can you elaborate on what you mean by "normal circumstances"? :) > >"normal circumstances" is rcc on a source file, as opposed to an >autogenerated >file. > >> How about e use S_D_E if it is setup/exported, otherwise use the >mtime >> of the file as before? > >I think that using S_D_E only makes sense if rcc is run on an >autogenerated >file, but I do think that most rcc runs is run on existing source >files. > > >I don't have a good idea to differentiate those. > >/Sune >-- >I didn’t stop pretending when I became an adult, it’s just that when I >was a >kid I was pretending that I fit into the rules and structures of this >world. >And now that I’m an adult, I pretend that those rules and structures >exist. > - zefrank One small clarification: in my case rcc *is* run on a nongenerated resource file. It's some of the files that the resource file list that are generated and whose timestamp end up in the cpp file generated by rcc. In other hand: foo.qrc mentions bar.qm which is generated from bar.ts. rcc foo.qrc generates resource_bar.cpp which contains constant data that encodes bar.qm timestamp and this create different resource_bar.o at every build. Best regards, Thomas
Bug#894476: RCC: provide option to encode EPOCH timestamp
Hi Sune, > "normal circumstances" is rcc on a source file, as opposed > to an autogenerated file I'm not /entirely/ sure what the difference is as I'm not in the Qt/RCC world too often these days (alas !), but why just not use SOURCE_DATE_EPOCH *iff* it is exported? Normal systems simply do not have this envvar set, so there is really no danger of it leaking elsewhere or causing unintended side-effects. It would also have the benefit of not having to differentiate between the two... :) Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#894476: RCC: provide option to encode EPOCH timestamp
On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote: > Hi Sune! > > > I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal > > circumstances > > Can you elaborate on what you mean by "normal circumstances"? :) "normal circumstances" is rcc on a source file, as opposed to an autogenerated file. > How about e use S_D_E if it is setup/exported, otherwise use the mtime > of the file as before? I think that using S_D_E only makes sense if rcc is run on an autogenerated file, but I do think that most rcc runs is run on existing source files. I don't have a good idea to differentiate those. /Sune -- I didn’t stop pretending when I became an adult, it’s just that when I was a kid I was pretending that I fit into the rules and structures of this world. And now that I’m an adult, I pretend that those rules and structures exist. - zefrank
Bug#894476: RCC: provide option to encode EPOCH timestamp
Hi Sune! > I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal > circumstances Can you elaborate on what you mean by "normal circumstances"? :) How about e use S_D_E if it is setup/exported, otherwise use the mtime of the file as before? (This is the pattern/logic used extensively elsewhere in toolchains for this very exact purpose; it would not be unique to rcc) Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#894476: RCC: provide option to encode EPOCH timestamp
On Tuesday, April 3, 2018 8:46:51 PM CEST Thomas Preud'homme wrote: > The problematic files are Qt message files (ie .qm files) generated at build > time via lrelease from translation files (ie .ts files). Therefore two Right. I didn't think about autogenerated files during build. > I think it would be nice if there was a way for > rcc to avoid doing that manually. I agree with Chris that honouring > SOURCE_DATE_EPOCH in rcc would be a nice solution. I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal circumstances, given that I do think that having the actual timestamp of the non-generated files makes sense. I have reached out to upstream about it. /Sune -- I didn’t stop pretending when I became an adult, it’s just that when I was a kid I was pretending that I fit into the rules and structures of this world. And now that I’m an adult, I pretend that those rules and structures exist. - zefrank
Bug#894476: RCC: provide option to encode EPOCH timestamp
On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote: > > While investigating ultracopier's lack of build reproducibility, I found > > out that rcc encodes the timestamp of the files the QRC file being > > compiled references I don't actually see why this should be a problem. If input changes, output changes. I do think that using touch(1) on the input should allow different output. It is quite easy to reproduce: qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../ qimage.qrc -o 1 qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../ qimage.qrc -o 2 qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../ qimage.qrc -o 3 Gives same output. Even adding in touch ../qimage.qrc keeps the same output. qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../ qimage.qrc -o 4 touch ../images/image.bmp qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../ qimage.qrc -o 5 is needed to get different output 14ef6dae8e4992ce907948c1c4af272b 1 14ef6dae8e4992ce907948c1c4af272b 2 14ef6dae8e4992ce907948c1c4af272b 3 14ef6dae8e4992ce907948c1c4af272b 4 54c6f8c09a347955ae2f36e68bbd2539 5 So. What touches the files? /Sune -- I didn’t stop pretending when I became an adult, it’s just that when I was a kid I was pretending that I fit into the rules and structures of this world. And now that I’m an adult, I pretend that those rules and structures exist. - zefrank
Bug#894476: RCC: provide option to encode EPOCH timestamp
On April 3, 2018 7:25:03 PM GMT+01:00, Sune Vuorela wrote: >On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote: >> > While investigating ultracopier's lack of build reproducibility, I >found >> > out that rcc encodes the timestamp of the files the QRC file being >> > compiled references > >I don't actually see why this should be a problem. If input changes, >output >changes. >I do think that using touch(1) on the input should allow different >output. > >It is quite easy to reproduce: > >qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version >2 ../ >qimage.qrc -o 1 >qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version >2 ../ >qimage.qrc -o 2 >qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version >2 ../ >qimage.qrc -o 3 > >Gives same output. >Even adding in touch ../qimage.qrc keeps the same output. > >qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version >2 ../ >qimage.qrc -o 4 > >touch ../images/image.bmp > >qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version >2 ../ >qimage.qrc -o 5 > >is needed to get different output > >14ef6dae8e4992ce907948c1c4af272b 1 >14ef6dae8e4992ce907948c1c4af272b 2 >14ef6dae8e4992ce907948c1c4af272b 3 >14ef6dae8e4992ce907948c1c4af272b 4 >54c6f8c09a347955ae2f36e68bbd2539 5 > > >So. What touches the files? > >/Sune >-- >I didn’t stop pretending when I became an adult, it’s just that when I >was a >kid I was pretending that I fit into the rules and structures of this >world. >And now that I’m an adult, I pretend that those rules and structures >exist. > - zefrank Hi Sune, The problematic files are Qt message files (ie .qm files) generated at build time via lrelease from translation files (ie .ts files). Therefore two different builds will generate those .qm files at different times which will end up with different cpp files generated by rcc. Currently I'm working around it by setting the modified time of those .qm files to EPOCH after they are generated. I think it would be nice if there was a way for rcc to avoid doing that manually. I agree with Chris that honouring SOURCE_DATE_EPOCH in rcc would be a nice solution. Best regards, Thomas
Bug#894476: RCC: provide option to encode EPOCH timestamp
Thanks for the bug! And sorry for the to posting :-( I'll try to take a look into this. El sáb., 31 de mar. de 2018 07:09, Chris Lamb escribió: > Hi, > > > While investigating ultracopier's lack of build reproducibility, I found > > out that rcc encodes the timestamp of the files the QRC file being > > compiled references > > Indeed, we've been tracking this for a while in the Reproducible Builds > project [0] but, without a patch, we did not wish to bother you with a > bug. :) > > It is affecting at least 4 or 5 other packages. I had previously tracked > it down to [1]. > > Regarding the solution, I think I would actually prefer to see something > consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a > specific timestamp. > > [0] https://reproducible-builds.org/ > [1] > https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212 > [2] https://reproducible-builds.org/specs/source-date-epoch/ > > > Best wishes, > > -- > ,''`. > : :' : Chris Lamb > `. `'` la...@debian.org / chris-lamb.co.uk >`- > >
Bug#894476: RCC: provide option to encode EPOCH timestamp
Hi, > While investigating ultracopier's lack of build reproducibility, I found > out that rcc encodes the timestamp of the files the QRC file being > compiled references Indeed, we've been tracking this for a while in the Reproducible Builds project [0] but, without a patch, we did not wish to bother you with a bug. :) It is affecting at least 4 or 5 other packages. I had previously tracked it down to [1]. Regarding the solution, I think I would actually prefer to see something consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a specific timestamp. [0] https://reproducible-builds.org/ [1] https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212 [2] https://reproducible-builds.org/specs/source-date-epoch/ Best wishes, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#894476: RCC: provide option to encode EPOCH timestamp
Package: qtbase5-dev-tools Version: 5.9.2+dfsg-12 Severity: wishlist Tags: upstream User: reproducible-bui...@lists.alioth.debian.org Usertags: timestamps toolchain Hi, While investigating ultracopier's lack of build reproducibility, I found out that rcc encodes the timestamp of the files the QRC file being compiled references (see end of RCCFileInfo::writeDataInfo()). This becomes a problem for generated files because the output of rcc is then different in 2 different builds. It would be nice if rcc had an option to encode a stable timestamp, eg. EPOCH. Best regards, Thomas -- System Information: Debian Release: buster/sid APT prefers unstable APT policy: (900, 'unstable'), (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'testing'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.15.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR:fr (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages qtbase5-dev-tools depends on: ii libc62.27-2 ii libgcc1 1:8-20180321-1 ii libqt5core5a [qtbase-abi-5-9-2] 5.9.2+dfsg-12 ii libqt5dbus5 5.9.2+dfsg-12 ii libstdc++6 8-20180321-1 ii perl 5.26.1-5 ii qtchooser64-ga1b6736-5 ii zlib1g 1:1.2.8.dfsg-5 qtbase5-dev-tools recommends no packages. qtbase5-dev-tools suggests no packages. -- no debconf information