Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
Lemures Lemniscati via Cygwin-apps writes: > But I feel uneasy about the usase of CYGPORT_TAR_CMD, because we must > specify 'tar' in it. It is a redundancy. It is also a way to do nefarious things, but there are already many more such places. > The following is a thought experiment to avoid this redundancy. […] As I said, you'll end up with configuration variables for everything except the actual command (which is found in PATH, so if I really wanted to mess with it I could do that anyway). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
Re: git repositories for cygwin packaging - please test
Ken Brown via Cygwin-apps writes:
>>> - take an inordinate amount of time to run (exceeding the resource limits)
>>
>> That is a problem that comes with CI I think and we didn't really have
>> had to consider so far. I have a few packages that I don't run tests on
>> by default because the test suite produces hangs or other unstable
>> behaviour, but that is dealt with in the src_test function itself.
>> If there are really resource hungry tests they usually need to be
>> enabled somewhere and one could skip those if the build runs on CI (how
>> to find that out?).
>
> Here's an example where Jon's suggestion would have been useful: While
> building php recently, I noticed that the test suite took forever. I
> would have been happy to have a way to tell the CI to skip the tests
> and avoid exceeding the resource limits. But I wouldn't want to do
> that by modifying src_test, because I would still want to run the
> tests locally.
I think I didn't express myself clearly enough. If you look in the
perl-IO-Tty cygport file you'll find this:
src_test() {
cd ${B}
[ "no" != "${CYGPORT_RUN_UNSTABLE_TESTS-no}" ] &&
cygtest || echo "Unstable test, set CYGPORT_RUN_UNSTABLE_TESTS to run."
}
This is clearly an exceptional deficiency (even if presumably caused by
some border case in Cygwin) that needs to be handled in the package and
not a general thing that should be in cygport.
But that's not to say that something like CYGPORT_RUN_EXPENSIVE_TESTS
could not exist and arguably it should even be directly provided by
cygport itself if it was. Now, "expensive" has multiple dimensions, so
I'd expect that variable to have a list of values to indicate which of
these is considered relevant (like time, memory, network). There'd need
to be a matching set of values / limits provided by the local or CI
configuration to enable / disable tests as appropriate.
CPU wise the CI seems to have a pretty beefy configuration, I've not run
into memory problems yet (but didn't check that specifically). So at
the moment the relevant limits would be total runtime and maybe network
activity (I don't think I have a package that exercises that). But then
again, this could change with each run and certainly will change over
time, so hard-coding these limits doesn't seem like a good idea.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: cmake upstream has merged cygwin patches
On Fri, 7 Aug 2020 21:51:00 +0200, Marco Atzeri via Cygwin-apps > On 07.08.2020 18:06, Lemures Lemniscati via Cygwin-apps wrote: > > Hi. > > > > The cygwin patches for cmake-3.17.3 have been merged into its upstream > > (though I should have discussed for it here). > > > > Merged by there merge-requests: > > https://gitlab.kitware.com/cmake/cmake/-/merge_requests/5092 > > https://gitlab.kitware.com/utils/kwsys/-/merge_requests/197 > > > > > > Merged patches for cmake-3.17.3-1 are > > > > perl-libs.patch > > 3.17.3-case-sensitivity.patch > > 3.17.3-opengl.patch > > 3.17.3-findtclsh.patch > > 3.17.3-cpuinfo.patch > > 3.17.3-cygwin-paths.patch > > https://github.com/matzeri/cygwin-pkg/tree/64864eb8f0d635ea44226e4b9a41a7ca59a1c7dd/cmake > > > > and > > > > 0001-3.17.3-cpuinfo_suppl.patch > > https://sourceware.org/pipermail/cygwin/2020-July/245563.html > > > > > > Regards, > > > > Lem > > > Thanks Lem, > > I am impressed, in the past we had much more discussion for having our > patches merged upstream. > > If you want to take over or just co-maintain, > let me know I'd like to just co-maintain :). Thank you. Regards, Lem
Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
On Fri, 07 Aug 2020 21:09:11 +0200, ASSI
> Lemures Lemniscati via Cygwin-apps writes:
> >> No, I was talking about things like XZ_OPT or ZSTD_CLEVEL.
> >
> > I think XZ_OPT or ZSTD_CLEVEL can be passed through environment.
> > And this should control xz or zstd..
> >
> > XZ_OPT="..." cygport foo.cygport package
> > ZSTD_CLEVEL="..." cygport foo.cygport package
> >
> > And cygport does not prevent from passing parameter through environment.
> > Maybe I understand you in a wrong way. Pardon me.
>
> Maybe if you'd gone back to the post I sent on June 14 when I originally
> offered the patch you'd have seen:
>
> >> This patch keeps the current defaults in place, but allows one to set up
> >> two environment variables to control how tar gets invoked during
> >> packaging. You could do that from either the cygport file or the
> >> .cygport.conf file:
> >>
> >> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar -I zstd --group nobody:65534
> >> --owner nobody:65534"
> >> CYGPORT_TAR_EXT=".tar.zst"
>
> I don't want to pollute the environment for any other command obviously…
> and really, I still think creating five new configuration variables to
> do what can be accomplished with two isn't a good trade-off.
>
> Factoring out the command invocation into a function probably is a net
> win, though -- so I'd keep that part. It'd probably be useful in some
> other instances to so a similar refactoring.
Thank you for your patience, Achim.
Now I've understood.
But I feel uneasy about the usase of CYGPORT_TAR_CMD, because we must
specify 'tar' in it. It is a redundancy.
The following is a thought experiment to avoid this redundancy.
Still, a work-around is to split it into two variables CYGPORT_TAR_ENV
and CYGPORT_TAR_OPTS, and do something like
(1) env ${CYGPORT_TAR_ENV} tar ${CYGPORT_TAR_OPTS} ...
or
(2) sh -c "env ${CYGPORT_TAR_ENV} tar ${CYGPORT_TAR_OPTS} ..."
The problem is that it is demanding more keystrokes when we use both of
them:
CYGPORT_TAR_ENV="..." CYGPORT_TAR_OPTS="..." cygport zzz.cygport package
But when specifying owner/group, we can do it simply,
CYGPORT_TAR_OPTS="--group=nobody:65534 --owner=nobody:65534" cygport
zzz.cygport package
or by invoking tar as (2) you would be able to specify compression level
and multi-threading like this:
CYGPORT_TAR_OPTS="-I 'zstd -19 -T2' --group=nobody:65534 --owner=nobody:65534"
cygport zzz.cygport package
This looks good enough to me.
And I've looked into the GNU tar source, and found that tar calls
a compression filter through execv 'sh' '-c' ...
https://git.savannah.gnu.org/cgit/tar.git/tree/src/system.c?h=release_1_32#n26
So, I guess it is not so bad to use 'sh' '-c' here in the script like
(2).
Regards,
Lem
Re: git repositories for cygwin packaging - please test
On 8/7/2020 3:42 PM, Achim Gratz wrote: Jon Turney writes: One problem I have noticed is that some packages have test suites (which are getting run via 'cygport test' invoking src_test()) which: - require lots of extra dependencies to run, or I currently subsume these in BUILD_REQUIRES as they are indeed required for a (complete) build. It'd be possible to split them out into a separate variable indeed, but that is yet another level of specificatrion that needs to be figured out if it should become useful. - don't succeed on Cygwin, or Any such tests I usually patch out or mark expected fail whichever seems easier. - take an inordinate amount of time to run (exceeding the resource limits) That is a problem that comes with CI I think and we didn't really have had to consider so far. I have a few packages that I don't run tests on by default because the test suite produces hangs or other unstable behaviour, but that is dealt with in the src_test function itself. If there are really resource hungry tests they usually need to be enabled somewhere and one could skip those if the build runs on CIm (how to find that out?). Here's an example where Jon's suggestion would have been useful: While building php recently, I noticed that the test suite took forever. I would have been happy to have a way to tell the CI to skip the tests and avoid exceeding the resource limits. But I wouldn't want to do that by modifying src_test, because I would still want to run the tests locally. Ken
Re: cmake upstream has merged cygwin patches
On 07.08.2020 18:06, Lemures Lemniscati via Cygwin-apps wrote: Hi. The cygwin patches for cmake-3.17.3 have been merged into its upstream (though I should have discussed for it here). Merged by there merge-requests: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/5092 https://gitlab.kitware.com/utils/kwsys/-/merge_requests/197 Merged patches for cmake-3.17.3-1 are perl-libs.patch 3.17.3-case-sensitivity.patch 3.17.3-opengl.patch 3.17.3-findtclsh.patch 3.17.3-cpuinfo.patch 3.17.3-cygwin-paths.patch https://github.com/matzeri/cygwin-pkg/tree/64864eb8f0d635ea44226e4b9a41a7ca59a1c7dd/cmake and 0001-3.17.3-cpuinfo_suppl.patch https://sourceware.org/pipermail/cygwin/2020-July/245563.html Regards, Lem Thanks Lem, I am impressed, in the past we had much more discussion for having our patches merged upstream. If you want to take over or just co-maintain, let me know Regards Marco
Re: git repositories for cygwin packaging - please test
Jon Turney writes: > One problem I have noticed is that some packages have test suites > (which are getting run via 'cygport test' invoking src_test()) which: > > - require lots of extra dependencies to run, or I currently subsume these in BUILD_REQUIRES as they are indeed required for a (complete) build. It'd be possible to split them out into a separate variable indeed, but that is yet another level of specificatrion that needs to be figured out if it should become useful. > - don't succeed on Cygwin, or Any such tests I usually patch out or mark expected fail whichever seems easier. > - take an inordinate amount of time to run (exceeding the resource limits) That is a problem that comes with CI I think and we didn't really have had to consider so far. I have a few packages that I don't run tests on by default because the test suite produces hangs or other unstable behaviour, but that is dealt with in the src_test function itself. If there are really resource hungry tests they usually need to be enabled somewhere and one could skip those if the build runs on CIm (how to find that out?). > So I'm wondering if .cygport files need: > > - an annotation to indicate tests shouldn't be run by this system (in > RESTRICT? or somewhere new?) Not RESTRICT please, that does things related to packaging. > - a separate TEST_REQUIRES to list packages which depended on for > running tests? That seems the most prudent name in this case. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
Lemures Lemniscati via Cygwin-apps writes: >> No, I was talking about things like XZ_OPT or ZSTD_CLEVEL. > > I think XZ_OPT or ZSTD_CLEVEL can be passed through environment. > And this should control xz or zstd.. > > XZ_OPT="..." cygport foo.cygport package > ZSTD_CLEVEL="..." cygport foo.cygport package > > And cygport does not prevent from passing parameter through environment. > Maybe I understand you in a wrong way. Pardon me. Maybe if you'd gone back to the post I sent on June 14 when I originally offered the patch you'd have seen: >> This patch keeps the current defaults in place, but allows one to set up >> two environment variables to control how tar gets invoked during >> packaging. You could do that from either the cygport file or the >> .cygport.conf file: >> >> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar -I zstd --group nobody:65534 --owner >> nobody:65534" >> CYGPORT_TAR_EXT=".tar.zst" I don't want to pollute the environment for any other command obviously… and really, I still think creating five new configuration variables to do what can be accomplished with two isn't a good trade-off. Factoring out the command invocation into a function probably is a net win, though -- so I'd keep that part. It'd probably be useful in some other instances to so a similar refactoring. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
cmake upstream has merged cygwin patches
Hi. The cygwin patches for cmake-3.17.3 have been merged into its upstream (though I should have discussed for it here). Merged by there merge-requests: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/5092 https://gitlab.kitware.com/utils/kwsys/-/merge_requests/197 Merged patches for cmake-3.17.3-1 are perl-libs.patch 3.17.3-case-sensitivity.patch 3.17.3-opengl.patch 3.17.3-findtclsh.patch 3.17.3-cpuinfo.patch 3.17.3-cygwin-paths.patch https://github.com/matzeri/cygwin-pkg/tree/64864eb8f0d635ea44226e4b9a41a7ca59a1c7dd/cmake and 0001-3.17.3-cpuinfo_suppl.patch https://sourceware.org/pipermail/cygwin/2020-July/245563.html Regards, Lem
Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
On Thu, 6 Aug 2020 18:13:52 +0100, Jon Turney > On 05/08/2020 22:50, Lemures Lemniscati via Cygwin-apps wrote: > > On Wed, 5 Aug 2020 20:34:24 +0100, Jon Turney > >> > >> However, I think there is a consistency problem here, as other variables > >> which should only produce a single line in the .hint file (e.g. REQUIRES, > >> etc.) don't get whitespace canonicalized (as least, that's how it seemed > >> to me when I briefly looked at this). > >> > >> Thanks for the patch. > > > > Thank you for reviewing. > > > > It would be easy to canonicalize through a functiion like > > __single_line_list () in the commit > > https://github.com/cygwin-lem/cygport/commit/7607782d3d1972aef6b88ee32f5211f21abbbcfb > > > > I'll check later for 'category:', 'requires:', 'obsoletes:', > > 'provides:', and 'conflicts:'. > > > > Any other field to be checked for canonicalization? > > Although it's not very clearly specified in [1], the only keys in a ..hint > file for which multiline values are currently permitted are ldesc: and > message: > > [1] https://cygwin.com/packaging-hint-files.html Thank you! Here is a revised commit: https://github.com/cygwin-lem/cygport/commit/7bd6b12ee10892b7ec18df93a51dbc274b3af968 Canonicalization of single-line fields in .hint files We mean trimming and squeezing white spaces by 'canonicalization', here. - Canonicalization of single-line lists: - category - requires - obsoletes - provides - conflicts - build-depends - Canonicalization of single-line double-quoted strings: - sdesc - message Regards, Lem
Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
On Fri, 7 Aug 2020 09:45:34 +0200, Thomas Wolff
> Am 04.08.2020 um 22:34 schrieb Lemures Lemniscati via Cygwin-apps:
> > CYGPORT_TAR_OPTS="--owner=foo --group=bar" cygport baz.cygport package
> >
> > And if you have tar >=1.31, these will also work:
> >
> > CYGPORT_TAR_EXT=".tar.zst" cygport baz.cygport package
> >
> > CYGPORT_TAR_OPTS="--owner=foo --group=bar" CYGPORT_TAR_EXT=".tar.zst"
> > cygport baz.cygport package
> I'd like to suggest, additionally to an explicit option, to set user and
> group by default, as it is also a privacy issue to spread the packager's user
> name out to the world in the tar archive.
> In that case I'd use the project name (no version) for the user name and
> "cygwin" for the group name.
Now the default is --owner=${NAME} --group=cygwin
And we can override like this
CYGPORT_TAR_OPTS="--owner=very --group=happy" cygport aaa.cygport package
On Thu, 06 Aug 2020 20:25:01 +0200, ASSI
> Lemures Lemniscati via Cygwin-apps writes:
> > Do you mean controlling with '-I' or '--use-compress-program='?
> > (And I didn't know these options of tar... oh.)
>
> No, I was talking about things like XZ_OPT or ZSTD_CLEVEL.
I think XZ_OPT or ZSTD_CLEVEL can be passed through environment.
And this should control xz or zstd..
XZ_OPT="..." cygport foo.cygport package
ZSTD_CLEVEL="..." cygport foo.cygport package
And cygport does not prevent from passing parameter through environment.
Maybe I understand you in a wrong way. Pardon me.
But I've introduced a variable CYGPORT_TAR_FILTER to be passed to tar
with '-I' option which can contain spaces.
For example, it's ok now to invoke like this:
CYGPORT_TAR_FILTER="zstd -T0 --long --19" CYGPORT_TAR_EXT=".tar.zst" cygport
aaa.cygport package
Here is the new commit revised.
https://github.com/cygwin-lem/cygport/commit/7155f24df1d6058c5c5554d28b9eeb8b2b9f1a16
Regards,
Lem
Re: cygport: Request a new feature in order to set owner/group names in packaged tarballs.
Am 04.08.2020 um 22:34 schrieb Lemures Lemniscati via Cygwin-apps: On Tue, 4 Aug 2020 15:46:13 +0200, Thomas Wolff Am 04.08.2020 um 14:54 schrieb Lemures Lemniscati via Cygwin-apps: Date: Mon, 03 Aug 2020 21:24:11 +0200 From: Achim Gratz Lemures Lemniscati via Cygwin writes: This is another patch, so that cygport shall make tarballs with specified owner and group names. Cygport patches should better go to cygwin-apps. I've already sent a patch that allows you to do the same thing some time ago, but it has not been reviewed yet. https://repo.or.cz/cygport/rpm-style.git/commitdiff/c6af2ca23aae5da3e99c70cf2b704430b929f431 Nice. Then, how about a commit following yours. It is much less than obvious in that older patch that you can trick the owner/group information into that option. I'd appreciate a simple explicit option for that. All right. I've simplified options to Simplify options to CYGPORT_TAR_OPTS and CYGPORT_TAR_EXT. https://github.com/cygwin-lem/cygport/commit/5a502cc84b8db0b47eae8b3571d363d106e74160 This will work: CYGPORT_TAR_OPTS="--owner=foo --group=bar" cygport baz.cygport package And if you have tar >=1.31, these will also work: CYGPORT_TAR_EXT=".tar.zst" cygport baz.cygport package CYGPORT_TAR_OPTS="--owner=foo --group=bar" CYGPORT_TAR_EXT=".tar.zst" cygport baz.cygport package I'd like to suggest, additionally to an explicit option, to set user and group by default, as it is also a privacy issue to spread the packager's user name out to the world in the tar archive. In that case I'd use the project name (no version) for the user name and "cygwin" for the group name. Thomas
