Re: [Toybox] [PATCH] Pass cflags and hostcc in mkroot.
> > Passing through HOSTCC makes sense, but passing through CFLAGS... is it > > HOSTCFLAGS or CROSS_CFLAGS? > > Except now that I sit down to do it (in the kernel HOSTCFLAGS is a variable > and > CFLAGS seems to go to the cross compiler, modulo whatever EXTRA_CFLAGS and > HOST_EXTRACFLAGS do), there's a design problem: > > The airlock step already wrapped the compiler. If you're specifying HOSTCC > then > that should be what was wrapped? (But if you do put the flags in the $HOSTCC > name, I'm not sure how to teach the airlock to wrap that...) > > Specifying HOSTCC="/path/to/thingy --options" on the command line bypasses the > airlock step entirely, so it'll wrap the hostcc it finds but then use > something > else. :) That's ugly but works. I meant after reading mkroot more closely I welcome the variables status quo. No need pass through more. However the airlock and all HOSTCC usage could inline -static by default, not in HOSTCC but in their commands after HOSTCC, then HOSTCC is a clean path to an executable. > kinda aimed at a common case and adding too many weird knobs to it might > defeat > the purpose? I'm unsure. Exactly, fewer knobs is good, fewer untested combinations. > > Pkgsrc sources include a reference shell and awk capable of a bootstrapping > > its > > bmake to build other software. > > Of course it does. Why wouldn't it? > > > Their build scripts want make and autotools on > > the host, > > So they include awk but expect autotools as an environment dependency. Sorry I word bad! Pkgsrc environment dependencies are kernel, libc, headers, toolchain, core utils, shell, awk and wget. They (pkgsrc) expect you to cross-compile these from a fat host, then pkgsrc downloads and builds everything else natively. I just figured out how to cross-compile less and bootstrap without the fat host, using mcm prebuilts and toybox, first pkgsrc and then a complete system that can rebuild toybox. XFCE didn't eventually work there but Weston does, so I think I'll upload that script somewhere including recent changes and maintain if someone's interested. ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] [PATCH 1/2] xargs: fix kernel builds.
On Mon, Oct 14, 2019 at 2:42 PM Rob Landley wrote:
>
> On 10/14/19 3:53 PM, enh via Toybox wrote:
> > It turns out that findutils xargs almost always uses an artificially low
> > limit of 128KiB. You can observe this with --show-limits (which I
> > refrained from adding to toybox since I believe it's only useful if
> > you're debugging a broken xargs).
>
> The kernel guys' decision not to give us a way to query this is wrong, by the
> way. (I'm honestly tempted to whack-a-mole based on queried kernel version in
> portability.h at this point. It'd never work for macos and freebsd though...)
>
> > I think an alternative fix for all this would be to go back to counting
> > the cost of the (char *)s, but separate the -s limit from the "system"
> > limit --- that way we could have the same behavior as findutils xargs
> > for explicit values of -s (which we all seem to agree should *not*
> > include the cost of the (char *)s), but also not accidentally overrun
> > the actual system limits when we do count the (char *)s. That's more
> > complicated though, and findutils' "128KiB is enough for anyone"
> > behavior is demonstrably "good enough", so let's go with that for now.
>
> Are you building this on a kernel that _does_ have the 128k limit? (Which was
> lifted in 2.6.22 which came out July 8, 2007?) Otherwise a near miss of 128k
> shouldn't cause a problem...?
i'm not sure what you're talking about. the point is that the
findutils "don't count the (char *)s" algorithm is an _underestimate_
of how much space we really need, so comparing that to the actual
kernel limit is wrong. but findutils' other bug is that it doesn't
actually use the actual kernel limit. it uses 128KiB, which is much
smaller than the actual kernel limit (2090766 / 131072 == 16x), so it
"works" ("two wrongs make a right").
~$ xargs --show-limits
Your environment variables take up 2169 bytes
POSIX upper limit on argument length (this system): 2092935
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2090766
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
Execution of xargs will continue now, and it will try to read its
input and run commands; if this is not what you wanted to happen,
please type the end-of-file keystroke.
Warning: echo will be run at least once. If you do not want that to
happen, then press the interrupt keystroke.
~$
that 128KiB (131072 bytes) is just hard-coded. maybe because of
ancient kernels? but they don't seem to check kernel version at all.
i'm seeing their "always 128KiB" behavior on 4.19 and 5.x. (don't have
anything older easily available.)
> > Tested by building an Android common kernel with toybox xargs, which
> > failed before.
>
> Sigh, it built upstream -git. :P
repro steps if you have many gigabytes of disk and a bunch of time:
mkdir common-android-mainline && cd common-android-mainline
repo init -u https://android.googlesource.com/kernel/manifest -b
common-android-mainline
repo sync
ln -s /tmp/toybox/toybox ./build/build-tools/path/linux-x86/xargs
BUILD_CONFIG=common/build.config.allmodconfig.x86_64 build/build.sh
the issue is a bit hidden in the logs, so repeat with an incremental build:
SKIP_DEFCONFIG=1 SKIP_MRPROPER=1 SKIP_CP_KERNEL_HDR=1
BUILD_CONFIG=common/build.config.allmodconfig.x86_64 build/build.sh
gets you "xargs: Unknown option 'P8' (see "xargs --help")".
not skipping mrproper now triggers issue number two, so after the above:
BUILD_CONFIG=common/build.config.allmodconfig.x86_64 build/build.sh
gets you "xargs: exec rm: Argument list too long".
> > Bug: http://b/140269206
>
> What's the translation URL to see this externally again? (I need to add that
> to
> the FAQ.)
it's issuetracker.something, but that's only for bugs that are
actually public bugs, so no help here. i've pasted the interesting
part above.
> (Sorry, my plane back to Austin leaves in two and a half hours, off to the
> airport. Probably just apply your stuff verbatim when I get home but I gotta
> go
> now.)
like i say in the commit message, we could try to do the _right_ thing
like BSD xargs, but being bug compatible with findutils is easy and
(demonstrably) "good enough", so i suspect we can get away with just
this. it upsets me a bit, but there's so many more useful things we
could fix first, and if findutils hasn't had to fix this, i doubt we
will either: 16x is one hell of a safety margin! i don't think we
_can_ hit that --- if every argument is 2 bytes, even if we're not
counting the 8 bytes for each pointer, that's still not off by a
factor of 16x.
> Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] [PATCH 1/2] xargs: fix kernel builds.
On 10/14/19 3:53 PM, enh via Toybox wrote: > It turns out that findutils xargs almost always uses an artificially low > limit of 128KiB. You can observe this with --show-limits (which I > refrained from adding to toybox since I believe it's only useful if > you're debugging a broken xargs). The kernel guys' decision not to give us a way to query this is wrong, by the way. (I'm honestly tempted to whack-a-mole based on queried kernel version in portability.h at this point. It'd never work for macos and freebsd though...) > I think an alternative fix for all this would be to go back to counting > the cost of the (char *)s, but separate the -s limit from the "system" > limit --- that way we could have the same behavior as findutils xargs > for explicit values of -s (which we all seem to agree should *not* > include the cost of the (char *)s), but also not accidentally overrun > the actual system limits when we do count the (char *)s. That's more > complicated though, and findutils' "128KiB is enough for anyone" > behavior is demonstrably "good enough", so let's go with that for now. Are you building this on a kernel that _does_ have the 128k limit? (Which was lifted in 2.6.22 which came out July 8, 2007?) Otherwise a near miss of 128k shouldn't cause a problem...? > Tested by building an Android common kernel with toybox xargs, which > failed before. Sigh, it built upstream -git. :P > Bug: http://b/140269206 What's the translation URL to see this externally again? (I need to add that to the FAQ.) (Sorry, my plane back to Austin leaves in two and a half hours, off to the airport. Probably just apply your stuff verbatim when I get home but I gotta go now.) Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] [PATCH] Pass cflags and hostcc in mkroot.
On 10/14/19 4:09 PM, Rob Landley wrote: >> Nah it makes perfect sense now. Config scripts are somewhat spaghetti but >> mkroot >> gets close to a right abstraction level. > > Passing through HOSTCC makes sense, but passing through CFLAGS... is it > HOSTCFLAGS or CROSS_CFLAGS? Except now that I sit down to do it (in the kernel HOSTCFLAGS is a variable and CFLAGS seems to go to the cross compiler, modulo whatever EXTRA_CFLAGS and HOST_EXTRACFLAGS do), there's a design problem: The airlock step already wrapped the compiler. If you're specifying HOSTCC then that should be what was wrapped? (But if you do put the flags in the $HOSTCC name, I'm not sure how to teach the airlock to wrap that...) Specifying HOSTCC="/path/to/thingy --options" on the command line bypasses the airlock step entirely, so it'll wrap the hostcc it finds but then use something else. :) That's ugly but works. Similarly, although toybox has $SED to let you use gsed, mkroot doesn't. It's kinda aimed at a common case and adding too many weird knobs to it might defeat the purpose? I'm unsure. Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] cmp tests failing
On 10/14/19 12:21 AM, enh via Toybox wrote: > did you have a corresponding cmp.c fix to go with the new tests in > https://github.com/landley/toybox/commit/99eeaec65441a03d8b0b3003de6aeb6a1b5a741b > which don't pass with toybox cmp? Yes, sorry. I checked them in about 15 minutes ago. :) Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] [PATCH] Pass cflags and hostcc in mkroot.
On 10/1/19 12:58 PM, Denys Nykula wrote: >> The string HOSTCC does not occur in scripts/mkroot.sh? > > HOSTCC appears in kconfig/Makefile, change.sh, install.sh, and make.sh called > by > Makefile on make calls from mkroot.sh called from the same Makefile on a make > launched by the shell user. Effectively always equal to cc lookup in host > PATH. Ah. >> Yeah, it's confusing. I pass through _some_ variables from the environment >> because they're mandatory ($PATH) and other because it's convenient >> ($CROSS_COMPILE), but... > > Nah it makes perfect sense now. Config scripts are somewhat spaghetti but > mkroot > gets close to a right abstraction level. Passing through HOSTCC makes sense, but passing through CFLAGS... is it HOSTCFLAGS or CROSS_CFLAGS? >> Except I'm not downloading anything in the current version, or using the host >> toolchain for anything, and mcm doesn't have distcc or ccache... > > Host usage is implicit. By not manually overriding everywhere, you get the cc > from host PATH building the config artifacts, and the host libc interpreter > for > the dynamic toybox in the airlock dir. It becomes apparent at runtime, when > cc or libc aren't at default paths, on for hypothetical example a statically > linked system without a toolchain in own firmware. Yeah, you've got a point. I've hit more than one build with things like HOSTCC="gcc --static" because it was the only way to specify what I ^#&@^!* MEANT. >> haven't added "make" to toybox yet >> (which comes after the shell), and it'll need an awk too... >> >> I take it you're cross compiling more stuff? There isn't really plumbing fo >> rtha >> tin the merged version. (There was in the seperate version...) > > Pkgsrc sources include a reference shell and awk capable of a bootstrapping > its > bmake to build other software. Of course it does. Why wouldn't it? > Their build scripts want make and autotools on > the host, So they include awk but expect autotools as an environment dependency. > so I read their debug output on a system that has all that, and write > a few lines of shell enough to build the shell with musl headers, chroot there > to make the awk and continue. The fun is in attachment. Haven't run all the > built tools yet. Wanted to see how many familiar things one can proceed to > build > without fatal errors after minimal patching. Next goal is to investigate zcat > and tar errors and fix them to remove a few lines from this script. And to > wait > on your toysh, removing more. I've got a tar error queued up that's been holding up the release. I should just cut a release and fix it next version, it's getting silly... Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] [PATCH 1/2] xargs: fix kernel builds.
It turns out that findutils xargs almost always uses an artificially low
limit of 128KiB. You can observe this with --show-limits (which I
refrained from adding to toybox since I believe it's only useful if
you're debugging a broken xargs).
I think an alternative fix for all this would be to go back to counting
the cost of the (char *)s, but separate the -s limit from the "system"
limit --- that way we could have the same behavior as findutils xargs
for explicit values of -s (which we all seem to agree should *not*
include the cost of the (char *)s), but also not accidentally overrun
the actual system limits when we do count the (char *)s. That's more
complicated though, and findutils' "128KiB is enough for anyone"
behavior is demonstrably "good enough", so let's go with that for now.
Tested by building an Android common kernel with toybox xargs, which
failed before.
Bug: http://b/140269206
---
toys/posix/xargs.c | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
From f49f42a3693147f25325fea59630a660237c2f34 Mon Sep 17 00:00:00 2001
From: Elliott Hughes
Date: Mon, 14 Oct 2019 13:46:22 -0700
Subject: [PATCH 1/2] xargs: fix kernel builds.
It turns out that findutils xargs almost always uses an artificially low
limit of 128KiB. You can observe this with --show-limits (which I
refrained from adding to toybox since I believe it's only useful if
you're debugging a broken xargs).
I think an alternative fix for all this would be to go back to counting
the cost of the (char *)s, but separate the -s limit from the "system"
limit --- that way we could have the same behavior as findutils xargs
for explicit values of -s (which we all seem to agree should *not*
include the cost of the (char *)s), but also not accidentally overrun
the actual system limits when we do count the (char *)s. That's more
complicated though, and findutils' "128KiB is enough for anyone"
behavior is demonstrably "good enough", so let's go with that for now.
Tested by building an Android common kernel with toybox xargs, which
failed before.
Bug: http://b/140269206
---
toys/posix/xargs.c | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/toys/posix/xargs.c b/toys/posix/xargs.c
index 6ba5fb07..3f0d3acf 100644
--- a/toys/posix/xargs.c
+++ b/toys/posix/xargs.c
@@ -76,7 +76,7 @@ static char *handle_entries(char *data, char **entry)
// for bug compatibility with busybox 1.30.1 and findutils 4.7.0.
for (;;) {
-if (++TT.bytes >= TT.s && TT.s) return save;
+if (++TT.bytes >= TT.s) return save;
if (!*s || isspace(*s)) break;
s++;
}
@@ -109,13 +109,19 @@ void xargs_main(void)
pid_t pid;
long posix_max_bytes;
+ // findutils 4.7.0 xargs works by "two wrongs make a right". It doesn't
+ // account for the space taken by the (char *)s, but because it uses an
+ // artifically low limit of 128KiB (observable via --show-limits), it gets
+ // away with it. Do the same for bug compatibility.
+ if (!TT.s) TT.s = 128*1024;
+
// POSIX requires that we never hit the ARG_MAX limit, even if we try to
// with -s. POSIX also says we have to reserve 2048 bytes "to guarantee
// that the invoked utility has room to modify its environment variables
// and command line arguments and still be able to invoke another utility",
// though obviously that's not really something you can guarantee.
posix_max_bytes = sysconf(_SC_ARG_MAX) - environ_bytes() - 2048;
- if (!TT.s || TT.s > posix_max_bytes) TT.s = posix_max_bytes;
+ if (TT.s > posix_max_bytes) TT.s = posix_max_bytes;
if (!FLAG(0)) TT.delim = '\n';
--
2.23.0.700.g56cf767bdb-goog
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] [PATCH 2/2] xargs: add no-op -P.
The Linux kernel has a hard-coded `-P 8`. As the simplest thing that lets us successfully build a kernel, add a no-op -P. Bug: http://b/140269206 --- toys/posix/xargs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) From ce454693f42b0cba8994dd59b19b0de57cc27986 Mon Sep 17 00:00:00 2001 From: Elliott Hughes Date: Mon, 14 Oct 2019 13:51:40 -0700 Subject: [PATCH 2/2] xargs: add no-op -P. The Linux kernel has a hard-coded `-P 8`. As the simplest thing that lets us successfully build a kernel, add a no-op -P. Bug: http://b/140269206 --- toys/posix/xargs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/toys/posix/xargs.c b/toys/posix/xargs.c index 3f0d3acf..5c1f9529 100644 --- a/toys/posix/xargs.c +++ b/toys/posix/xargs.c @@ -10,7 +10,7 @@ * TODO: -x Exit if can't fit everything in one command * TODO: -P NUM Run up to NUM processes at once -USE_XARGS(NEWTOY(xargs, "^E:optrn#<1(max-args)s#0[!0E]", TOYFLAG_USR|TOYFLAG_BIN)) +USE_XARGS(NEWTOY(xargs, "^E:oP#ptrn#<1(max-args)s#0[!0E]", TOYFLAG_USR|TOYFLAG_BIN)) config XARGS bool "xargs" @@ -36,7 +36,7 @@ config XARGS #include "toys.h" GLOBALS( - long s, n; + long s, n, P; char *E; long entries, bytes; -- 2.23.0.700.g56cf767bdb-goog ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] How can I contribute
> saw one of the Toybox talks and wanted to see how I could contribute
Completing landley.net/toybox/cleanup.html on toys/pending/{dhcp,route}.c
to help them out of pending would be most demanded I think.
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] How can I contribute
Greetings, I saw one of the Toybox talks and wanted to see how I could contribute to the project. Respectfully, Andrew -- Andrew Ilijic Information Systems Consultant Cell: 609.738.0521 | Fax: 609.298.4806 | Email: ilijic.and...@gmail.com ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] [PATCH] losetup: Fix null-termination of src string instead of dest after copy
The function loopback_setup(), after copying the loopback device name
with xstrncpy(), ensures the null-termination of the string by forcing
its last byte to 0.
Unfortunately, this operation:
- was probably intended to null-terminate dest instead;
- does not affect the program execution because src is free()d right
after;
- if the size of src is smaller than the offset of the written zero, it
modifies an unknown byte in the heap.
Drop the null-termination line to fix the issue: xstrcpy() automatically
null-terminates dest, or fails if the size of src is bigger than the the
requested number of bytes to copy.
Signed-off-by: Alessio Balsini
---
toys/other/losetup.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/toys/other/losetup.c b/toys/other/losetup.c
index e73761a0..917e64ea 100644
--- a/toys/other/losetup.c
+++ b/toys/other/losetup.c
@@ -113,7 +113,6 @@ static int loopback_setup(char *device, char *file)
loop->lo_offset = TT.o;
loop->lo_sizelimit = TT.S;
xstrncpy((char *)loop->lo_file_name, s, LO_NAME_SIZE);
-s[LO_NAME_SIZE-1] = 0;
if (ioctl(lfd, LOOP_SET_STATUS64, loop)) perror_exit("%s=%s", device,
file);
if (FLAG(s)) puts(device);
free(s);
--
2.23.0.700.g56cf767bdb-goog
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] cmp tests failing
(patch sent.) On Sun, Oct 13, 2019 at 10:21 PM enh wrote: > > did you have a corresponding cmp.c fix to go with the new tests in > https://github.com/landley/toybox/commit/99eeaec65441a03d8b0b3003de6aeb6a1b5a741b > which don't pass with toybox cmp? ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] [PATCH] cmp.c: fix behavior when given a single argument.
--- toys/posix/cmp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) From bde302e4202353e386ad7566b578d960a5796ec0 Mon Sep 17 00:00:00 2001 From: Elliott Hughes Date: Mon, 14 Oct 2019 08:28:59 -0700 Subject: [PATCH] cmp.c: fix behavior when given a single argument. --- toys/posix/cmp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/toys/posix/cmp.c b/toys/posix/cmp.c index c573f02e..9e1e555b 100644 --- a/toys/posix/cmp.c +++ b/toys/posix/cmp.c @@ -4,7 +4,7 @@ * * See http://opengroup.org/onlinepubs/9699919799/utilities/cmp.html -USE_CMP(NEWTOY(cmp, "<2>2ls(silent)(quiet)[!ls]", TOYFLAG_USR|TOYFLAG_BIN|TOYFLAG_ARGFAIL(2))) +USE_CMP(NEWTOY(cmp, "<1>2ls(silent)(quiet)[!ls]", TOYFLAG_USR|TOYFLAG_BIN|TOYFLAG_ARGFAIL(2))) config CMP bool "cmp" @@ -80,4 +80,5 @@ void cmp_main(void) toys.exitval = 2; loopfiles_rw(toys.optargs, O_CLOEXEC|(WARN_ONLY*!(toys.optflags&FLAG_s)), 0, do_cmp); + if (toys.optc == 1) do_cmp(0, "-"); } -- 2.23.0.700.g56cf767bdb-goog ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
[Toybox] [PATCH] lib getdirname()
This patch relates pull request created by E5ten
https://github.com/landley/toybox/pull/145
I would write getdirname() following way so it should output similar
output as libc dirname
but return always pointer that can be freed. You might have more
elegant solution but here is my take.
return similar dirnames as libc dirname
path getdirname
/usr/lib/ /usr
/usr/lib /usr
/usr/ /
usr.
/ /
. .
.. .
---
lib/lib.c | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
From 2197840947813992d3b7ef5e8230150fbb2872e3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jarno=20M=C3=A4kip=C3=A4=C3=A4?=
Date: Mon, 14 Oct 2019 17:50:07 +0300
Subject: [PATCH] lib getdirname()
return similar dirnames as libc dirname
path getdirname
/usr/lib/ /usr
/usr/lib /usr
/usr/ /
usr.
/ /
. .
.. .
---
lib/lib.c | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/lib/lib.c b/lib/lib.c
index fe402af8..25bd3b65 100644
--- a/lib/lib.c
+++ b/lib/lib.c
@@ -1010,7 +1010,13 @@ char *getdirname(char *name)
{
char *s = xstrdup(name), *ss = strrchr(s, '/');
- while (ss && *ss && *ss == '/' && s != ss) *ss-- = 0;
+ if (!ss) free(s), xstrdup(".");
+ else {
+ss = s+strlen(s)-1;
+if (ss != s) do {
+ *ss-- = 0;
+} while(*ss != '/');
+ }
return s;
}
--
2.19.1
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
