On Sun, Oct 6, 2019 at 11:38 AM Rob Landley <r...@landley.net> wrote: > > I have soooo many open but unfinished email reply windows. This one's a month > old! (You can tell how much time I've had to "clean" by how many half-finished > open windows my laptop's got...) > > On 9/1/19 12:57 PM, enh wrote: > >> You listed lsof out of the second list but not the others? > > > > correct. lsof is (odd thought it may seem) used in the build --- but > > only if something goes wrong; the resulting bug reporting includes > > things like lsof and pstree output. so they're third-class citizens as > > far as i'm concerned: > > > > 1. non-pending, used in successful build > > 2. pending, used in successful build > > 3. pending, not used normally > > Ok. > > >>>> The android directory's your baby, but could you explain a bit more what > >>>> you > >>>> were using them for before, and where you're using them now? > >>> > >>> i used to just have one .config so i'd have to jump through a few > >>> hoops to be able to use that on the linux host too. now i have > >>> separate configs, Android stuff only needs to build on Android, so > >>> portability shmortability. > >> > >> If they're not needed during the AOSP build and only have to be present on > >> the > >> target, then sure. > > > > if anyone starts using traceroute as part of the build, we're going to > > have to have words with them :-) > > > > (i assume the nsjail they're run in prevents that kind of silliness anyway.) > > > >> The reason toybox can do ps is it has magic selinux attributes only created > >> during the AOSP build, not available in the NDK build. I take it all the > >> selinux > >> attributes in the new filesystem are created during the filesystem > >> packaging step? > > > > yes. > > Where is this packaging done in AOSP? (I can dig my way to it eventually, but > you'd probably know off the top of your head. :)
system/sepolicy/private/file_contexts > >> (I tried to dig into AOSP recently. The Makefile sourced > >> build/make/core/main.mk > >> which started with ANDROID_BUILD_SHELL and I went off to look up what that > >> was > >> and wound up reading about > >> https://developer.android.com/ndk/guides/wrap-script > >> instead and never got back... :) > >> > >>>> Yay. In theory I'm doing that here, in practice it's sometimes just > >>>> every couple > >>>> weeks so I can take a while to spot regressions. And I'm testing > >>>> vanilla, not > >>>> android kernel, and not your config... > >>> > >>> (i don't actually know _which_ kernel they're building. it could > >>> actually be upstream, or it could be the "android common kernel", or > >>> it could be -- and this is actually the least likely -- a device > >>> kernel.) > >> > >> I love reproducible bug reports. Sending real world data through the > >> commands is > >> how you find real world problems, I'm all for it. > > > > i can tell you already that one of the issues is going to be xargs -P > > --- i almost sent you a patch last week that made -P a no-op (given > > that the docs say "up to ... processes"); i don't know whether it > > actually makes a difference to build times. > > While xargs is > running, you can send its process a SIGUSR1 signal to increase > the number of commands to run simultaneously, or a SIGUSR2 to > decrease the number. > > Seriously? > > Sigh, I'm ok with managing parallelism, you may notice that I'm doing so from > a > bash script in scripts/make.sh: > > https://github.com/landley/toybox/blob/master/scripts/make.sh#L314 > > Ooh, when did wait -n show up? I really _wanted_ that when I was writing this, > and it wasn't available. Hmmm... looks like bash 4.3. Released Feb 2014. Hmmm. > Not quite 7 years (5.5 at this point). But once I can build toybox under > toysh I > might make an exception. (Assuming it doesn't take that long to complete > toysh. > Grrr.) > > Anyway, I might whip up an xargs -P if it's quick and I get a gap, but this > SIGUSR increment/decrement thing is kind of _conceptually_ silly? (What... > why...?) i don't know of anyone who wants the signal nonsense. actually, if you think you can get the hard-coded -P 8 removed from the kernel, i think we could all forget about this. (the scary part is that searching all the source available to me internally, -P 8 is basically the only use of -P. there's the odd pointless -P 1, and one -P 100, but presumably everyone's copy & pasted 8 from somewhere? too high for my MacBook Pro which starts to melt around 4, and too low for my desktop, which can't count that low.) i do wonder how long it would take for anyone to notice if we made -P a no-op (and am very disappointed with myself that i didn't do that long ago). > >> Presumably the response is "oh the markdown command is broken, github does > >> it > >> magically right so you can only see what your changes look like after you > >> upload > >> them to an external website"... which has kinda slowed my acceptance of > >> this > >> patch. :P > > > > so much as i think markdown is the least worst option for markup ... > > the biggest problem is that there is no One True Markdown. aiui, the > > original one is so anemic that no-one uses it, and there are a bunch > > of independent reimplementations with no conformance test suite. > > github and gitiles are different, for example, which are the two i > > dealt with regularly. > > > > _but_ there is a common subset that you can rely on. > > How do I test what the output looks like locally? > > [I got pointed at a thing since and have an open terminal window for that...] > > > i suspect here you want to use ``` to blockquote. > > > > (but i can tell you that what i sent you does work with github because > > i edited it in the github bug tracker. you can just switch between > > "write" and "preview" there. and github at least uses the same > > markdown implementation everywhere :-) ) > > Well that's something. Hmmm... > > >>>>> * unrelated, i built a static toybox binary. > >>>> > >>>> Yay! (Against bionic?) > >>> > >>> correct. the current (R, not Q!) libc.a. > >> > >> I'm all for it. > > > > the NDK guy is too, so even though i'm a bit unsure about the > > usefulness without adbd fixes on old devices (which aren't possible) > > and a mksh static binary (which i could also do), i'll get this done. > > "something is better than nothing", after all. > > Yay! > > >>>>> * the transitive dependencies are pretty gross thanks to the cgroup > >>>>> stuff. i wonder if anyone uses that enough for it to be really worth > >>>>> it, or whether we could drop it without anyone noticing? > >>>> > >>>> Huh, I did this in nsenter: > >>>> > >>>> #define unshare(flags) syscall(SYS_unshare, flags) > >>>> #define setns(fd, nstype) syscall(SYS_setns, fd, nstype) > >>>> > >>>> Precisely to _reduce_ dependencies. Do you mean the CLONE flags being > >>>> defined, > >>>> or...? > >>> > >>> no, the Android cgroup stuff. grep ps.c for ANDROID. basically so ps > >>> can say "bg" or "fg" or whatever rather than an int. > >> > >> In all caps there's just the one? > >> > >> else if (CFG_TOYBOX_ON_ANDROID) > >> sprintf(not_o = toybuf+128, > >> "USER,%%sPPID,VSIZE,RSS,WCHAN:10,ADDR:10,S,%s", > >> FLAG(T) ? "CMD" : "NAME"); > >> > >> The other case insensitive hits are about scheduling policy and a todo > >> about > >> field size? > >> > >> bg and fg... the shell commands? > > > > no, the background and foreground cgroups used for scheduling policy. > > (the first of the case-insensitive matches you found.) > > I need to get an AOSP qemu instance building from source so I can test this > stuff... > > >> I've missed a curve here. > >> > >>>>> * ran tests (with some difficulty) on jellybean, and had about 100 > >>>>> failures out of about 1200 tests. (`adb shell` didn't return exit > >>>>> status that far back, so this is grep for PASS and FAIL rather than > >>>>> the usual output from my little test runner.) > >>>> > >>>> Statically linked against a newer bionic, but running on the older > >>>> kernel? > >>> > >>> correct. > >> > >> Seems pretty reasonable then. Do you have a dump of what the failures are > >> so we > >> can categorize them? > > > > not yet, i was doing it in parallel with a bunch of other stuff. i'll > > have a proper run though now everyone seems in favor of doing this. > > Did that ever happen? I've had a crazy month and totally lost context of what > I > was doing before... it's still lying around in a tree somewhere, once i've popped off all the current problems. > >>>> You sent a link to https://developer.android.com/about/dashboards which > >>>> says > >>>> Jellybean is 3.2% of machines checking into the play store or some such. > >>>> Even > >>>> adding in the 0.6% from the previous 2 releases, that means 96% of all > >>>> android > >>>> devices are running kitkat or newer. > >>> > >>> aye, but our self-imposed albatross is that we wait until the share is < > >>> 1%. > >> > >> Works for me. (And technically the 7 year horizon agrees with you. :) > > So the jellybean ABI will still be in the NDK until J drops below 1%, yes. > and the > static toybox binary on developer.android.com should be built with the > _oldest_ ABI? no, the *only* libc.a is the latest one, whatever that is at any given time. you could build with the 16 headers and link with the current libc.a, but it'd be a bit of a waste of time! > >>>> While the 7 year horizon is a bit over Kit Kat's introduction 6 years > >>>> and 1 > >>>> month ago, historical static binary thing is a new feature. I'm ok > >>>> anchoring it > >>>> to kitkat if you are. That's still going back 2 releases before toybox > >>>> was > >>>> _introduced_. > >>> > >>> ...which is exactly why those devices need a static toybox the most :-) > >>> > >>> i wish i had a good idea of "oldest release anyone's doing any serious > >>> testing on", but i don't. (and like i say, the shell and adb are awful > >>> that far back too. i don't think adb was fixed until N :-( ) > >> > >> I'm in the process of writing a proper shell you can |& with, which can > >> handle > >> its own job control and such. I have: > >> > >> $ cat boing |& sed s/boing/walrus/ > >> cat: walrus: No such file or directory > >> > >> In my test pile. Alas, it is by _no_ means there yet, but I am at least > >> working > >> on it. :) > > > > the shell's actually the easy part: it's easier to package that up > > than a static toybox (because fewer dependencies, and especially not > > that awkward libcutils one). i pity anyone who has to do any automated > > testing on old releases. (this is why the world is full of weird stuff > > like parsing ls(1) output to test whether files exist... <movie > > trailer voice>i a world where you can't rely on exit status...) > > Circa 2006 reiserfs didn't set d_type in getdents() so I had to change > something > in busybox to do an extra stat(). Only filesystem that _ever_ got that wrong, > as > far as I know... > > (Before I knew the context I mentally pronounced that re-i-serfs, and after > the > trial I went _back_ to mentally pronouncing it that way. I don't think I've > ever > named a software project after myself, and even Linus tried to call Linux > "freax", it was sysadmin Ari Lemmke who gave him an ftp directory called > "linux" > to publish it in...) > > >>>> I'm happy to do that, and from the website's nav bar. And yes, I need to > >>>> catch > >>>> up on patches this weekend, that's one in the stack for this release. > > [a month later, still no release out. Yesterday I plugged toybox into the old > aboriginal linux build again and this time I got most of the way through the > gcc > build, at which point the tar "no ../ files outside this directory" check is > false positiving on "./" for some reason, gotta track that down. All the > corner > cases, gotta constantly regression test, old builds are a good way to do that. > They test stuff I'd never think off...] releases have to be time-based or feature-based, you can't have both :-) > >>> it's certainly easier than folks trying to build toybox themselves... > >> > >> I need to get the FAQ updated with build instructions for various targets. > >> (It's > >> in the README, but more docs doesn't hurt...) > > > > toybox is an example in the NDK docs, > > For the old https://developer.android.com/ndk/guides/standalone_toolchain > toolchain type with the python script. The new one at > https://developer.android.com/ndk/guides/other_build_systems uses libpng and > bzip, but the principle's the same. :) > > I need to add a FAQ entry on building toybox with the NDK, I have to work it > out > again each time I do it... > > > but i'm not sure how many people > > actually build it themselves rather than just suffer whatever happened > > to be on whatever device they have. > > Given how many people are still sticking busybox on devices, I'd expect > there's > a demand for newer toybox binaries on older systems. (Especially if I get the > rest of the 1.0 roadmap sorted.) aren't busybox users folks who've rooted their devices? so they can just use the latest toybox anyway? > Rob _______________________________________________ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net