Re: [Toybox] toybox in Android status
On Sun, Jan 25, 2015 at 12:15 PM, Rob Landley wrote: > On 01/17/15 00:09, enh wrote: >> more progress... > ... >> I’ve moved the following commands over to toybox, removing the old >> toolbox implementations: >> >> vmstat > > Going into a bit more depth on vmstat, on my netbook right now: > > $ vmstat > procs ---memory-- ---swap-- -io -system-- cpu > r b swpd free buff cache si sobibo in cs us sy id wa > 7 0 2677468 163892 129332 970508 15 12 132 1172 15 37 7 54 2 > $ ./toybox vmstat > procs ---memory-- ---swap-- -io -system-- cpu > r b swpd free buff cache si sobibo in cs us sy id wa > 2 0 2677468 166900 129376 971616 14 11 473 4180 777 37 7 54 2 > > The "memory" and "system" fields (swpd, free, buff, cache, us, sy, id, wa) > look right. (And at least mine gets the rounding right. :) > > The "swap" fields (si/so) are close but _consistently_ off by one, some > sort of rounding difference?. > > The i/o stuff isn't even close and I haven't got a clue what it thinks > it's reporting. The ratios are consistent but: > print 473.0/132.0 > 3.583 print 418/117.0 > 3.57264957265 > > Three and a bit? (Rounding _and_ different block size...?) > > And then there's "system". I have no idea what that's displaying. System in: The number of interrupts per second, including the clock. cs: The number of context switches per second. iirc, toybox, toolbox, and desktop procps were all in disagreement in one way or another. also, my desktop vmstat has more columns than yours: procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 1 1 2314312 2398456 1818852 4950084000 74222 6 1 93 0 0 the man page also explains what 'st' is: st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown. > Also, vmstat and ls both do variable indenting of fields, but use completely > different implementations that don't genericize easily. (ls is doing readahead > and measuring, vmstat has an array of fields each marked with a padding > length and if it goes over it forces the padding of all future fields to one > space until it's back under budget. Not really the same, but the human > readable changes pending elsewhere open this can of worms in a _third_ way...) as shown above, desktop vmstat isn't even close to lining up. presumably hardware was much less meaty the last time anyone was seriously maintaining this code? oh; turns out there's an option: -w, --wide Wide output mode (useful for systems with higher amount of mem‐ ory, where the default output mode suffers from unwanted column breakage). The output is wider than 80 characters per line. but it's off by default, even if you have lots of RAM. neither toolbox nor toybox implemented -f, -m, -s, -d, et cetera, and i hadn't even heard of any of those options until i read the man page to learn what some of the fields in the regular output are supposed to mean. > So that's why _that_ one is in the pending README. > > Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Sun, Jan 25, 2015 at 12:14 PM, Rob Landley wrote: > On 01/17/15 00:09, enh wrote: >> more progress... >> >> Right now toybox provides the following new commands that we didn’t >> have with toolbox: >> >> acpi >> basename blockdev bzcat >> cal chgrp chroot cksum comm cut >> dirname dos2unix >> echo egrep env expand >> fallocate fgrep find free >> groups >> head hostname hwclock >> killall >> logname losetup lspci lsusb >> md5sum modinfo more mountpoint >> nice nl >> od >> paste patch pidof pmap pwd >> realpath >> sed seq setsid sha1sum sort split stat strings sysctl >> tac tail taskset tee time timeout tr truncate >> uname uniq unix2dos usleep >> wc which whoami xargs >> yes > > Some of these commands (hwclock, more, and tr) are in pending. > As you've seen from the review of Luis' patches, things in pending often > have reasons for being there. If android is going to start using those, > I _really_ need to clean them up. the reason these commands are in a separate list is that they're commands we didn't have before. i'm assuming that "something is better than nothing", and also that nothing helps shake out the bugs than getting a bunch of users. (more(1) is actually something we had before, albeit as a shell alias. but the toybox one is still better, so that didn't feel like a risky move.) > I'm in the process of cleaning up hwclock, I'll prioritize more and tr. > > Actually tr is already high on my list, it's #3 on the list of most > commonly called remaining busybox commands in aboriginal: > > http://landley.net/notes-2015.html#16-01-2015 > > It's after sed (which can switch over next release), and expr (which I > was poking at yesterday and needs that redo for priority grouping I had > to poke the posix guys to update the html version of the spec about)... > >> I’ve moved the following commands over to toybox, removing the old >> toolbox implementations: >> >> cat chcon chmod chown clear cmp cp >> date dmesg >> false >> getenforce >> id ifconfig inotifyd(notify) insmod >> kill >> ln lsmod >> mkdir mknod mkswap mv >> netstat nohup >> printenv >> readlink rm rmdir rmmod >> setenforce sleep sync swapoff swapon >> true >> vmstat > > Pending here is chcon, getenforce, setenforce, and netstat. > > Three of those are from you, so I have no fears about you running into > something unexpected. > > Netstat is from Ashwini Sharma's team (which may choose to decloak > someday), and I just haven't had a chance to look at 650 lines of > network code yet. the reason i didn't just switch everything over on day one is because i've actually been going through the tools one by one trying to convince myself that they're at least as good as and roughly compatible with what we had before. (again, this is why there are two lists. my bar can be lower from with the stuff that wasn't part of the default system image in the past.) the pending netstat seemed to be a compatible superset of what we had that produced similar results, so i'm not too worried about that. > (Promoting route.c is my next networking todo item. > Longer-term, we probably need something like iptables but that's not in > the roadmap yet because I can't even scope it without more research. > ipchains is toast but there's a new one, ipspikes? iprestraints? > ipwhips? I forget, I assume it's in keeping with the established naming > theme...) > >> next to look at: >> >> df --- missing -h; column alignment broken. no need to support old >> Android output format. >> du --- our default block size was 512; -h output slightly >> different. [patch sent upstream] >> ls --- missing -Z >> route --- superset? >> touch --- ours has nonstandard -l (equivalent to standard -h), >> better resolution. [patch sent upstream] > > Which means I need to look at all of those too, of course. (But ls -Z I > need a patch from you.) yeah, i'll get back to the SELinux toybox patches at some point but i've been too busy to make any progress recently. i think the reason ls is in this list (rather than my separate "missing SELinux stuff" list) is because you said you had some known output formatting bugs you wanted to fix first? > In addition to the actual toys/pending/*.c files, toys/pending/README > has a list of commands that predate the pending directory, but could use > another review/cleanup pass. (These aren't as bad, but I'm not happy > with them yet.) Of that, you're using: > > vmstat: as hg commit 1171 (http://127.0.0.1/hg/toybox/rev/1171) > explains, several of the fields aren't displaying what other > versions do, and it's not obvious _what_ the other versions > are displaying. (It's clearly not raw /proc data.) > > chroot: because of http://landley.net/notes-2011.html#02-06-2011 > (I need to add a pivot_root mode that you can't trivially >escape from as root
Re: [Toybox] toybox in Android status
On 01/17/15 00:09, enh wrote: > more progress... ... > I’ve moved the following commands over to toybox, removing the old > toolbox implementations: > > vmstat Going into a bit more depth on vmstat, on my netbook right now: $ vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 7 0 2677468 163892 129332 970508 15 12 132 1172 15 37 7 54 2 $ ./toybox vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 2 0 2677468 166900 129376 971616 14 11 473 4180 777 37 7 54 2 The "memory" and "system" fields (swpd, free, buff, cache, us, sy, id, wa) look right. (And at least mine gets the rounding right. :) The "swap" fields (si/so) are close but _consistently_ off by one, some sort of rounding difference?. The i/o stuff isn't even close and I haven't got a clue what it thinks it's reporting. The ratios are consistent but: >>> print 473.0/132.0 3.583 >>> print 418/117.0 3.57264957265 Three and a bit? (Rounding _and_ different block size...?) And then there's "system". I have no idea what that's displaying. Also, vmstat and ls both do variable indenting of fields, but use completely different implementations that don't genericize easily. (ls is doing readahead and measuring, vmstat has an array of fields each marked with a padding length and if it goes over it forces the padding of all future fields to one space until it's back under budget. Not really the same, but the human readable changes pending elsewhere open this can of worms in a _third_ way...) So that's why _that_ one is in the pending README. Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/17/15 00:09, enh wrote: > more progress... > > Right now toybox provides the following new commands that we didn’t > have with toolbox: > > acpi > basename blockdev bzcat > cal chgrp chroot cksum comm cut > dirname dos2unix > echo egrep env expand > fallocate fgrep find free > groups > head hostname hwclock > killall > logname losetup lspci lsusb > md5sum modinfo more mountpoint > nice nl > od > paste patch pidof pmap pwd > realpath > sed seq setsid sha1sum sort split stat strings sysctl > tac tail taskset tee time timeout tr truncate > uname uniq unix2dos usleep > wc which whoami xargs > yes Some of these commands (hwclock, more, and tr) are in pending. As you've seen from the review of Luis' patches, things in pending often have reasons for being there. If android is going to start using those, I _really_ need to clean them up. I'm in the process of cleaning up hwclock, I'll prioritize more and tr. Actually tr is already high on my list, it's #3 on the list of most commonly called remaining busybox commands in aboriginal: http://landley.net/notes-2015.html#16-01-2015 It's after sed (which can switch over next release), and expr (which I was poking at yesterday and needs that redo for priority grouping I had to poke the posix guys to update the html version of the spec about)... > I’ve moved the following commands over to toybox, removing the old > toolbox implementations: > > cat chcon chmod chown clear cmp cp > date dmesg > false > getenforce > id ifconfig inotifyd(notify) insmod > kill > ln lsmod > mkdir mknod mkswap mv > netstat nohup > printenv > readlink rm rmdir rmmod > setenforce sleep sync swapoff swapon > true > vmstat Pending here is chcon, getenforce, setenforce, and netstat. Three of those are from you, so I have no fears about you running into something unexpected. Netstat is from Ashwini Sharma's team (which may choose to decloak someday), and I just haven't had a chance to look at 650 lines of network code yet. (Promoting route.c is my next networking todo item. Longer-term, we probably need something like iptables but that's not in the roadmap yet because I can't even scope it without more research. ipchains is toast but there's a new one, ipspikes? iprestraints? ipwhips? I forget, I assume it's in keeping with the established naming theme...) > next to look at: > > df --- missing -h; column alignment broken. no need to support old > Android output format. > du --- our default block size was 512; -h output slightly > different. [patch sent upstream] > ls --- missing -Z > route --- superset? > touch --- ours has nonstandard -l (equivalent to standard -h), > better resolution. [patch sent upstream] Which means I need to look at all of those too, of course. (But ls -Z I need a patch from you.) In addition to the actual toys/pending/*.c files, toys/pending/README has a list of commands that predate the pending directory, but could use another review/cleanup pass. (These aren't as bad, but I'm not happy with them yet.) Of that, you're using: vmstat: as hg commit 1171 (http://127.0.0.1/hg/toybox/rev/1171) explains, several of the fields aren't displaying what other versions do, and it's not obvious _what_ the other versions are displaying. (It's clearly not raw /proc data.) chroot: because of http://landley.net/notes-2011.html#02-06-2011 (I need to add a pivot_root mode that you can't trivially escape from as root within the chroot, for containers. Really this is a workaround for a longstanding kernel bug. Currently no worse than other chroot implementations, but I want to do it _right. Design issue, not implementation.) mountpoint: because of the comment "Ignore the fact a file can be a mountpoint for --bind mounts.", and because it thinks a symlink to a mountpoint is a mountpoint, and because minor code duplication. But I can fix two of those right now... xargs: Rich Felker rightfully points out that it doesn't implement the INSANE posix newline and whitespace parsing stuff. (because -0 renders all that obsolete, but posix is stuck back in the 1980's in a number of places, such as this.) Hence XARGS_PEDANTIC, which I haven't implemented yet. Also several options are commented out in the help text, which is a reminder to me I haven't implemented those yet. cut: I haven't even reviewed this one enough to change the coding style yet. (We've got lines over 80 chars, single line /* */ style comments, error messages that assume the reader knows more than a couple dozen words of english...) The code may be solid, I just haven't verified it yet. modinfo: I think commit 934 got most of this
Re: [Toybox] toybox in Android status
On Sat, Jan 17, 2015 at 11:00 PM, Rich Felker wrote: > On Sat, Jan 17, 2015 at 06:59:57PM -0600, Rob Landley wrote: >> On 01/17/2015 01:53 PM, Rob Landley wrote: >> > On 01/17/2015 12:09 AM, enh wrote: >> > So anyway I checked that in, and now I'm porting your patch on top of >> > that change and I hit the _weirdest_ thing where "./touch nonexistent" >> > doesn't cause an error because utimensat() on a file that didn't exist >> > returned ok (file still doesn't exist afterwards), >> >> Because if both timestamps have nanoseconds set to UTIME_OMIT it returns >> success even if the file did not exist. >> >> Possibly the man page should mention that. > > No, it's a kernel bug that should be fixed, and that may be necessary > to work around in userspace if the kernel doesn't fix it. See > http://pubs.opengroup.org/onlinepubs/9699919799/functions/futimens.html > > In particular: > > The utimensat() and utimes() functions shall fail if: > > [ENOENT] > A component of path does not name an existing file or path is an > empty string. > > There is no option to ignore this error condition just because both > timestamps are UTIME_OMIT. but the "shall" sections include EACCES and EPERM which are both explicitly excluded by... > This is made even more clear in the description: > > If both tv_nsec fields are set to UTIME_OMIT, no ownership or > permissions check shall be performed for the file, but other error > conditions may still be detected (including [EACCES] errors > related to the path prefix). ...which only says "may" for other error conditions (like ENOENT). _we_ all agree this is undesirable, but i'm not sure it's out of spec. a quick grep of the Android source tree turns up a qemu implementation of utimensat that contains the same UTIME_OMIT [too-]early exit. i plan on reporting it upstream anyway. in the meantime, the man page now at least contains a warning. --elliott ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/18/2015 01:00 AM, Rich Felker wrote: > This is made even more clear in the description: > > If both tv_nsec fields are set to UTIME_OMIT, no ownership or > permissions check shall be performed for the file, but other error > conditions may still be detected (including [EACCES] errors > related to the path prefix). > > If you can confirm the incorrect behavior, we should report this to > the kernel folks. You can make a trivial test program for it, and I'm all for fixing it, but there are existing kernels out in the field we have to deal with that do this, and fixing future versions doesn't help those. I've adjusted my code not to call the function with those arguments so I can avoid hitting it. (We're arguing over the behavior of a NOP. Yes, the kernel is doing the NOP wrong, but if it's a NOP my code shouldn't bother doing it.) > Rich Rob (Having reread the above twice I'm _fairly_ certain it's intelligible english, but going bedwards and practicing horizontality may be advisable at this point.) ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Sat, Jan 17, 2015 at 11:53 AM, Rob Landley wrote: > On 01/17/2015 12:09 AM, enh wrote: >> more progress... > > I need to update roadmap.html. > >> next to look at: >> >> df --- missing -h; column alignment broken. no need to support old >> Android output format. > > I have a half-finished rewrite of this to do the ls -l "buffer the > output, count up the sizes, output everything aligned to the size of the > longest entry in each column" trick. > > I'd genericize it in lib if I could figure out how. Maybe when a third > command does this it'll look like common code... > >> du --- our default block size was 512; -h output slightly >> different. [patch sent upstream] > > The only old one I have pending from you us the touch patch, although I > see there are some new ones this morning. (I may have lost track of this > one, might need a resend.) i'll look it out and resend. iirc it was worded more as a query because i know BSD has a lot more options to its equivalent 'human_readable' and i suspect when we're done with all the -h options we'll have needed the lot, but i'm guessing you just want to add what's needed right now? > (I was trying to clear your patches so I could tackle Ashwini Sharma's > pending pile from October, but there's a strong appeal in staying > current as well. Although I found one more #*%&(# thing wrong with sed...) > >> ls --- missing -Z > > I don't have a patch for this and am not competent to do it myself: I > don't use selinux, dunno what's expected here. no worries. this is just me showing my own to-do list. (mostly the SELinux stuff is on a separate list that i don't even bother including. not sure why ls is on the wrong list.) actually, someone asked "when do we get toybox ls?" the other day because they were sick of -h not working. they were disappointed to hear that toybox doesn't support -h yet either, but that wouldn't be a regression. afaik, the lack of -Z is the only thing i need to address. it looked like the tizen guys found and fixed a number of existing bugs with ls output. did they send the patches to you? if so, i'll probably wait until that's done. https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=f7a1a508ee6c713e93c030bba3ce1ba69c2a https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=54e0fb4d814aeb8ddd8584351cea8f4c4b8b8322 https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=131ea0f2410651754c442ce77eca67bbba5d271d btw, you can see in those patches that they (and i) need to add a temporary buffer for the security context. it's well documented that you don't like #if but -- as with the addition of declarations of functions that won't be defined -- it's not clear to me whether you're happy with unused locals, and if so, whether you want __attribute__((unused)) to keep things quiet? >> route --- superset? > > I haven't cleaned up route yet. I should do that, ifconfig's kinda silly > without it. (Last night I started cleaning up "fold" because it looked > small. I keep trying to do "smallest job first" scheduling of cleanups, > and it blows up into rewrites...) > >> touch --- ours has nonstandard -l (equivalent to standard -h), >> better resolution. [patch sent upstream] > > I spent some time yesterday merging the resolution increase patch. (You > noticed I had a pending "fix fractional seconds" patch in my tree that > wasn't fully debugged? Ideally I'd like both -d and -t to handle > fractional seconds. A lot of commands have half-finished patches, I keep > getting interrupted in the middle to go do other things. My tree's a > mess, I have "toybox/toybox" for pending work and "toybox/clean" to test > that what's checked in builds/works...) > > So anyway I checked that in, and now I'm porting your patch on top of > that change and I hit the _weirdest_ thing where "./touch nonexistent" > doesn't cause an error because utimensat() on a file that didn't exist > returned ok (file still doesn't exist afterwards), and when it does this > its argument variable becomes NULL (wasn't null before the call, is > after), and it's _weird_ and I think stack smashing is happening or > something and I'm trying to figure out what the blank is going on. (It's > gotta be that ts is wrong so the syscall is writing to the stack and > zeroing out the pointer but why didn't it return an _error_ for the > nonexistent file then...? I am now running my own code under strace > because sticking printfs into it is not being revealing. I'm -> <- this > close to doing objdump to look at the assembly and then tracing into the > #*%(&#@ C library. _I_HAVE_DONE_THIS_BEFORE_...) > > Debugging. If I knew what I was doing, I'd have done it already. > > Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Sat, Jan 17, 2015 at 4:59 PM, Rob Landley wrote: > On 01/17/2015 01:53 PM, Rob Landley wrote: >> On 01/17/2015 12:09 AM, enh wrote: >> So anyway I checked that in, and now I'm porting your patch on top of >> that change and I hit the _weirdest_ thing where "./touch nonexistent" >> doesn't cause an error because utimensat() on a file that didn't exist >> returned ok (file still doesn't exist afterwards), > > Because if both timestamps have nanoseconds set to UTIME_OMIT it returns > success even if the file did not exist. > > Possibly the man page should mention that. yeah, the Linux man page probably based its text on the POSIX "may": "If both tv_nsec fields are set to UTIME_OMIT, no ownership or permissions check shall be performed for the file, but other error conditions may still be detected (including [EACCES] errors related to the path prefix)." the comment in the kernel source raises this to a "must not" for some reason: "/* Nothing to do, we must not even check the path. */". i've mailed the linux man pages folks. (even having read the source to send the patch to strace so that they [will in future] output the times correctly for UTIME_NOW and UTIME_OMIT when the seconds field isn't 0, and even though "must not even check the path" is explicitly stated, i guess that was so obviously wrong it didn't pass my plausibility filter. lucky you had a test for that case!) >> and when it does this its argument variable becomes NULL > > Was debugging debris (bug introduced by adding debugging code). > > Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/17/2015 08:10 PM, enh wrote: > (even having read the source to send the patch to strace so that they > [will in future] output the times correctly for UTIME_NOW and > UTIME_OMIT when the seconds field isn't 0, and even though "must not > even check the path" is explicitly stated, i guess that was so > obviously wrong it didn't pass my plausibility filter. lucky you had a > test for that case!) I didn't have a test for it, and it's probably a bug that my code was doing it. I just have this ingrained habit to DROP EVERYTHING when the code does something I can't explain, and stare at it until I've root caused it. (Making a bug go away is not fixing it. A reproducible test case of a failure is gold: I don't just want to beat the correct behavior out of it, I need to know why it _didn't_ work.) (On a related note, I try to test error paths to make sure things _fail_ right. I haven't put enough time into the testsuite same as everything else, and some things like "integer overflow entered from the command line" are arguably pilot error. But "it worked for me" is not a comfortable stopping point.) If you were wondering why I'm so slow at this, I get stuck on the weirdest things... Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Sat, Jan 17, 2015 at 06:59:57PM -0600, Rob Landley wrote: > On 01/17/2015 01:53 PM, Rob Landley wrote: > > On 01/17/2015 12:09 AM, enh wrote: > > So anyway I checked that in, and now I'm porting your patch on top of > > that change and I hit the _weirdest_ thing where "./touch nonexistent" > > doesn't cause an error because utimensat() on a file that didn't exist > > returned ok (file still doesn't exist afterwards), > > Because if both timestamps have nanoseconds set to UTIME_OMIT it returns > success even if the file did not exist. > > Possibly the man page should mention that. No, it's a kernel bug that should be fixed, and that may be necessary to work around in userspace if the kernel doesn't fix it. See http://pubs.opengroup.org/onlinepubs/9699919799/functions/futimens.html In particular: The utimensat() and utimes() functions shall fail if: [ENOENT] A component of path does not name an existing file or path is an empty string. There is no option to ignore this error condition just because both timestamps are UTIME_OMIT. This is made even more clear in the description: If both tv_nsec fields are set to UTIME_OMIT, no ownership or permissions check shall be performed for the file, but other error conditions may still be detected (including [EACCES] errors related to the path prefix). If you can confirm the incorrect behavior, we should report this to the kernel folks. Rich ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/17/2015 08:08 PM, enh wrote:
> On Sat, Jan 17, 2015 at 11:53 AM, Rob Landley wrote:
>> On 01/17/2015 12:09 AM, enh wrote:
>>> more progress...
>>
>> I need to update roadmap.html.
>>
>>> next to look at:
>>>
>>> df --- missing -h; column alignment broken. no need to support old
>>> Android output format.
>>
>> I have a half-finished rewrite of this to do the ls -l "buffer the
>> output, count up the sizes, output everything aligned to the size of the
>> longest entry in each column" trick.
>>
>> I'd genericize it in lib if I could figure out how. Maybe when a third
>> command does this it'll look like common code...
>>
>>> du --- our default block size was 512; -h output slightly
>>> different. [patch sent upstream]
>>
>> The only old one I have pending from you us the touch patch, although I
>> see there are some new ones this morning. (I may have lost track of this
>> one, might need a resend.)
>
> i'll look it out and resend. iirc it was worded more as a query
> because i know BSD has a lot more options to its equivalent
> 'human_readable' and i suspect when we're done with all the -h options
> we'll have needed the lot, but i'm guessing you just want to add
> what's needed right now?
I like to be guided by users. Myself I err on the side of not including
things, and then wait for somebody to argue that they need it.
>> (I was trying to clear your patches so I could tackle Ashwini Sharma's
>> pending pile from October, but there's a strong appeal in staying
>> current as well. Although I found one more #*%&(# thing wrong with sed...)
>>
>>> ls --- missing -Z
>>
>> I don't have a patch for this and am not competent to do it myself: I
>> don't use selinux, dunno what's expected here.
>
> no worries. this is just me showing my own to-do list. (mostly the
> SELinux stuff is on a separate list that i don't even bother
> including. not sure why ls is on the wrong list.) actually, someone
> asked "when do we get toybox ls?" the other day because they were sick
> of -h not working. they were disappointed to hear that toybox doesn't
> support -h yet either, but that wouldn't be a regression. afaik, the
> lack of -Z is the only thing i need to address. it looked like the
> tizen guys found and fixed a number of existing bugs with ls output.
> did they send the patches to you? if so, i'll probably wait until
> that's done.
>
> https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=f7a1a508ee6c713e93c030bba3ce1ba69c2a
>
> https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=54e0fb4d814aeb8ddd8584351cea8f4c4b8b8322
>
> https://git.tizen.org/cgit/platform/upstream/toybox.git/commit/?id=131ea0f2410651754c442ce77eca67bbba5d271d
Actually I'm waiting for them to send me patches, but I can poke Xavier
again...
> btw, you can see in those patches that they (and i) need to add a
> temporary buffer for the security context. it's well documented that
> you don't like #if but -- as with the addition of declarations of
> functions that won't be defined -- it's not clear to me whether you're
> happy with unused locals, and if so, whether you want
> __attribute__((unused)) to keep things quiet?
Send me a patch that works and I'll clean it up to my coding standards,
I'm not expecting you to guess. :)
The compiler doesn't warn about:
void blah(void)
{
int i;
if (0) printf("%d", i);
}
Because the i is used. Dead code elimination comparisons against a
constant are assumed to be intentional, so if it's used in an if
(CFG_THINGY) block then it's used even when CFG_THINGY is 0.
Rob
___
Toybox mailing list
Toybox@lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/17/2015 01:53 PM, Rob Landley wrote: > On 01/17/2015 12:09 AM, enh wrote: > So anyway I checked that in, and now I'm porting your patch on top of > that change and I hit the _weirdest_ thing where "./touch nonexistent" > doesn't cause an error because utimensat() on a file that didn't exist > returned ok (file still doesn't exist afterwards), Because if both timestamps have nanoseconds set to UTIME_OMIT it returns success even if the file did not exist. Possibly the man page should mention that. > and when it does this its argument variable becomes NULL Was debugging debris (bug introduced by adding debugging code). Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/17/2015 12:09 AM, enh wrote: > more progress... I need to update roadmap.html. > next to look at: > > df --- missing -h; column alignment broken. no need to support old > Android output format. I have a half-finished rewrite of this to do the ls -l "buffer the output, count up the sizes, output everything aligned to the size of the longest entry in each column" trick. I'd genericize it in lib if I could figure out how. Maybe when a third command does this it'll look like common code... > du --- our default block size was 512; -h output slightly > different. [patch sent upstream] The only old one I have pending from you us the touch patch, although I see there are some new ones this morning. (I may have lost track of this one, might need a resend.) (I was trying to clear your patches so I could tackle Ashwini Sharma's pending pile from October, but there's a strong appeal in staying current as well. Although I found one more #*%&(# thing wrong with sed...) > ls --- missing -Z I don't have a patch for this and am not competent to do it myself: I don't use selinux, dunno what's expected here. > route --- superset? I haven't cleaned up route yet. I should do that, ifconfig's kinda silly without it. (Last night I started cleaning up "fold" because it looked small. I keep trying to do "smallest job first" scheduling of cleanups, and it blows up into rewrites...) > touch --- ours has nonstandard -l (equivalent to standard -h), > better resolution. [patch sent upstream] I spent some time yesterday merging the resolution increase patch. (You noticed I had a pending "fix fractional seconds" patch in my tree that wasn't fully debugged? Ideally I'd like both -d and -t to handle fractional seconds. A lot of commands have half-finished patches, I keep getting interrupted in the middle to go do other things. My tree's a mess, I have "toybox/toybox" for pending work and "toybox/clean" to test that what's checked in builds/works...) So anyway I checked that in, and now I'm porting your patch on top of that change and I hit the _weirdest_ thing where "./touch nonexistent" doesn't cause an error because utimensat() on a file that didn't exist returned ok (file still doesn't exist afterwards), and when it does this its argument variable becomes NULL (wasn't null before the call, is after), and it's _weird_ and I think stack smashing is happening or something and I'm trying to figure out what the blank is going on. (It's gotta be that ts is wrong so the syscall is writing to the stack and zeroing out the pointer but why didn't it return an _error_ for the nonexistent file then...? I am now running my own code under strace because sticking printfs into it is not being revealing. I'm -> <- this close to doing objdump to look at the assembly and then tracing into the #*%(&#@ C library. _I_HAVE_DONE_THIS_BEFORE_...) Debugging. If I knew what I was doing, I'd have done it already. Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
more progress... Right now toybox provides the following new commands that we didn’t have with toolbox: acpi basename blockdev bzcat cal chgrp chroot cksum comm cut dirname dos2unix echo egrep env expand fallocate fgrep find free groups head hostname hwclock killall logname losetup lspci lsusb md5sum modinfo more mountpoint nice nl od paste patch pidof pmap pwd realpath sed seq setsid sha1sum sort split stat strings sysctl tac tail taskset tee time timeout tr truncate uname uniq unix2dos usleep wc which whoami xargs yes I’ve moved the following commands over to toybox, removing the old toolbox implementations: cat chcon chmod chown clear cmp cp date dmesg false getenforce id ifconfig inotifyd(notify) insmod kill ln lsmod mkdir mknod mkswap mv netstat nohup printenv readlink rm rmdir rmmod setenforce sleep sync swapoff swapon true vmstat next to look at: df --- missing -h; column alignment broken. no need to support old Android output format. du --- our default block size was 512; -h output slightly different. [patch sent upstream] ls --- missing -Z route --- superset? touch --- ours has nonstandard -l (equivalent to standard -h), better resolution. [patch sent upstream] On Thu, Jan 1, 2015 at 1:14 PM, enh wrote: > current status... > > not previously available, now provided by toybox: > > acpi > basename blockdev bzcat > cal chgrp chroot cksum comm cut > dirname dos2unix > echo egrep env expand > factor fallocate fgrep find free > groups > head hostname > killall > logname losetup lspci lsusb > md5sum modinfo more mountpoint > nice nl > od > paste patch pidof pmap pwd > readahead realpath > seq setsid sha1sum sort split stat strings sysctl > tac tail taskset tee time timeout truncate > uname uniq unix2dos usleep > wc which whoami xargs > yes > > previously toolbox, now toybox: > > chcon chmod clear cp > dmesg > false > getenforce > insmod > kill > ln lsmod > mkdir mknod mkswap mv > netstat nohup > printenv > readlink rm rmdir rmmod > setenforce sleep sync swapoff swapon > true > vmstat > > close to switching: > > cat --- missing -n (though we do now have nl(1)) [patch sent upstream] > chown --- bugs with argument parsing [patch sent upstream] > cmp --- missing -b, -n. (both dating from the original git import, > with no changes since. ignore and try toybox anyway?) > date --- probably okay to split this into date/hwclock (will need > local patch for hwclock to use /dev/alarm instead of /dev/rtc; should > fix /dev/rtc handling) > df --- missing -h; column alignment broken. [tradefed now patched > to support new format; no need to support either old Android format] > du --- our default block size was 512; -h output slightly > different. [patch sent upstream] > grep --- missing -A, -B, -C, color. compare performance with BSD grep? > id --- missing -Z, output slightly different. [patch sent upstream] > ifconfig --- seems to be superset; test networking after reboot > and switch over? > inotifyd (vs notify) --- seems to be incompatible superset; move > over and see what happens? > ls --- missing -Z > renice --- ours has nonstandard -r, -t (equivalent to -n?), -g (“get”). > touch --- ours has nonstandard -l (equivalent to standard -h), > better resolution. [patch sent upstream] > > most blocking for me is probably your decision on whether to have a > single TOYBOX_SELINUX configuration option that turns on all the > selinux stuff or not. iirc i sent that as part of the id patch, where > i also changed stuff like setenforce to say "default y depends on > TOYBOX_SELINUX" so there's just one switch to flip. i'll get back to > the other selinux commands next week, so let me know which style to > use. -- Elliott Hughes - http://who/enh - http://jessies.org/~enh/ Java i18n/JNI/NIO, or bionic questions? Mail me/drop by/add me as a reviewer. ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Fri, Jan 2, 2015 at 11:20 AM, Rob Landley wrote: > On 01/01/2015 09:57 PM, David Seikel wrote: >> So if Android is switching to toybox, is my unfinished project to >> create a toybox installing app now obsolete? > > There's still stuff to do. In increasing order of difficulty: > > 1) It's only going into new ones at some point in the future. The > installed base won't have it for a while. > > 2) Considering I've never managed to get an xterm in a basic android > install, something that lets me run a shell prompt without going full > cyanogenmod is nice. see packages/apps/Terminal in AOSP, which you can enable from developer options. it's very basic, but patches welcome. in particular, if you're not using a physical keyboard, ConnectBot's ctrl/alt implementation would be very useful. or there's ConnectBot if you're on a production device. > So far I've been using that champion gnuroot thing > (since he packaged up my aboriginal linux root filesystem and > everything, and that gives me fullscreen terminal): > > http://www.linux-magazine.com/Online/Blogs/Productivity-Sauce/GNURoot-Linux-on-Android-No-Root-Required > > (Alas if I leave that running, android has an annoying tendency to > reboot the phone. Not sure why.) > > 3) If you remember my CELF talk (and I went over it again on the linux > luddites interview, episode 11 I think), turning an android phone into a > development environment means plugging it into a USB hub and adding: > > 1) USB keyboard > 2) USB mouse > 3) USB video adapter (USB to VGA for usb2, USB to HDMI for usb3). > > All that hardware's available off the shelf at best buy today (I > checked), and possibly you could work around 3 with a chromecast and use > bluetooth for 1 and 2 instead (although the advantage of plugging it > into a hub is it charges the phone battery, and you can trivially add a > big hard drive and gigabit ethernet connection and so on). > > But the problem I had with my current android phone is it hasn't got > drivers for any of that. It's always in USB gadget mode, not in USB host > mode. So if I plug it into a hub, it can't scan the hub to see what's on > there and use any of it. > >> I was stuck on "what to do if busybox is already installed and has >> symlinks plastered all over the file system". I'm still thinking about >> how to deal with that. > > Put the symlinks in /bin/toybox and add it to the $PATH? (This is why > toybox has "make install_flat" for when it's not part of the base > system. You can have it be first in the $PATH or last in the $PATH as > you like...) > > The only things that really call stuff at absolute paths are executable > loaders, so the dynamic linker is at an absolute path and so are script > interpreters (the #! at the start of shell/perl/python scripts), and > toysh isn't ready to carry any real weight yet. > > Rob > ___ > Toybox mailing list > Toybox@lists.landley.net > http://lists.landley.net/listinfo.cgi/toybox-landley.net ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/01/2015 09:57 PM, David Seikel wrote: > So if Android is switching to toybox, is my unfinished project to > create a toybox installing app now obsolete? There's still stuff to do. In increasing order of difficulty: 1) It's only going into new ones at some point in the future. The installed base won't have it for a while. 2) Considering I've never managed to get an xterm in a basic android install, something that lets me run a shell prompt without going full cyanogenmod is nice. So far I've been using that champion gnuroot thing (since he packaged up my aboriginal linux root filesystem and everything, and that gives me fullscreen terminal): http://www.linux-magazine.com/Online/Blogs/Productivity-Sauce/GNURoot-Linux-on-Android-No-Root-Required (Alas if I leave that running, android has an annoying tendency to reboot the phone. Not sure why.) 3) If you remember my CELF talk (and I went over it again on the linux luddites interview, episode 11 I think), turning an android phone into a development environment means plugging it into a USB hub and adding: 1) USB keyboard 2) USB mouse 3) USB video adapter (USB to VGA for usb2, USB to HDMI for usb3). All that hardware's available off the shelf at best buy today (I checked), and possibly you could work around 3 with a chromecast and use bluetooth for 1 and 2 instead (although the advantage of plugging it into a hub is it charges the phone battery, and you can trivially add a big hard drive and gigabit ethernet connection and so on). But the problem I had with my current android phone is it hasn't got drivers for any of that. It's always in USB gadget mode, not in USB host mode. So if I plug it into a hub, it can't scan the hub to see what's on there and use any of it. > I was stuck on "what to do if busybox is already installed and has > symlinks plastered all over the file system". I'm still thinking about > how to deal with that. Put the symlinks in /bin/toybox and add it to the $PATH? (This is why toybox has "make install_flat" for when it's not part of the base system. You can have it be first in the $PATH or last in the $PATH as you like...) The only things that really call stuff at absolute paths are executable loaders, so the dynamic linker is at an absolute path and so are script interpreters (the #! at the start of shell/perl/python scripts), and toysh isn't ready to carry any real weight yet. Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 01/01/2015 03:14 PM, enh wrote: > close to switching: > > cat --- missing -n (though we do now have nl(1)) [patch sent upstream] > chown --- bugs with argument parsing [patch sent upstream] > cmp --- missing -b, -n. (both dating from the original git import, > with no changes since. ignore and try toybox anyway?) > date --- probably okay to split this into date/hwclock (will need > local patch for hwclock to use /dev/alarm instead of /dev/rtc; should > fix /dev/rtc handling) > df --- missing -h; column alignment broken. [tradefed now patched > to support new format; no need to support either old Android format] > du --- our default block size was 512; -h output slightly > different. [patch sent upstream] > grep --- missing -A, -B, -C, color. compare performance with BSD grep? > id --- missing -Z, output slightly different. [patch sent upstream] > ifconfig --- seems to be superset; test networking after reboot > and switch over? > inotifyd (vs notify) --- seems to be incompatible superset; move > over and see what happens? > ls --- missing -Z > renice --- ours has nonstandard -r, -t (equivalent to -n?), -g (“get”). > touch --- ours has nonstandard -l (equivalent to standard -h), > better resolution. [patch sent upstream] Working on finishing and checking some of these fixes in. (Last week "hg diff | wc" was over 10,000 lines of partial local changes I hadn't checked in, now it's down to 1043 lines although some of that was me moving things like the partial grep -ABC into a separate patch for the moment. (Bit off a little more than I could chew there. Chewing...) > most blocking for me is probably your decision on whether to have a > single TOYBOX_SELINUX configuration option that turns on all the > selinux stuff or not. iirc i sent that as part of the id patch, where > i also changed stuff like setenforce to say "default y depends on > TOYBOX_SELINUX" so there's just one switch to flip. i'll get back to > the other selinux commands next week, so let me know which style to > use. That sounds good to me. I may add a compile-time probe for support, but that can go in later. I'm trying to get caught up on your pending patches this weekend. Thanks for your patience. (Back to squinting at sed. It's ALMOST done, but Linux From Scratch isn't quite building with it yet. It's getting _whitespace_ wrong, of all things. Which is significant when you have a \ line continuation followed by an extra blank line...) Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
So if Android is switching to toybox, is my unfinished project to create a toybox installing app now obsolete? I was stuck on "what to do if busybox is already installed and has symlinks plastered all over the file system". I'm still thinking about how to deal with that. -- A big old stinking pile of genius that no one wants coz there are too many silver coated monkeys in the world. signature.asc Description: PGP signature ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
current status... not previously available, now provided by toybox: acpi basename blockdev bzcat cal chgrp chroot cksum comm cut dirname dos2unix echo egrep env expand factor fallocate fgrep find free groups head hostname killall logname losetup lspci lsusb md5sum modinfo more mountpoint nice nl od paste patch pidof pmap pwd readahead realpath seq setsid sha1sum sort split stat strings sysctl tac tail taskset tee time timeout truncate uname uniq unix2dos usleep wc which whoami xargs yes previously toolbox, now toybox: chcon chmod clear cp dmesg false getenforce insmod kill ln lsmod mkdir mknod mkswap mv netstat nohup printenv readlink rm rmdir rmmod setenforce sleep sync swapoff swapon true vmstat close to switching: cat --- missing -n (though we do now have nl(1)) [patch sent upstream] chown --- bugs with argument parsing [patch sent upstream] cmp --- missing -b, -n. (both dating from the original git import, with no changes since. ignore and try toybox anyway?) date --- probably okay to split this into date/hwclock (will need local patch for hwclock to use /dev/alarm instead of /dev/rtc; should fix /dev/rtc handling) df --- missing -h; column alignment broken. [tradefed now patched to support new format; no need to support either old Android format] du --- our default block size was 512; -h output slightly different. [patch sent upstream] grep --- missing -A, -B, -C, color. compare performance with BSD grep? id --- missing -Z, output slightly different. [patch sent upstream] ifconfig --- seems to be superset; test networking after reboot and switch over? inotifyd (vs notify) --- seems to be incompatible superset; move over and see what happens? ls --- missing -Z renice --- ours has nonstandard -r, -t (equivalent to -n?), -g (“get”). touch --- ours has nonstandard -l (equivalent to standard -h), better resolution. [patch sent upstream] most blocking for me is probably your decision on whether to have a single TOYBOX_SELINUX configuration option that turns on all the selinux stuff or not. iirc i sent that as part of the id patch, where i also changed stuff like setenforce to say "default y depends on TOYBOX_SELINUX" so there's just one switch to flip. i'll get back to the other selinux commands next week, so let me know which style to use. ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On Fri, Dec 19, 2014 at 12:02 PM, Rob Landley wrote: > On 12/19/2014 12:40 AM, enh wrote: >> (note that all lists were generated by hand so might not be exactly >> right, but should give the rough idea.) > > This is entire message is awesome and I'll probably devote the whole > weekend to follow-up, but lemme reply to one specific part now: > >>df --- our non-standard output is parsed by tradefed. > > In response to the earlier human readable thread I've torn apart my > local copy and am rewriting it to do the "readahead and measure the > column spacing" trick (since adding -h spacing as a third special case > got kind of silly). > > It looks like your output is 5 columns (Filesystem, Size, Used, Free, > Blksize), columns 2-4 are -h format, and the first column is mount point > rather than block device. > > I can do all that, but: > > 1) Can I add a new command line option to produce this output? Maybe > dash capital A? (Trying to resolve conflicting expectations for users in > different contexts...) i'd actually bump df down your priority list. i was certainly planning on coming back to it last. i want to talk to the testing team about exactly this, and whether they need to talk to old and new devices. > 2) Does the stuff parsing it care about the spacing, or will the > auto-spacing (varying based on the readings) work for them? looks like they use this regular expression right now: "%s\\s+[\\w\\d\\.]+\\s+[\\w\\d\\.]+\\s+([\\d\\.]+)(\\w)" one thing that's interesting is that they already have code that says "Parses a partitions available space from the legacy output of a 'df' command.". i'll get in touch with them, but things are already winding down here so it might be 2015 before i can get a canonical answer. --elliott ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
Re: [Toybox] toybox in Android status
On 12/19/2014 12:40 AM, enh wrote: > (note that all lists were generated by hand so might not be exactly > right, but should give the rough idea.) This is entire message is awesome and I'll probably devote the whole weekend to follow-up, but lemme reply to one specific part now: >df --- our non-standard output is parsed by tradefed. In response to the earlier human readable thread I've torn apart my local copy and am rewriting it to do the "readahead and measure the column spacing" trick (since adding -h spacing as a third special case got kind of silly). It looks like your output is 5 columns (Filesystem, Size, Used, Free, Blksize), columns 2-4 are -h format, and the first column is mount point rather than block device. I can do all that, but: 1) Can I add a new command line option to produce this output? Maybe dash capital A? (Trying to resolve conflicting expectations for users in different contexts...) 2) Does the stuff parsing it care about the spacing, or will the auto-spacing (varying based on the readings) work for them? Rob ___ Toybox mailing list Toybox@lists.landley.net http://lists.landley.net/listinfo.cgi/toybox-landley.net
