Re: [musl] bbox: musl versus uclibc

[Denys Vlasenko]

On Mon, Aug 14, 2017 at 8:48 PM, Rich Felker  wrote:
>> Now, the good news - musl has smaller data!
>> 6695 bytes versus 7129 bytes for uclibc:
>>
>>text  data   bss dechex filename
>>  894902   465  6664  902031  dc38f busybox.uclibc
>>  912538   563  6132  919233  e06c1 busybox.musl
>
> Probably getpw*/getgr* static buffers or something. musl's backend for
> these just uses the buffer out of getline so as not to impose cost in
> apps that don't need the legacy (non-_r) functions or an arbitrary
> limit on record length.
>
> It would be interesting to know where the text size increase comes
> from.

See attached (only musl data is in the file).


busybox_unstripped.map.data
Description: Binary data
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: [musl] bbox: musl versus uclibc

[Rich Felker]

On Mon, Aug 14, 2017 at 07:43:39PM +0200, Denys Vlasenko wrote:
> As uclibc is increasingly aging, I am finally forced
> to switch to musl: I'm bitten by a nasty bug in
> getopt() - hush is using it in a slightly unusual way,
> which uclibc does not expect.

While I'm glad musl is working for you, this sounds fragile unless
it's actually just a bug in uclibc, in which case it'll hopefully get
fixed. If you're relying on weird internal-state of getopt there's a
chance this could break in the future on musl or on other libcs.
Can you explain the problem you encountered on uclibc?

> I built a toolchain using
> https://github.com/richfelker/musl-cross-make
> (Rich, is this the thing I should be using?)
> and it worked with no issues at all.

Yes, it's the easiest and canonical way to get a full toolchain.

> (I can probably only wish for the README
> to also mention how to make this a _static_
> toolchain... I have a box with 32-bit userspace,
> would be awesome to be able to copy this fresh
> 64-bit toolchain to it and have it working).

If you put LDFLAGS="-static --static" (second form is a hack to
workaround libtool's stripping of -static :) in COMMON_CONFIG you
should get a static build of the toolchain, but it's linked to
whatever the libc in your build environment is.

In my config.mak I use:

ifeq ($(HOST),)
COMMON_CONFIG += CC="i486-linux-musl-gcc -static --static" 
CXX="i486-linux-musl-g++ -static --static"
endif
COMMON_CONFIG += CFLAGS="-g0 -Os" CXXFLAGS="-g0 -Os" LDFLAGS="-s -static 
--static"

to build the toolchain static-linked using an existing
i486/musl-targeting toolchain. The static in $CC/$CXX is needed
because some broken configure tests in GMP try to run test binaries
built without honoring LDFLAGS, and will fail to run if you don't have
ld-musl-i386 installed.

Now that canadian-cross support works, you could probably instead
treat i486-linux-musl as a cross compiler (host!=build) instead.

If you prefer you can use x86_64 instead, too, but I like the
toolchain binaries themselves being 32-bit simply because it saves
roughly half the ram usage when running them.

> Then I built busybox. Impressions:
> 
> Only a few options did not build:
> EXTRA_COMPAT and FEATURE_VI_REGEX_SEARCH
> failed because they need GNU regexp extensions.

Yes. There was a request for the corresponding version of some of
these extensions in the POSIX regex API, which bb vi could be
converted to use if we adopted them, but there is some nontrivial
runtime cost. It probably doesn't matter now since the regexec core is
fairly slow (well, good big-O but bad constant) right now, but it may
impose a more noticable relative cost once we make regexec faster.

> FEATURE_MOUNT_NFS and FEATURE_INETD_RPC do not build
> because they need rpc/rpc.h.
> Not complaining, since them being in libc was a mistake
> in the first place.

I think glibc has dropped these too now, haven't they? AFAIK they're
just keeping the symbols for ABI compat but they're not an exposed API
for linking new code; you're supposed to use tirpc.

> Now, the good news - musl has smaller data!
> 6695 bytes versus 7129 bytes for uclibc:
> 
>text  data   bss dechex filename
>  894902   465  6664  902031  dc38f busybox.uclibc
>  912538   563  6132  919233  e06c1 busybox.musl

Probably getpw*/getgr* static buffers or something. musl's backend for
these just uses the buffer out of getline so as not to impose cost in
apps that don't need the legacy (non-_r) functions or an arbitrary
limit on record length.

It would be interesting to know where the text size increase comes
from. Not that big in abs difference, but given that a large portion
of the text is busybox code and not libc, it's a fairly large relative
difference. Come to think about it, it might actually be regex..

Rich
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: bbox: musl versus uclibc

[Mattias Schlenker]

Am 14.08.2017 um 19:43 schrieb Denys Vlasenko:


As uclibc is increasingly aging, I am finally forced
to switch to musl: I'm bitten by a nasty bug in
getopt() - hush is using it in a slightly unusual way,
which uclibc does not expect.

I built a toolchain using
 https://github.com/richfelker/musl-cross-make
(Rich, is this the thing I should be using?)
and it worked with no issues at all.

(I can probably only wish for the README
to also mention how to make this a _static_
toolchain... I have a box with 32-bit userspace,
would be awesome to be able to copy this fresh
64-bit toolchain to it and have it working).


From time to time I am updating:

https://github.com/mschlenker/TinyCrossLinux

This might not be as clean as Rich Felkers Makefile collection, it uses 
the "classic" approach of building the compiler in two steps. It is 
extensively tested in x86_64 and i686, but not at all for other 
architectures. Comments are welcome.


Yours,
Mattias




Then I built busybox. Impressions:

Only a few options did not build:
EXTRA_COMPAT and FEATURE_VI_REGEX_SEARCH
failed because they need GNU regexp extensions.
FEATURE_MOUNT_NFS and FEATURE_INETD_RPC do not build
because they need rpc/rpc.h.
Not complaining, since them being in libc was a mistake
in the first place.

Now, the good news - musl has smaller data!
6695 bytes versus 7129 bytes for uclibc:

text  data   bss dechex filename
  894902   465  6664  902031  dc38f busybox.uclibc
  912538   563  6132  919233  e06c1 busybox.musl

Whee!
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox


--
Mattias Schlenker - Freier IT-Fachredakteur und -autor
redakt...@mattiasschlenker.de

Mattias Schlenker - IT-Consulting, Softwareentwicklung
consult...@mattiasschlenker.de

August-Bebel-Str. 74 - D-04275 LEIPZIG - GERMANY
Phone: +49 341 39290767 - Fax: +49 341 25365955
Mobile (primary): +49 163 6953657
Mobile (secondary): +49 163 1848610
VATIN DE240998538

https://github.com/mschlenker
http://www.mattiasschlenker.de/
http://www.arduino-hausautomation.de/

___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: bbox: musl versus uclibc

[Emmanuel Deloget]

On Mon, Aug 14, 2017 at 7:43 PM, Denys Vlasenko
 wrote:
> Only a few options did not build:
> EXTRA_COMPAT and FEATURE_VI_REGEX_SEARCH
> failed because they need GNU regexp extensions.

I have a patch somewhere that enable parts of VI_REGEX_SEARCH (it only
does forward regex search while the GNU counterpart implements both
forward and backward regex search). I can submit that if the community
is interested.

BR,

-- Emmanuel Deloget
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: bbox: musl versus uclibc

[wdlkmpx]

I'm sure there was plenty of people willing to contribute  to uclibc,
there is even an updated fork.

The project has been badly managed.. thats the only reason i can think
of for this situation  to happen

On 8/14/17, Denys Vlasenko  wrote:
> As uclibc is increasingly aging, I am finally forced
> to switch to musl: I'm bitten by a nasty bug in
> getopt() - hush is using it in a slightly unusual way,
> which uclibc does not expect.
>
> I built a toolchain using
> https://github.com/richfelker/musl-cross-make
> (Rich, is this the thing I should be using?)
> and it worked with no issues at all.
>
> (I can probably only wish for the README
> to also mention how to make this a _static_
> toolchain... I have a box with 32-bit userspace,
> would be awesome to be able to copy this fresh
> 64-bit toolchain to it and have it working).
>
> Then I built busybox. Impressions:
>
> Only a few options did not build:
> EXTRA_COMPAT and FEATURE_VI_REGEX_SEARCH
> failed because they need GNU regexp extensions.
> FEATURE_MOUNT_NFS and FEATURE_INETD_RPC do not build
> because they need rpc/rpc.h.
> Not complaining, since them being in libc was a mistake
> in the first place.
>
> Now, the good news - musl has smaller data!
> 6695 bytes versus 7129 bytes for uclibc:
>
>text  data   bss dechex filename
>  894902   465  6664  902031  dc38f busybox.uclibc
>  912538   563  6132  919233  e06c1 busybox.musl
>
> Whee!
> ___
> busybox mailing list
> busybox@busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox
>
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

bbox: musl versus uclibc

[Denys Vlasenko]

As uclibc is increasingly aging, I am finally forced
to switch to musl: I'm bitten by a nasty bug in
getopt() - hush is using it in a slightly unusual way,
which uclibc does not expect.

I built a toolchain using
https://github.com/richfelker/musl-cross-make
(Rich, is this the thing I should be using?)
and it worked with no issues at all.

(I can probably only wish for the README
to also mention how to make this a _static_
toolchain... I have a box with 32-bit userspace,
would be awesome to be able to copy this fresh
64-bit toolchain to it and have it working).

Then I built busybox. Impressions:

Only a few options did not build:
EXTRA_COMPAT and FEATURE_VI_REGEX_SEARCH
failed because they need GNU regexp extensions.
FEATURE_MOUNT_NFS and FEATURE_INETD_RPC do not build
because they need rpc/rpc.h.
Not complaining, since them being in libc was a mistake
in the first place.

Now, the good news - musl has smaller data!
6695 bytes versus 7129 bytes for uclibc:

   text  data   bss dechex filename
 894902   465  6664  902031  dc38f busybox.uclibc
 912538   563  6132  919233  e06c1 busybox.musl

Whee!
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: Busybox in Debian

[Mattias Schlenker]

Am 14.08.2017 um 16:42 schrieb Denys Vlasenko:

If you want "resume" and "ipconfig" in bbox, I can do that.


The weird thing with Debian is that any package can add scripts to the 
initramfs and call an initramfs update after installation. This 
perfectly makes sense for tools that manage RAIDs or special 
filesystems. Thus I guess properly updating busybox in Debian and 
testing it with a "normal" hard drive installed system and the installer 
won't be such a big deal. But afterwards packages have to be identified 
that rely on certain commands in the initramfs and those have to be 
checked or adjusted as well.


Regards,
Mattias

--
Mattias Schlenker - Freier IT-Fachredakteur und -autor
redakt...@mattiasschlenker.de

Mattias Schlenker - IT-Consulting, Softwareentwicklung
consult...@mattiasschlenker.de

August-Bebel-Str. 74 - D-04275 LEIPZIG - GERMANY
Phone: +49 341 39290767 - Fax: +49 341 25365955
Mobile (primary): +49 163 6953657
Mobile (secondary): +49 163 1848610
VATIN DE240998538

https://github.com/mschlenker
http://www.mattiasschlenker.de/
http://www.arduino-hausautomation.de/

___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: Busybox in Debian

[Denys Vlasenko]

>> ipconfig

Looks like it is used to run DHCP / BOOTP / RARP
on several interfaces, including a possibility
to run it on _all_ existing interfaces
(excluding loopback and point-to-point).
I see how this can be useful during boot.


>> minips

It's just mini-ps. Proliferation of reinvented tools
with slightly different names.


>> nuke

This is "rm -rf -- FILE". Should not be necessary.


>> resume

Writes "maj:min:offset" to /sys/power/resume.
Looks like this is support for resume from disk?


>> run-init

This tool is doing this:

Usage: exec run-init [-d caps] [-c consoledev] /real-root /sbin/init [args]
/*
 * run_init(realroot, consoledev, drop_caps, init, initargs)
 *
 * This function should be called as the last thing in kinit,
 * from initramfs, it does the following:
 *
 * - Delete all files in the initramfs;
 * - Remounts /real-root onto the root filesystem;
 * - Chroots;
 * - Drops comma-separated list of capabilities;
 * - Opens /dev/console;
 * - Spawns the specified init program (with arguments.)
 *
 * On failure, returns a human-readable error message.
 */

There is the "switch_root" tool in util-linux which does the
crucial part of this functionality - deleting / remounting / chrooting.
It is in bbox too.



If you want "resume" and "ipconfig" in bbox, I can do that.
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: Busybox in Debian

[Denys Vlasenko]

On Mon, Aug 14, 2017 at 3:05 PM, Ben Hutchings  wrote:
> On Mon, 2017-08-14 at 14:55 +0200, Denys Vlasenko wrote:
>> On Sat, Aug 12, 2017 at 4:15 PM, Ben Hutchings 
>> wrote:
>> > On Fri, 2017-08-11 at 14:54 -0400, Chris Boot wrote:
>> > [...]
>> > > - The default initramfs assembly system, initramfs-tools, incorporates
>> > > busybox into the initramfs and it's used until the root filesystem (and
>> > > /usr if separate) is mounted and can be pivot_rooted into. We also use
>> > > parts of klibc in the initramfs, and I'm not yet entirely clear what
>> > > tools in the initramfs are klibc tools, busybox applets or executables
>> > > copied from the running system.
>> >
>> > [...]
>> >
>> > If initramfs-tools is configured to use busybox, busybox installs all
>> > its commands and then klibc-utils installs any commands that are not
>> > already there.  So for any command that they both implement, the
>> > busybox implementation is used.
>>
>> Do you have a list of tools klibc-utils installs?
>
> If busybox is used, then klibc-utils provides these:
>
> chroot
> losetup
> pivot_root

There are such applets in busybox too.
If they are buggy or incompatible (e.g. a missing option),
please let me know.

> halt
> poweroff
> reboot

These applets are in busybox too, but they may do
bbox-specific actions depending on configuration.

If FEATURE_CALL_TELINIT=y, they run "telinit N",
expecting it to know how to signal init.

Otherwise they assume the bbox init is on the system,
and they simply signal it:
SIGUSR1 = halt, SIGUSR2 = poweroff, SIGTERM = reboot.

/* talk to init */
if (!ENABLE_FEATURE_CALL_TELINIT) {
/* bbox init assumed */
rc = kill(1, signals[which]);
} else {
/* SysV style init assumed */
/* runlevels:
 * 0 == shutdown
 * 6 == reboot */
execlp(CONFIG_TELINIT_PATH,
CONFIG_TELINIT_PATH,
which == 2 ? "6" : "0",
(char *)NULL
);
bb_perror_msg_and_die("can't execute '%s'",
CONFIG_TELINIT_PATH);
}


> fstype

What does this do?


> nfsmount

And this one? If ordinary mount can't mount NFS for some reason,
it's interesting to look into that reason. Sometime ago,
kernel grew the ability to mount NFS without funky binary API
in mount syscall, so it's even easier now - mount tool only has to
resolve DNS name to an IP address.

If you are going to play with it, not that you probaly do _not_
need FEATURE_MOUNT_NFS=y: it is
"Support mounting NFS file systems on Linux < 2.6.23" option,
not "You need this, or else NFS mounts will not work".


> ipconfig
> minips
> nuke
> resume
> run-init

These look like klibc-utils inventions?
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: Busybox in Debian

[Mattias Schlenker]

Am 14.08.2017 um 14:55 schrieb Denys Vlasenko:

On Sat, Aug 12, 2017 at 4:15 PM, Ben Hutchings  wrote:

On Fri, 2017-08-11 at 14:54 -0400, Chris Boot wrote:
[...]

- The default initramfs assembly system, initramfs-tools, incorporates
busybox into the initramfs and it's used until the root filesystem (and
/usr if separate) is mounted and can be pivot_rooted into. We also use
parts of klibc in the initramfs, and I'm not yet entirely clear what
tools in the initramfs are klibc tools, busybox applets or executables
copied from the running system.

[...]

If initramfs-tools is configured to use busybox, busybox installs all
its commands and then klibc-utils installs any commands that are not
already there.  So for any command that they both implement, the
busybox implementation is used.

Do you have a list of tools klibc-utils installs?


https://packages.debian.org/sid/i386/klibc-utils/filelis
Regards,
Mattias

--
Mattias Schlenker - Freier IT-Fachredakteur und -autor
redakt...@mattiasschlenker.de

Mattias Schlenker - IT-Consulting, Softwareentwicklung
consult...@mattiasschlenker.de

August-Bebel-Str. 74 - D-04275 LEIPZIG - GERMANY
Phone: +49 341 39290767 - Fax: +49 341 25365955
Mobile (primary): +49 163 6953657
Mobile (secondary): +49 163 1848610
VATIN DE240998538

https://github.com/mschlenker
http://www.mattiasschlenker.de/
http://www.arduino-hausautomation.de/

___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: Busybox in Debian

[Denys Vlasenko]

On Sat, Aug 12, 2017 at 4:15 PM, Ben Hutchings  wrote:
> On Fri, 2017-08-11 at 14:54 -0400, Chris Boot wrote:
> [...]
>> - The default initramfs assembly system, initramfs-tools, incorporates
>> busybox into the initramfs and it's used until the root filesystem (and
>> /usr if separate) is mounted and can be pivot_rooted into. We also use
>> parts of klibc in the initramfs, and I'm not yet entirely clear what
>> tools in the initramfs are klibc tools, busybox applets or executables
>> copied from the running system.
> [...]
>
> If initramfs-tools is configured to use busybox, busybox installs all
> its commands and then klibc-utils installs any commands that are not
> already there.  So for any command that they both implement, the
> busybox implementation is used.

Do you have a list of tools klibc-utils installs?
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox

Re: [PATCH v2] Silence misguided GCC warning about alignment issues

[Daniel Thompson]

On 07/08/17 21:58, Johannes Schindelin wrote:

When compiling xz_dec_stream.c with GCC (at least with versions 5.4.0
and 7.1.0), it complains thusly:

In function 'dec_stream_footer':
error: dereferencing type-punned pointer will break
  strict-aliasing rules [-Werror=strict-aliasing]
   if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf))
   ^~


A different title might be better.

This is *not* a warning about alignment issue; it is a warning about a 
potential aliasing issue: two pointers of different (non-char) types 
that could point to the same memory.



Daniel.




Despite the arrow pointing to the xz_crc32() call, the actual problem is
the get_le32() macro expansion.

The thing is, the `buf` field was put just after two fields of type
size_t for the express purpose of avoiding alignment issues, as per the
comment above the `temp` struct >
Meaning: GCC gets this all wrong and should not complain.

An earlier attempt to fix (76b65624b (unxz: get_le32
macro is obviously wrong, 2017-08-07)) this by simply forcing
get_unaligned_32() (which is unnecessarily slow because it rebuilds a
32-bit int from accessing already-aligned bytes one by one) was a bit
heavy-handed.

But we can help GCC by turning the get_le32() macro into an inline
function. So let's do that, it also helps compile time safety by making
the code a little bit stricter.

Signed-off-by: Johannes Schindelin 
---
Published-As: 
https://github.com/dscho/busybox-w32/releases/tag/busybox-type-punned-warning-v2
Fetch-It-Via: git fetch https://github.com/dscho/busybox-w32 
busybox-type-punned-warning-v2

Interdiff vs v1:
  diff --git a/archival/libarchive/decompress_unxz.c 
b/archival/libarchive/decompress_unxz.c
  index 0be85500c..e24cff98b 100644
  --- a/archival/libarchive/decompress_unxz.c
  +++ b/archival/libarchive/decompress_unxz.c
  @@ -37,6 +37,11 @@ static uint32_t xz_crc32(const uint8_t *buf, size_t size, 
uint32_t crc)
|| !defined(put_unaligned_be32)
   # error get_unaligned_le32 accessors are not defined
   #endif
  +static ALWAYS_INLINE uint32_t get_le32_fast(const void *p)
  +{
  +  return *(uint32_t *)p;
  +}
  +#define get_le32 get_le32_fast
   
   #include "unxz/xz_dec_bcj.c"

   #include "unxz/xz_dec_lzma2.c"
  diff --git a/archival/libarchive/unxz/xz_dec_stream.c 
b/archival/libarchive/unxz/xz_dec_stream.c
  index 8131ee30a..bf791055b 100644
  --- a/archival/libarchive/unxz/xz_dec_stream.c
  +++ b/archival/libarchive/unxz/xz_dec_stream.c
  @@ -423,15 +423,6 @@ static enum xz_ret XZ_FUNC dec_stream_footer(struct 
xz_dec *s)
if (!memeq(s->temp.buf + 10, FOOTER_MAGIC, FOOTER_MAGIC_SIZE))
return XZ_DATA_ERROR;
   
  -#if defined(__GNUC__)

  - /*
  -  * The temp.buf field is put just after two fields of type size_t for
  -  * the express purpose of avoiding alignment issues. But GCC complains
  -  * about it nevertheless... so: shut GCC up for a few lines.
  -  */
  -#pragma GCC diagnostic push
  -#pragma GCC diagnostic ignored "-Wstrict-aliasing"
  -#endif
if (xz_crc32(s->temp.buf + 4, 6, 0) != get_le32(s->temp.buf))
return XZ_DATA_ERROR;
   
  @@ -442,9 +433,6 @@ static enum xz_ret XZ_FUNC dec_stream_footer(struct xz_dec *s)

 */
if ((s->index.size >> 2) != get_le32(s->temp.buf + 4))
return XZ_DATA_ERROR;
  -#if defined(__GNUC__)
  -#pragma GCC diagnostic pop
  -#endif
   
   	if (s->temp.buf[8] != 0 || s->temp.buf[9] != s->check_type)

return XZ_DATA_ERROR;
  archival/libarchive/decompress_unxz.c | 5 +
  1 file changed, 5 insertions(+)

diff --git a/archival/libarchive/decompress_unxz.c 
b/archival/libarchive/decompress_unxz.c
index 0be85500c..e24cff98b 100644
--- a/archival/libarchive/decompress_unxz.c
+++ b/archival/libarchive/decompress_unxz.c
@@ -37,6 +37,11 @@ static uint32_t xz_crc32(const uint8_t *buf, size_t size, 
uint32_t crc)
   || !defined(put_unaligned_be32)
  # error get_unaligned_le32 accessors are not defined
  #endif
+static ALWAYS_INLINE uint32_t get_le32_fast(const void *p)
+{
+  return *(uint32_t *)p;
+}
+#define get_le32 get_le32_fast
  
  #include "unxz/xz_dec_bcj.c"

  #include "unxz/xz_dec_lzma2.c"

base-commit: a907b828d6e9f1357fc2e1db09d3eb1d3fb9b826



___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox