Re: How do I (unconditionally) enable unicode support in busybox?
Hi James! The odd thing, of course, is that even after the code above runs, unicode support remains disabled unless I had added a lang=utf boot parameter (see below). I'm not sure if I understand your problem completely. Have you ever considered to set an export LANG=en_US.UTF-8 in /etc/profile? This script file gets run whenever ash is started as a login shell (e.g. via getty/login or with a leading dash from /etc/inittab -/bin/sh). Setting this one line enables UTF-8 support for me. For any other shell script, executed during boot, you can just set export LANG=... at the start of the script. So what is wrong with this? -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
RE: ftpd authentication
The authentication works great now. And with code from libbb, nice. But the change_identity function is conflicting with the chroot. Even if you reorder the sequence, the change_identity is still conflicting somehow. I have to study it some more, before I can suggest a solution. (Besides removing change_identity). -Original Message- From: Denys Vlasenko [mailto:vda.li...@googlemail.com] Sent: 5. august 2014 22:00 To: Morten Kvistgaard Cc: busybox@busybox.net Subject: Re: ftpd authentication On Mon, Aug 4, 2014 at 12:38 PM, Morten Kvistgaard MK@pch- engineering.dk wrote: I've attached a patch for adding basic authentication to the ftpd. This used to work with version 1.21.1. And walter harms tested it with 1.22.1. And it worked with trunk 3 months ago. It doesn't seem to work with the current trunk though? The difference lies with getpwnam I think. (It's returning NULL on my Ubuntu.) getpwnam will not be very happy in chroot. I fixed that, and also added actual change of user identity, and refactored password check to not duplicate code. Applied to git, please try it now. For example, I'm curious whether people who want _anon_ ftp are unhappy now -- This message has been scanned for viruses and dangerous content by CronLab (www.cronlab.com), and is believed to be clean. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: ftpd authentication
Here is my final commit about getpwnam (and getpwuid) https://github.com/CyanogenMod/android_external_busybox/commit/d8482dbecc75c8a94bd15a02fa8844cfea91e6d2 maybe the _r functions should be used... http://unixhelp.ed.ac.uk/CGI/man-cgi?getpwuid+3 2014-08-06 12:19 GMT+02:00 Morten Kvistgaard m...@pch-engineering.dk: The authentication works great now. And with code from libbb, nice. But the change_identity function is conflicting with the chroot. Even if you reorder the sequence, the change_identity is still conflicting somehow. I have to study it some more, before I can suggest a solution. (Besides removing change_identity). -Original Message- From: Denys Vlasenko [mailto:vda.li...@googlemail.com] Sent: 5. august 2014 22:00 To: Morten Kvistgaard Cc: busybox@busybox.net Subject: Re: ftpd authentication On Mon, Aug 4, 2014 at 12:38 PM, Morten Kvistgaard MK@pch- engineering.dk wrote: I've attached a patch for adding basic authentication to the ftpd. This used to work with version 1.21.1. And walter harms tested it with 1.22.1. And it worked with trunk 3 months ago. It doesn't seem to work with the current trunk though? The difference lies with getpwnam I think. (It's returning NULL on my Ubuntu.) getpwnam will not be very happy in chroot. I fixed that, and also added actual change of user identity, and refactored password check to not duplicate code. Applied to git, please try it now. For example, I'm curious whether people who want _anon_ ftp are unhappy now -- This message has been scanned for viruses and dangerous content by CronLab (www.cronlab.com), and is believed to be clean. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
On Wed, Aug 06, 2014 at 11:28 AM, Harald Becker said: Have you ever considered to set an export LANG=en_US.UTF-8 in /etc/profile? This script file gets run whenever ash is started as a login shell (e.g. via getty/login or with a leading dash from /etc/inittab -/bin/sh). Setting this one line enables UTF-8 support for me. For any other shell script, executed during boot, you can just set export LANG=... at the start of the script. So what is wrong with this? Thanks for the reply. I already have it working in login shells just fine. The problem is mainly when I use a busybox script as /init inside an initrd (initramfs). It runs as process 1. The /init script is called directly by the bootloader. Its environment is controlled by the command line parameters which are set by the *user*. Worse, in this situation: export LANG=... DOES NOT WORK for changing subsequent unicode behavior. The behavior is set and locked in by the value of LANG in the initial environment which I don't have control over. It works fine from the command line on my development system. I only see the strange behavior when it is run inside an initrd to boot a system which is what I actually want to do. I only run it from the command line on my development system for testing. This is why I am writing to this list. When busybox is called by a bootloader, exporting LANG does not change the unicode behavior. If exporting worked then the original code I posted would work: echo -n $x | LANG=utf sed 's/./x/g' | wc -c but this does not work when I use it in the initrd. The sed in that code ignores its own environment and goes by whatever the very first value of LANG was in the first busybox shell that runs. Perhaps I can phrase it more succinctly as: The command export LANG=... has no effect on subsequent busybox unicode behavior when a busybox script is called directly from a bootloader. The only thing that controls the unicode behavior in this case is the initial value of LANG when the first busybox shell script is called as /init. I admit this is strange, bordering on unbelievable. Especially so because this same problem does not exist when running from the command line. In that case export LANG=... works just as expected for changing subsequent unicode behavior and the line of code above works as expected. Someone else hit this problem back in January: http://lists.busybox.net/pipermail/busybox/2014-June/081021.html They said: Exporting LANG in rcS didnt have an effect They ended up submitting a patch to hard-code the LANG variable into busybox. If export LANG=... had worked as we expect then there would have been no need for that patch to hard-code a default LANG. BTW: I've found an ugly kludge that fixes my problem but does not fix the busybox unicode problem in general. Instead of running: echo -n $x | LANG=utf sed 's/./x/g' | wc -c I can use (effectively): echo -n $x | tr -d '\200-\277' | wc -c This will give the correct length for all utf-8 sequences. The busybox tr does not allow octal to mix with range, -, so all 64 octal sequences must be used. TR='\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217' TR=$TR'\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' TR=$TR'\240\241\242\243\244\245\246\247\250\251\252\253\254\255\256\257' TR=$TR'\260\261\262\263\264\265\266\267\270\271\272\273\274\275\276\277' echo -n $x | tr -d $TR | wc -c I'm still willing to help to find a more general solution to the busybox unicode problem. Peace, James ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
On android, the terminal is utf-8 (only one forced locale)
All is fine except vi cursor with utf-8 data... all is displayed correctly,
even multi-column chinese chars but cursor is not at the right place if
simple utf8 (european accents) is present in the line.
i was trying to implement it, here is a simple implementation of mblen :
+#ifdef BIONIC_UTF8_MBLEN
+static int utf_mblen(char *s) // Char len in bytes
+{
+ int buf_mb_len = 1;
+ if (s == NULL) {
+ return -1;
+ }
+
+ if ((*s 0xF0) == 0xF0)
+ buf_mb_len = 4; // start of 4-char utf series
+ else if ((*s 0xF0) == 0xE0)
+ buf_mb_len = 3; // start of 3-char
+ else if ((*s 0xF0) == 0xC0)
+ buf_mb_len = 2; // start of 2-char
+ else if ((*s 0xC0) == 0x80)
+ buf_mb_len = 0; // utf hidden part
+
+ return buf_mb_len;
+}
+#endif
2014-08-06 14:04 GMT+02:00 James Bowlin bit...@gmail.com:
On Wed, Aug 06, 2014 at 11:28 AM, Harald Becker said:
Have you ever considered to set an export LANG=en_US.UTF-8 in
/etc/profile?
This script file gets run whenever ash is started as a login shell
(e.g. via getty/login or with a leading dash from /etc/inittab
-/bin/sh). Setting this one line enables UTF-8 support for me. For
any other shell script, executed during boot, you can just set
export LANG=... at the start of the script. So what is wrong with
this?
Thanks for the reply. I already have it working in login shells
just fine. The problem is mainly when I use a busybox script
as /init inside an initrd (initramfs). It runs as process 1.
The /init script is called directly by the bootloader. Its
environment is controlled by the command line parameters which
are set by the *user*. Worse, in this situation:
export LANG=...
DOES NOT WORK for changing subsequent unicode behavior. The
behavior is set and locked in by the value of LANG in the initial
environment which I don't have control over. It works fine from the
command line on my development system. I only see the strange behavior
when it is run inside an initrd to boot a system which is what I
actually want to do. I only run it from the command line on my
development system for testing.
This is why I am writing to this list. When busybox is called by
a bootloader, exporting LANG does not change the unicode behavior.
If exporting worked then the original code I posted would work:
echo -n $x | LANG=utf sed 's/./x/g' | wc -c
but this does not work when I use it in the initrd. The sed in
that code ignores its own environment and goes by whatever the
very first value of LANG was in the first busybox shell that
runs.
Perhaps I can phrase it more succinctly as:
The command export LANG=... has no effect on subsequent
busybox unicode behavior when a busybox script is called directly
from a bootloader. The only thing that controls the unicode
behavior in this case is the initial value of LANG when the
first busybox shell script is called as /init.
I admit this is strange, bordering on unbelievable. Especially
so because this same problem does not exist when running from
the command line. In that case export LANG=... works just
as expected for changing subsequent unicode behavior and the
line of code above works as expected.
Someone else hit this problem back in January:
http://lists.busybox.net/pipermail/busybox/2014-June/081021.html
They said:
Exporting LANG in rcS didnt have an effect
They ended up submitting a patch to hard-code the LANG variable
into busybox. If export LANG=... had worked as we expect then
there would have been no need for that patch to hard-code a
default LANG.
BTW: I've found an ugly kludge that fixes my problem but does
not fix the busybox unicode problem in general. Instead of running:
echo -n $x | LANG=utf sed 's/./x/g' | wc -c
I can use (effectively):
echo -n $x | tr -d '\200-\277' | wc -c
This will give the correct length for all utf-8 sequences. The
busybox tr does not allow octal to mix with range, -, so all 64
octal sequences must be used.
TR='\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217'
TR=$TR'\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237'
TR=$TR'\240\241\242\243\244\245\246\247\250\251\252\253\254\255\256\257'
TR=$TR'\260\261\262\263\264\265\266\267\270\271\272\273\274\275\276\277'
echo -n $x | tr -d $TR | wc -c
I'm still willing to help to find a more general solution to the
busybox unicode problem.
Peace, James
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox
___
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
Hi James! The problem is mainly when I use a busybox script as /init inside an initrd (initramfs). It runs as process 1. The /init script is called directly by the bootloader. Its environment is controlled by the command line parameters which are set by the *user*. Worse, in this situation: export LANG=... DOES NOT WORK for changing subsequent unicode behavior. Why shall it not work in this situation? It works for me. At the start of the init script I use export LANG=... whatever I need. As it is exported it goes to the environment of every process started from init script, so UTF-8 support is kept. The behavior is set and locked in by the value of LANG in the initial environment which I don't have control over. No control? Why? It is usally not needed, but in case you really want to achieve this: first init script: export LANG=... exec /bin/sh second_init_script MAY EVEN SET COMMAND LINE ARGUMENTS second init script runs as process 1 with environment set as you like. It works fine from the command line on my development system. I only see the strange behavior when it is run inside an initrd to boot a system which is what I actually want to do. I only run it from the command line on my development system for testing. initrd? Do you have one of those systems which have an initial system in kernel and then load another initrd/system image. Here the problem is YOUR init script is not the first script run. There is already a script or program controlling your startup. So you need to look, what this script/program may do to your environment. This is not a Busybox failure, as BB ash is not in control from the beginning of the system startup. May be you hit this case, check it out, please. This is why I am writing to this list. When busybox is called by a bootloader, exporting LANG does not change the unicode behavior. Why shall it not work? It works, if called correct. I used it in so many situations, can't remember I had any trouble with this. If exporting worked then the original code I posted would work: echo -n $x | LANG=utf sed 's/./x/g' | wc -c Argh ... this kind of exporting makes trouble with ash (and many other Unix shells, not all behave correct as expected). Always set it at the beginning of your script: export LANG=en_US.utf8 echo -n $x | sed 's/./x/g' | wc -c shall give sed a LANG variable which enables UTF-8 ... but keep an eye on not be catched by the subshell trap. but this does not work when I use it in the initrd. The sed in that code ignores its own environment and goes by whatever the very first value of LANG was in the first busybox shell that runs. Use export to set environment values at start of your script Perhaps I can phrase it more succinctly as: The command export LANG=... has no effect on subsequent busybox unicode behavior when a busybox script is called directly from a bootloader. The only thing that controls the unicode behavior in this case is the initial value of LANG when the first busybox shell script is called as /init. give exact examples on how you call your script, and your scripts. You are doing something ill. Busybox ash works pretty good as init script and allows to control every aspect. I'm still willing to help to find a more general solution to the busybox unicode problem. First give absolute and complete examples on how you call your scripts and your complete scripts. It looks to me you are not calling everything in the correct order. If there has not gone a very new bug in one of the newer versions, ash shall behave correct. -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
Hi ! All is fine except vi cursor with utf-8 data... all is displayed correctly, even multi-column chinese chars but cursor is not at the right place if simple utf8 (european accents) is present in the line. This depends on what lib Busybox is linked with. Some libs have only limited UTF-8 support and handle only the most common UTF-8 range. -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: ftpd authentication
On Wednesday 06 August 2014 13:58:31 Tanguy Pruvot wrote: Here is my final commit about getpwnam (and getpwuid) https://github.com/CyanogenMod/android_external_busybox/commit/d8482dbecc75c8a94bd15a02fa8844cfea91e6d2 Hi, just out of curiosity why are you returning an empty passwd field? Isn't it a sort of security risk, shoudn't it be better change it to x I rember the null passwd field was a problem when I hacked on the HTC dream and dropbear if set up for password authentication and linked with bionic allowed root access by simply hitting enter. There must a bug report on the cyanogenmod bugzilla about it. Rather than patching busybox and every other app you port wouldn't it be easier to patch bionic directly to return always x in the passwd field as android as far as i know doesn't use passwords. Ciao, Tito maybe the _r functions should be used... http://unixhelp.ed.ac.uk/CGI/man-cgi?getpwuid+3 2014-08-06 12:19 GMT+02:00 Morten Kvistgaard m...@pch-engineering.dk: The authentication works great now. And with code from libbb, nice. But the change_identity function is conflicting with the chroot. Even if you reorder the sequence, the change_identity is still conflicting somehow. I have to study it some more, before I can suggest a solution. (Besides removing change_identity). -Original Message- From: Denys Vlasenko [mailto:vda.li...@googlemail.com] Sent: 5. august 2014 22:00 To: Morten Kvistgaard Cc: busybox@busybox.net Subject: Re: ftpd authentication On Mon, Aug 4, 2014 at 12:38 PM, Morten Kvistgaard MK@pch- engineering.dk wrote: I've attached a patch for adding basic authentication to the ftpd. This used to work with version 1.21.1. And walter harms tested it with 1.22.1. And it worked with trunk 3 months ago. It doesn't seem to work with the current trunk though? The difference lies with getpwnam I think. (It's returning NULL on my Ubuntu.) getpwnam will not be very happy in chroot. I fixed that, and also added actual change of user identity, and refactored password check to not duplicate code. Applied to git, please try it now. For example, I'm curious whether people who want _anon_ ftp are unhappy now -- This message has been scanned for viruses and dangerous content by CronLab (www.cronlab.com), and is believed to be clean. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
On Wed, Aug 06, 2014 at 02:51:21PM +0200, Harald Becker wrote: Hi James! The problem is mainly when I use a busybox script as /init inside an initrd (initramfs). It runs as process 1. The /init script is called directly by the bootloader. Its environment is controlled by the command line parameters which are set by the *user*. Worse, in this situation: export LANG=... DOES NOT WORK for changing subsequent unicode behavior. Why shall it not work in this situation? It works for me. At the start of the init script I use export LANG=... whatever I need. As it is exported it goes to the environment of every process started from init script, so UTF-8 support is kept. Perhaps the issue is that it doesn't work in the initial shell itself -- for example, glob bracket expressions treat a character as a set of bytes and match any one of those bytes rather than the character. The behavior is set and locked in by the value of LANG in the initial environment which I don't have control over. No control? Why? It is usally not needed, but in case you really want to achieve this: first init script: export LANG=... exec /bin/sh second_init_script MAY EVEN SET COMMAND LINE ARGUMENTS second init script runs as process 1 with environment set as you like. I agree this is a good workaround. If exporting worked then the original code I posted would work: echo -n $x | LANG=utf sed 's/./x/g' | wc -c Argh ... this kind of exporting makes trouble with ash (and many other Unix shells, not all behave correct as expected). Always set it at the beginning of your script: Are you sure? It's well-documented, required by the standard, and has always worked fine with ash for me. I would consider it the preferred way of exporting when you don't want to affect subsequent commands or the state of the current shell, only the current command. Rich ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
Hi Rich! echo -n $x | LANG=utf sed 's/./x/g' | wc -c Argh ... this kind of exporting makes trouble with ash (and many other Unix shells, not all behave correct as expected). Always set it at the beginning of your script: Are you sure? It's well-documented, required by the standard, and has always worked fine with ash for me. It shall work, but BB ash has a bug, which triggers randomly and just ignores those environment settings ahead of a command. I can't tell when this happens exactly, but to avoid this using export always helped and worked fine for me. I would consider it the preferred way of exporting when you don't want to affect subsequent commands or the state of the current shell, only the current command. I know. Would be fine to find the reason for this very long standing bug. -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: How do I (unconditionally) enable unicode support in busybox?
Hi Rich! Perhaps the issue is that it doesn't work in the initial shell itself -- for example, glob bracket expressions treat a character as a set of bytes and match any one of those bytes rather than the character. Ok I see the problem ... and I know what's the reason: setlocale is called at the start of the shell. Changing the value afterwards does not effect locale settings in the shell itself. This would need a setlocale after assigning a new value to the variable LANG. So if you need this handling a restart of process 1 is required. Example how to do, given below: first init script: export LANG=... exec /bin/sh second_init_script MAY EVEN SET COMMAND LINE ARGUMENTS second init script runs as process 1 with environment set as you like. -- Harald ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
Re: ftpd authentication
Its not a security risk, there is no passwords in android and busybox is not suid... so only root can do it. Also selinux has rules to prevent this kind of stuff. These applets are not enabled in busybox neither but could be later. ex: login system or login media can help to debug permission problems in a service. Le 6 août 2014 15:33, tito farmat...@tiscali.it a écrit : On Wednesday 06 August 2014 13:58:31 Tanguy Pruvot wrote: Here is my final commit about getpwnam (and getpwuid) https://github.com/CyanogenMod/android_external_busybox/commit/d8482dbecc75c8a94bd15a02fa8844cfea91e6d2 Hi, just out of curiosity why are you returning an empty passwd field? Isn't it a sort of security risk, shoudn't it be better change it to x I rember the null passwd field was a problem when I hacked on the HTC dream and dropbear if set up for password authentication and linked with bionic allowed root access by simply hitting enter. There must a bug report on the cyanogenmod bugzilla about it. Rather than patching busybox and every other app you port wouldn't it be easier to patch bionic directly to return always x in the passwd field as android as far as i know doesn't use passwords. Ciao, Tito maybe the _r functions should be used... http://unixhelp.ed.ac.uk/CGI/man-cgi?getpwuid+3 2014-08-06 12:19 GMT+02:00 Morten Kvistgaard m...@pch-engineering.dk: The authentication works great now. And with code from libbb, nice. But the change_identity function is conflicting with the chroot. Even if you reorder the sequence, the change_identity is still conflicting somehow. I have to study it some more, before I can suggest a solution. (Besides removing change_identity). -Original Message- From: Denys Vlasenko [mailto:vda.li...@googlemail.com] Sent: 5. august 2014 22:00 To: Morten Kvistgaard Cc: busybox@busybox.net Subject: Re: ftpd authentication On Mon, Aug 4, 2014 at 12:38 PM, Morten Kvistgaard MK@pch- engineering.dk wrote: I've attached a patch for adding basic authentication to the ftpd. This used to work with version 1.21.1. And walter harms tested it with 1.22.1. And it worked with trunk 3 months ago. It doesn't seem to work with the current trunk though? The difference lies with getpwnam I think. (It's returning NULL on my Ubuntu.) getpwnam will not be very happy in chroot. I fixed that, and also added actual change of user identity, and refactored password check to not duplicate code. Applied to git, please try it now. For example, I'm curious whether people who want _anon_ ftp are unhappy now -- This message has been scanned for viruses and dangerous content by CronLab (www.cronlab.com), and is believed to be clean. ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox ___ busybox mailing list busybox@busybox.net http://lists.busybox.net/mailman/listinfo/busybox
