Re: POSIX character classes

2017-03-24 Thread Thorsten Glaser
Martijn Dekker dixit:

>> Can I get by making them match ASCII only even in UTF-8 mode?
>
>IMHO, that would defeat their primary purpose, namely locale-dependent
>class matching, so no, not really. :)
>
>If Greeks or Russians (or Germans, for that matter) can't count on
>[:upper:] matching an upper case letter in their alphabets, then I'd say

There’s no alphabets in UTF-8, only global Unicode.

>> Strictly speaking, POSIX requires only support for the C locale,
>[...]
>
>Yes, but on systems supporting other locales (e.g. UTF-8), it would not
>be conforming for character classes to match ASCII only. You either
>support UTF-8 or you don't.

For POSIX purposes, we really don’t, as we use our own routines
to read and write multibyte characters and handle them as wide
characters internally. We _really_ cannot use POSIX locales in
mksh at all. So if a system has 32-bit wchar_t and supports the
Unicode astral planes, mksh isn’t conforming in UTF-8 mode there
either. (POSIX does, however, not demand UTF-8 or Unicode support
at all, only the C locale, so that’s okay.)


The question was more whether [[:upper:]] matching [A-Z] would
be more useful than not matching anything at all.

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2


[Bug 1675842] [NEW] ^O vs. modified command lines

2017-03-24 Thread Thorsten Glaser
Public bug reported:

$ echo a
$ echo b
 should put “echo b” into the buffer (like 
) but doesn’t.

** Affects: mksh
 Importance: Wishlist
 Assignee: Thorsten Glaser (mirabilos)
 Status: Triaged

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1675842

Title:
  ^O vs. modified command lines

Status in mksh:
  Triaged

Bug description:
  $ echo a
  $ echo b
   should put “echo b” into the buffer (like 
) but doesn’t.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1675842/+subscriptions


Re: POSIX character classes (was Re: pipes and sub-shells)

2017-03-24 Thread Martijn Dekker
Op 23-03-17 om 22:02 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> * BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>> [expressions] are not supported in glob patterns.
> 
> I really really REALLY hate that this will make mksh really big.
> We’re talking about 36K .rodata even without titlecase conversion
> and BMP-only (16-bit Unicode) here.

I sympathise.

Even fnmatch(3) is not compliant on all systems; the BSDs don't seem to
have caught up yet. :(  I don't suppose using that is an option in any
case because of mksh's extended globbing functionality.

Is adding 36k really that much in 2017? On my system, the current
development binary of mksh is 283k after stripping when built with -O2,
235k with -Os. Adding 36k would make it 316k/271k, still quite small.

If that's too much, I guess you should continue to not support them. The
reason modernish detects BUG_NOCHCLASS is not to make some sort of
statement, but to enable programs using the library to easily check for
the presence of the issue and implement alternative methods (such as
falling back to external commands, or just matching ASCII only without
character classes).

> Can I get by making them match ASCII only even in UTF-8 mode?

IMHO, that would defeat their primary purpose, namely locale-dependent
class matching, so no, not really. :)

If Greeks or Russians (or Germans, for that matter) can't count on
[:upper:] matching an upper case letter in their alphabets, then I'd say
for them it would be better to have no support than broken support.

> Strictly speaking, POSIX requires only support for the C locale,
[...]

Yes, but on systems supporting other locales (e.g. UTF-8), it would not
be conforming for character classes to match ASCII only. You either
support UTF-8 or you don't.

- M.