Re: About std.ascii.toLower
On 20/09/12 18:57, Jonathan M Davis wrote:
On Thursday, September 20, 2012 18:35:21 bearophile wrote:
monarch_dodra:
It's not, it only *operates* on ASCII, but non ascii is still a
legal arg:
Then maybe std.ascii.toLower needs a pre-condition that
constraints it to just ASCII inputs, so it's free to return a
char.
Goodness no.
1. Operating on a char is almost always the wrong thing to do. If you really
want to do that, then cast. It should _not_ be encouraged.
2. It would be disastrous if std.ascii's funtions didn't work on unicode.
Right now, you can use them with ranges on strings which are unicode, which
can be very useful.
> I grant you that that's more obvious with something like
isDigit than toLower, but regardless, std.ascii is designed such that its
functions will all operate on unicode strings. It just doesn't alter unicode
characters and returns false for them with any of the query functions.
Are there any use cases of toLower() on non-ASCII strings?
Seriously? I think it's _always_ a bug.
At the very least that function should have a name like
toLowerIgnoringNonAscii() to indicate that it is performing a really,
really foul operation.
The fact that toLower("Ü") doesn't generate an error, but doesn't return
"ü" is a wrong-code bug IMHO. It isn't any better than if it returned a
random garbage character (eg, it's OK in my opinion for ASCII toLower to
consider only the lower 7 bits).
OTOH I can see some value in a cased ASCII vs unicode comparison.
ie, given an ASCII string and a unicode string, do a case-insensitive
comparison, eg look for
"" inside "öähaøſ€đ@ſŋħŋ€ł¶"
Re: About std.ascii.toLower
On Friday, September 21, 2012 14:10:25 monarch_dodra wrote: > I did not know conv's to did cast validation. For conversions which can be done with both casting and std.conv.to, std.conv.to does runtime checks wherever a narrowing conversion would take place and throws if the conversion would lose precision. - Jonathan M Davis
Re: About std.ascii.toLower
On Friday, 21 September 2012 at 11:25:54 UTC, Jonathan M Davis wrote: On Friday, September 21, 2012 13:18:01 monarch_dodra wrote: Related, could "toChar" be considered for inclusion? I think it would be a convenient tool for validation. I certainly would be against adding it. I think that it's a relatively uncommon use case and considering how easy it is to just write the function yourself, the functionality gain is minimal. I just don't think that it carries it's weight as far as the standard library goes. But I don't know how others like Andrei would feel. That said, if we go that way, we might as well just have a more generic safeCast in std.conv or something: You're basically asking for a version of std.conv.to which uses assertions instead of exceptions. - Jonathan M Davis I know my ideas usually get shot down, but I usually learn a LOT from your answers, so sorry for insisting. I did not know conv's to did cast validation. In my defense, the doc is actually missing: http://dlang.org/phobos/std_conv.html I made a doc pull request so that it would appear. https://github.com/D-Programming-Language/phobos/pull/811
Re: About std.ascii.toLower
On Friday, September 21, 2012 13:18:01 monarch_dodra wrote: > Related, could "toChar" be considered for inclusion? I think it > would be a convenient tool for validation. I certainly would be against adding it. I think that it's a relatively uncommon use case and considering how easy it is to just write the function yourself, the functionality gain is minimal. I just don't think that it carries it's weight as far as the standard library goes. But I don't know how others like Andrei would feel. > That said, if we go that way, we might as well just have a more > generic safeCast in std.conv or something: You're basically asking for a version of std.conv.to which uses assertions instead of exceptions. - Jonathan M Davis
Re: About std.ascii.toLower
On Friday, 21 September 2012 at 10:45:42 UTC, Jonathan M Davis
wrote:
On Friday, September 21, 2012 12:38:07 monarch_dodra wrote:
On Friday, 21 September 2012 at 10:23:39 UTC, Jonathan M Davis
wrote:
> On Friday, September 21, 2012 11:00:31 monarch_dodra wrote:
>> What do you (you two) think of my proposition for a
>> "std.strictascii" module?
>
> I don't think that it's at all worth it. It's just duplicate
> functionality in
> order to avoid a cast.
(and contract)
If that's what you want, it's easy enough to create a helper
function which
you use instead of a cast which does the contract check as
well. e.g.
char toChar(dchar c)
{
assert(isAscii(c));
return cast(char)c;
}
foreach(ref char c; str)
c = toChar(std.ascii.toLower(c));
It should be completely optimized out with -release and -inline.
- Jonathan M Davis
That's a real good idea. Also, I find it is these kinds of
situations where UFCS really shines (IMO):
foreach(i, c; s1)
cs[i] = c.toUpper().toChar();
I love this syntax.
Related, could "toChar" be considered for inclusion? I think it
would be a convenient tool for validation.
/*
* Casts dchar to a char.
*
* Preconditions:
* $(D c) must be representable in a single char.
*/
char toChar(dchar c)
{
assert(c < 256, "toChar: Input too large for char");
return cast(char)c;
}
That said, if we go that way, we might as well just have a more
generic safeCast in std.conv or something:
T safeCast(T, U)(U i)
if(isBasicType!T && isBasicType!U)
{
assert(cast(T)i == i, "safeCast: Cast failed");
return cast(T)i;
}
foreach(i, c; s1)
cs[i] = c.toUpper().safeCast!char();
Hum... yeah... I don't know...
I seem to be typing faster than I can really think of the
consequences of such a function.
Re: About std.ascii.toLower
On Friday, September 21, 2012 12:38:07 monarch_dodra wrote:
> On Friday, 21 September 2012 at 10:23:39 UTC, Jonathan M Davis
>
> wrote:
> > On Friday, September 21, 2012 11:00:31 monarch_dodra wrote:
> >> What do you (you two) think of my proposition for a
> >> "std.strictascii" module?
> >
> > I don't think that it's at all worth it. It's just duplicate
> > functionality in
> > order to avoid a cast.
>
> (and contract)
If that's what you want, it's easy enough to create a helper function which
you use instead of a cast which does the contract check as well. e.g.
char toChar(dchar c)
{
assert(isAscii(c));
return cast(char)c;
}
foreach(ref char c; str)
c = toChar(std.ascii.toLower(c));
It should be completely optimized out with -release and -inline.
- Jonathan M Davis
Re: About std.ascii.toLower
On Friday, 21 September 2012 at 10:23:39 UTC, Jonathan M Davis wrote: On Friday, September 21, 2012 11:00:31 monarch_dodra wrote: What do you (you two) think of my proposition for a "std.strictascii" module? I don't think that it's at all worth it. It's just duplicate functionality in order to avoid a cast. (and contract) - Jonathan M Davis Somehow, I expected that reply, but I had to ask anyways :D Thanks.
Re: About std.ascii.toLower
On Friday, September 21, 2012 11:00:31 monarch_dodra wrote: > What do you (you two) think of my proposition for a > "std.strictascii" module? I don't think that it's at all worth it. It's just duplicate functionality in order to avoid a cast. - Jonathan M Davis
Re: About std.ascii.toLower
On Thursday, 20 September 2012 at 17:32:52 UTC, bearophile wrote:
Jonathan M Davis:
Functions which operate on ASCII characters. All of the
functions in std.ascii accept unicode characters but
effectively ignore them. All isX functions return false for
unicode characters, and all toX functions do nothing to unicode
characters.<
So now I'd like a new set of functions designed for ASCII text,
with contracts to refuse not-ASCII things ;-)
Thank you for the answers Jonathan.
Bye,
bearophile
What do you (you two) think of my proposition for a
"std.strictascii" module?
The signatures would be:
char toLower(dchar c);
And the implementations be like:
char toLower(dchar c)
in
{
assert(c.std.ascii.isAscii());
}
body
{
cast(char) c.std.ascii.toLower();
}
The rational for taking a dchar as input is so that it's own
input can be correctly validated, and so that it can easilly
operate with foreach etc, doing the cast internally. The returned
value would be pre-cast to char.
Usage:
import std.stdio;
import std.strictascii;
void main(){
string s1 = "axbacxf";
string s2 = "àxbécxf";
char[] cs = new char[](7);
//bearophile use case: no casts
foreach(i, c; s1)
cs[i] = c.toUpper();
//illegal use case: correct input validation
foreach(i, c; s1)
cs[i] = c.toUpper(); //in assert
}
It doesn't add *much* functionality, and arguably, it is a
specialized functionality, but there are usecases where you want
to operate ONLY on ascii, as pointed out by bearophile.
Just curious if I should even consider investing some effort in
this.
Re: About std.ascii.toLower
Jonathan M Davis: Goodness no. :-) 1. Operating on a char is almost always the wrong thing to do. A single char is often not so useful but I have to keep many mutable chars, keeping them as char[] instead of dchar[] saves both memory and reduces cache misses. The same is true for types like short or float, single ones are not so useful, but they sometimes become useful when you have many of them in arrays. If I have to modify such char[], using toUpper() requires me a cast. And in my opinion it's not a good idea to return a dchar if you know the both the input and output of the function are a char. If you really want to do that, then cast. On the other hand casts in D have a certain risk, so reducing their number as much as possible is a good idea. It should _not_ be encouraged. This is silly, see the above explanation. 2. It would be disastrous if std.ascii's funtions didn't work on unicode. Right now, you can use them with ranges on strings which are unicode, which can be very useful. [...] but regardless, std.ascii is designed such that its functions will all operate on unicode strings. It just doesn't alter unicode characters and returns false for them with any of the query functions. I see, and I didn't know this, I have misunderstood. I have thought of std.ascii functions as functions meant to work on just ASCII characters/text. But they are better defined as Unicode-passing functions. And yeah, it's written at the top of the module: Functions which operate on ASCII characters. All of the functions in std.ascii accept unicode characters but effectively ignore them. All isX functions return false for unicode characters, and all toX functions do nothing to unicode characters.< So now I'd like a new set of functions designed for ASCII text, with contracts to refuse not-ASCII things ;-) Thank you for the answers Jonathan. Bye, bearophile
Re: About std.ascii.toLower
On Thursday, 20 September 2012 at 17:05:18 UTC, bearophile wrote: monarch_dodra: Would that actually change anything though? I mean what with alignment and everything, wouldn't returning a char be just as expansive? I'm not 100% sure. If you are thinking about the number of operations, then it's the same, as both a char and dchar value go in a register. The run time is the same, especially after inlining. What is your use case that would require this? I have a char[] like: ['a','x','b','a','c','x','f'] Every char encodes something. Putting it to upper case means that that data was already used: ['a','X','b','a','C','x','f'] In this case to use toUpper I have to use: cast(char)toUpper(foo[1]) What's I am trying to minimize is the number of cast(). On the other hand even in C toupper returns a type larger than char: http://www.acm.uiuc.edu/webmonkeys/book/c_guide/2.2.html It's just D has contract programming, and this module is written for ASCII, so it's able to be smarter than C functions, and return a char. Bye, bearophile That's what I thought. You have a valid point (IMO) but at the same time, using the ASCII methods on non-ascii characters is also legit operation. I guess we'd need the extra "std.strictascii" module (!) for operations that would accept ASCII char, and return a ASCII char. I'd support such an ER, I think it would. Allow users (such as you) to have tighter constraints if needed, while still keeping std.ascii for "safer" ASCII operations.
Re: About std.ascii.toLower
On Thursday, September 20, 2012 18:35:21 bearophile wrote: > monarch_dodra: > > It's not, it only *operates* on ASCII, but non ascii is still a > > > legal arg: > Then maybe std.ascii.toLower needs a pre-condition that > constraints it to just ASCII inputs, so it's free to return a > char. Goodness no. 1. Operating on a char is almost always the wrong thing to do. If you really want to do that, then cast. It should _not_ be encouraged. 2. It would be disastrous if std.ascii's funtions didn't work on unicode. Right now, you can use them with ranges on strings which are unicode, which can be very useful. I grant you that that's more obvious with something like isDigit than toLower, but regardless, std.ascii is designed such that its functions will all operate on unicode strings. It just doesn't alter unicode characters and returns false for them with any of the query functions. - Jonathan M Davis
Re: About std.ascii.toLower
monarch_dodra: Would that actually change anything though? I mean what with alignment and everything, wouldn't returning a char be just as expansive? I'm not 100% sure. If you are thinking about the number of operations, then it's the same, as both a char and dchar value go in a register. The run time is the same, especially after inlining. What is your use case that would require this? I have a char[] like: ['a','x','b','a','c','x','f'] Every char encodes something. Putting it to upper case means that that data was already used: ['a','X','b','a','C','x','f'] In this case to use toUpper I have to use: cast(char)toUpper(foo[1]) What's I am trying to minimize is the number of cast(). On the other hand even in C toupper returns a type larger than char: http://www.acm.uiuc.edu/webmonkeys/book/c_guide/2.2.html It's just D has contract programming, and this module is written for ASCII, so it's able to be smarter than C functions, and return a char. Bye, bearophile
Re: About std.ascii.toLower
On Thursday, 20 September 2012 at 16:34:22 UTC, bearophile wrote: monarch_dodra: It's not, it only *operates* on ASCII, but non ascii is still a legal arg: Then maybe std.ascii.toLower needs a pre-condition that constraints it to just ASCII inputs, so it's free to return a char. Bye, bearophile I was thinking the exact same thing right after replying actually. Would that actually change anything though? I mean what with alignment and everything, wouldn't returning a char be just as expansive? I'm not 100% sure. What is your use case that would require this?
Re: About std.ascii.toLower
monarch_dodra: It's not, it only *operates* on ASCII, but non ascii is still a legal arg: Then maybe std.ascii.toLower needs a pre-condition that constraints it to just ASCII inputs, so it's free to return a char. Bye, bearophile
About std.ascii.toLower
Sorry, the thread title was "About std.ascii.toLower"...
