>
>
>
> Martin Sebor wrote:
>
> Travis Vitek wrote:
>>
>>
>>> From: Apache Wiki [mailto:[EMAIL PROTECTED]
>>>
>>> The new
>>> interface will need to make it easy to specify such a set of
>>> locales without explicitly naming them, and it will need to
>>> retrieve such locales without returning duplicates.
>>>
>>
>> As mentioned before I don't know a good way to avoid duplicates other
>> than to compare every attribute of each facet of each locale to all of
>> the other locales. Just testing to see if the return from setlocale() is
>> the same as the input string is not enough. The user could have intalled
>> locales that have unique names but are copies of the data from some
>> other locale.
>
> True, but we don't care about how long the test might run on
> some user's system. What we care about here is that *we* don't
> run tests unnecessarily on our own build servers, and we can
> safely make the simplifying assumption that there are no user
> defined locales installed on them.
>
>>
>>> The interface should make it easy to
>>> express conjunction, disjunction, and negation of the terms
>>> (parameters) and support (a perhaps simplified version of)
>>> [http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_cha
>>> p09.html#tag_09_03 Basic Regular Expression] syntax.
>>
>> Conjunction, disjunction and negation? Are you saying you want to be
>> able to select all locales that are _not_ in some set, something like
>> you would get with a caret (^} in a grep expression?
>
> No, I meant something simple like grep -v.
>
Okay, so this is an all-or-none type negation. I understand that, but I'm
not sure if it is necessary given the objective.
>
>>
>> I'm hoping that I'm just misunderstanding your comments. If not, then
>> this is news to me and I'm a bit curious just how this addition is
>> necessary to minimize the number of locales tested [i.e. the objective].
>
> It may not be necessary. I included it for completeness, thinking
> if it wasn't already there it could be easily added in the form
> of an argument of the function. If it isn't there we can leave
> it out until we need it.
>
>>
>>> We've
>>> decided to use shell brace expansion as a means of expressing
>>> logical conjunction between terms: a valid brace expression is
>>> expanded to obtain a set of terms implicitly connected by a
>>> logical AND. Individual ('\n'-separated) lines of the query
>>> string are taken to be implicitly connected by a logical OR.
>>> This approach models the
>>> [http://www.opengroup.org/onlinepubs/009695399/utilities/grep.h
>>> tml grep] interface with each line loosely corresponding to
>>> the argument of the `-e` option to `grep`.
>>>
>>
>> I've seen you mention the '\n' seperated list thing before, but I still
>> can't make sense of it. Are you saying
>
> In my mind the query expression consists of terms connected
> by operators for conjunction, disjunction (let's forget about
> negation for now). E.g., like so:
>
> qry-expr ::= <dis-expr>
> dis-expr ::= <con-expr> | <dis-expr> <NL> <con-expr>
> con-expr ::= <term> | <term> <SP> <con-expr>
>
> For example:
>
> "foo bar" is a con-expr of the terms "foo" and "bar" denoting
> the intersection of foo and bar, and
>
> "123 xyz\nKLM" is a dis-expr of the terms "123 xyz" and "KLM"
> denoting the union of the the two terms. "123 xyz" is itself
> a con-expr denoting the intersection of 123 and xyz.
>
>> that to select `en_US.*' with a 1
>> byte encoding or `zh_*.UTF-8' with a 2, 3, or 4 byte encoding, I would
>> write the following query?
>
> I think it might be simpler to keep things abstract but given my
> specification above a simple query string would look like this:
>
> "en_US.* 1\n"
> "zh_*.UTF-8 2\n"
> "zh_*.UTF-8 3\n"
> "zh_*.UTF-8 4\n"
>
> for the equivalent of:
>
> locale == "en_US.*" && MB_CUR_MAX == 1
> || locale == "zh_*.UTF-8" && MB_CUR_MAX == 2
> || locale == "zh_*.UTF-8" && MB_CUR_MAX == 3
> || locale == "zh_*.UTF-8" && MB_CUR_MAX == 4
>
I'm totally confused. If we're going to write out each of the expansions,
then why did I take the time to implement brace expansion?
I thought the idea was to allow us to select locales using a brace expanded
query string. If we are explicitly writing out each of the names, then we
wasted a bunch of time writing brace expansion code.
> I'm not sure how you could use brace expressions here. Maybe it
> should be the other way around (<SP> should be OR and <NL> AND).
> But then the grep -e idea is out the window.
Well, if we're going down the road of rewriting this _again_ then how
about using something like '&&' and '||', or even 'and' and 'or' for the
logical operations and then '(' and ')' for grouping? Almost like the
'equivalent of' that you wrote above. Something that is readable by a
C/C++ programmer or the average guy off of the street?
The truth is that not every guy knows grep, and I'm sure that those who
do wouldn't expect to see a grammar that used '\n' and ' ' to represent
logical operations.
> Or maybe we need
> a way to denote/group terms. Then we might be able to say:
>
> "en_US.* 1\n"
> "zh_*.UTF-8 ({2..4})"
>
> expand it to
>
> "en_US.* 1\n"
> "zh_*.UTF-8 (2 3 4)"
>
> and "know" that the spaces in "2 3 4" denote OR even though the
> space in "zh_*.UTF-8 (" denotes an AND. Ugh. I'm not sure I lik
> this all that much better.
>
>> const char* locales =
>> rw_locale_query ("en_US.* 1\nzh_*.UTF-8 {2..4}", 10);
>>
>> I don't see why that would be necessary. You can do it with the
>> following query using normal brace expansion, and it's human readable.
>>
>> const char* locales =
>> rw_locale_query ("{en_US.* 1,zh_*.UTF-8 {2..4}}", 10);
>
> What's "{en_US.* 1,zh_*.UTF-8 {2..4}}" supposed to expand to?
> Bash 3.2 doesn't expand it. I suppose it could be
> "en_US.* 1 zh_*.UTF-8 2 3 4" or
> "en_US.* 1 zh_*.UTF-8 2 zh_*.UTF-8 3 zh_*.UTF-8 4"
I believe that this is _exactly_ what you suggested in our meetings when
I was in Boulder the last time. Maybe I'm just confused, but I am pretty
sure that was what was presented.
The shell does whitespace collapse and tokenization before it does the
expansion. To use whitespace in a brace expansion in the shell you have
to escape it.
So the following expansion should work just fine in csh...
{en_US.*\ 1,zh_*.UTF-8\ {2..4}}
It should expand to
en_US.* 1
zh_*.UTF-8 2
zh_*.UTF-8 3
zh_*.UTF-8 4
Remember that I originally provided rw_brace_expand() that doesn't do all
of that. It treats whitespace like any other character. Of course if you
insist on 100% compatibility with shell brace expansion, then feel free to
escape the spaces. Personally I prefer strings without escapes.
>
> Either way I think I'm getting confused by the lack of distinction
> between what's OR and what's AND.
>
I give an example above of how a brace expansion already solves the
problem.
If the brace expansion routine I've written returns a null terminated
buffer of null terminated strings that are the brace expansions and we
have a function for doing primitive string matching [rw_fnmatch], then
this is a pretty simple problem to solve.
This is exactly what you are doing with the xfail.txt thing. The platform
string is just a brace expansion and grep-like expression...
aix-5.3-*-vacpp-9.0-{12,15}?
Why can't ours be seperated by spaces, or some other character? Is it so
different?
I suppose the big difference is that the format above is rigid and well
defined, whereas the locale match format is still in flux.
>>
>> I know that the '\n' is how you'd use `grep -e', but does it really make
>> sense? We aren't using `grep -e' here.
>
> I'm trying to model the interface on something we all know how
> to use. grep -e seemed the closest example of an known interface
> that would let us do what we want that I could think of.
>
> Maybe it would help to go back to the basics and try to approach
> this by analyzing the problem first. Would putting this together
> be helpful?
>
That depends on how you define helpful. It will not be helpful in getting
this task done in reasonable time. It may be helpful in convincing me to
reimplement this functionality for a third time.
>
> 1. the set of locale attributes we want to keep track of
> in our locale "database"
>
What details are necessary to reduce the number of locales tested? The
honest answer to this is _none_. We could just pick N random locales and
run the test with them. That would satisfy the original issue of testing
to many locales.
That idea has been discarded, so the next best thing to do is to have it
include a list of installed locales, and the language, territory and
codeset canonical names as well as the MB_CUR_LEN value for each. Those
are the only criteria that we currently use for selecting locales in the
tests.
I don't see anything else useful. If there is some detail that is useful,
most likely we could check it by loading the locale and getting the data
directly instead of caching that data ourselves.
> 2. one or more possible formats of the database
>
Because of all of the differences between similarly named locales on
different systems, I don't think it makes sense to keep the locale
data in revision control. It should probably be generated at runtime
and flushed to a file for reuse by later tests.
Given that, I don't feel that the format of the data is significant. It
might be nice for it to be human readable, but that is about it.
> 3. the kinds of queries done in our locale tests, and the
> ones we expect to do in future tests
This is the important question. As mentioned above, the only thing that
I see being used is selecting locales by name by MB_CUR_LEN.
> With that, we can create a prototype solution using an existing
> query language of our choice (such as grep). Once that works,
> the grammar should naturally fall out and we can reimplement
> the prototype in the test driver.
>
Isn't that what you did while I was in Boulder? That is how we arrived
at this system of brace expansion and name matching that we are talking
about now.
Your prototype boils down to something like this, where the fictional
'my_locale' utility lists the names of all installed locales followed
by a seperator and then the MB_CUR_LEN value.
for i in `echo $brace_expr`;
do
my_locale -a | grep -e $i
done
Honestly, I don't care what the grammar is. I don't care what the format
of the file is, and I don't care what shell utility we are trying to fake
today.
All I care about is finishing up this task. Two months is more than enough
time for something like this to be designed and implemented.
> Martin
>
>
