Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
Yes. The codes have the bug. For example, when the locale is TR, users are
unable to create a table with a table name containing `I`. This does not make
sense to me. I believe we have more
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
Ah, sorry, it was only about fixing tests. I thought we have bugs in the
main codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
Yes you have a point. It is minor in that it is just a test that is now
locale sensitive and supporting the locale in tests is much less important.
However ideally whatever fails should be fixed as
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
Sorry, my previous comment is to @HyukjinKwon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user nihavend commented on the issue:
https://github.com/apache/spark/pull/17527
maybe
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
: ) It sounds like you did not get my points. Anyway, I will fix it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user nihavend commented on the issue:
https://github.com/apache/spark/pull/17527
I logged the comment for the case below. The column and/or table name(s)
are not user or locale dependent as they are internals.
e.g :
"SERDEINFO" lowers to "serdeınfo"
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
If you refer column names being `ı` from `I` in Turkish locale, it might
be correct per the discussion above as they are correct lower cases in Turkish
locale.
I would like to know
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
@HyukjinKwon You can set the locale to `tr`. You will see the test failure.
The test cases failed because the column names are incorrectly set.
---
If your project is set up for it, you can
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
Do you mind if I ask a example case? I just would like to look into this to
help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
Thank you for your explanation. Just did a few run in our DDL support. We
still have a few bugs in the locale support. If we use Turkish locale, a few
test cases failed.
Do you know
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
The general idea is to leave any lower-casing that affects strings in the
user program alone, to use the locale-sensitive `toLowerCase()`. This is more
conservative. All of the changes should only
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17527
I am wondering what is the reason some of `toLowerCase` is changed, but the
others remain unchanged?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user nihavend commented on the issue:
https://github.com/apache/spark/pull/17527
Thank you very much all of you for all your efforts. Sometimes, facing the
same issue different platforms and looking for a way to set jvm options for
locale explicitly. But many times there is no
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75657/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75657 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75657/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75657 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75657/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #3647 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3647/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #3647 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3647/testReport)**
for PR 17527 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75608/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75608 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75608/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75608 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75608/testReport)**
for PR 17527 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75603/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75603 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75603/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75603 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75603/testReport)**
for PR 17527 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
Thanks @HyukjinKwon I backed out the changes that seem to directly affect
column, table and database names.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75572/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75572 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75572/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75572 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75572/testReport)**
for PR 17527 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
I reverted the changes to UTF8String, StopWordsRemover, and Tokenizer. The
rest look like either purely internal string-insensitive comparison, excepting
operations on database table and column
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
Thank you for clarifying it, @srowen and @viirya . I am okay with it too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17527
@srowen That sounds good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
Yes that's certainly the question. Is it more surprising or less surprising
for Lower()'s behavior to vary by machine's Locale? Generally I'd say that the
varying behavior is bad. However in this
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
Yea, that's the concern. The downside is when these are exposed to users.
However, it might be an advantage as well. The behavior doesn't depend on
default JVM locale and is consistent. I think
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17527
@HyukjinKwon yap, so for such cases exposed to users, I think it is better
to leave it out for the default locale?
---
If your project is set up for it, you can reply to this email and have your
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
@viirya, I think it is possible. In case of `Lower`, `Upper` and `InitCap`
as an example maybe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17527
Out of curious, is there any situation we do really need the locale
setting, instead of `Locale.ROOT`?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17527
I support this idea in general. I can at least identify several references,
for example,,
https://hibernate.atlassian.net/plugins/servlet/mobile#issue/HHH-9722,
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17527
Maybe @HyukjinKwon can confirm whether this sounds like a good idea? simple
change, just touches so many instances.
---
If your project is set up for it, you can reply to this email and have your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75507/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75507 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75507/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75507 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75507/testReport)**
for PR 17527 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75506 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75506/testReport)**
for PR 17527 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17527
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75506/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17527
**[Test build #75506 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75506/testReport)**
for PR 17527 at commit
52 matches
Mail list logo
