Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96935 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96935/testReport)**
for PR 22610 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96935/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96935 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96935/testReport)**
for PR 22610 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
So I've added a bit document for this. @HyukjinKwon @BryanCutler please
check it when you have time.
---
-
To unsubscribe,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
One clear thing looks adding some documentation ..
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
Btw, I checked our `_minimum_pyarrow_version` is 0.8.0, so seems like even
there is next upgrade available, for users with pyarrow versions before 0.11.0,
this is still an potential issue. Isn't?
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
I think it is more reasonable to use the option when converting from Pandas
to raise an error for unsafe casts. It should be better than to display warning
message.
Not sure how long before
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
> Yeah, it's part of pyarrow.Array now, but will only be in the 0.11.0
release so we would have to do it after the next upgrade.
Then I think we can wait for next upgrade to use this feature
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/22610
> It is pretty new one, is it said we need to upgrade to latest PyArrow in
order to use it? Since it is an option at Table.from_pandas, is it possible to
extend it to pyarrow.Array?
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/22610
> Thanks, @BryanCutler. WDYT about documenting the type map thing?
I think that would help in the cases of dates/times because those can get a
little confusing. For primitives, I think
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
Thanks @BryanCutler! Looks like an useful option. It is pretty new one, is
it said we need to upgrade to latest PyArrow in order to use it? Since it is an
option at `Table.from_pandas`, is it
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
Thanks, @BryanCutler. WDYT about documenting the type map thing?
---
-
To unsubscribe, e-mail:
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/22610
So pyarrow just added an option when converting from Pandas to raise an
error for unsafe casts. I'd have to try it out to see if it would prevent this
case though. It's a common option when
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
@HyukjinKwon Thanks!
I agree that having document of this is definitely useful. I will try to
add it and let's see if it is ok for you. I think it is good to mention that
users are
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
> he only other idea I have is to provide an option to raise an error if
the type needs to be cast.
Actually sounds good to me.
I think the problem is we are not quite clear
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
Thanks @BryanCutler! Yes, this should not be a bug but is used as a warning
to users that there might be some type conversion they are not noticed at first
glance on the Pandas UDFs. For now the
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/22610
Thanks for looking into this @viirya ! I agree that there are lots of
cases where casting to another type is intentional and works fine, so this
isn't a bug. The only other idea I have is to
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
The idea sounds good to me from a cursory look for now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22610
Yea, I will do this week. Sorry I missed the cc in the JIRA.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96853/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96853 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96853/testReport)**
for PR 22610 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22610
cc @HyukjinKwon Can you take a look at this when you have time? Thanks.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96853 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96853/testReport)**
for PR 22610 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96849 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96849/testReport)**
for PR 22610 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96849/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22610
**[Test build #96849 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96849/testReport)**
for PR 22610 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22610
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
35 matches
Mail list logo
