Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20465
The logging approach has been merged and I'm closing this one.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
@cloud-fan, will try it. Thank you sincerely.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20465
I've not worked in the logging stuff yet, feel free to take it, thanks!
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
> My proposal is, pandas and pyarrow should be a hard requirement for our
jenkins, to make sure the features are well tested.
If this is a goal, I think another simple way is just to
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
Thank you for bearing with me @cloud-fan. I agree with it.
BTW, are you working on the logging thing BTW? I was thinking the simplest
way to check is just print out once if PyArrow /
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20465
I agree that pandas and pyarrow should not be a hard requirement for users,
and this is what it is today: PySpark only throws exception when users try to
use pandas related functions without
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
Also, if we should go in this way, I think we should enable some tests with
PyPy too if I understood correctly and there isn't another problem I maybe
missed:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
Yup, explicitly logging sounds fine for now so that we can easily check.
> I do prefer to have these conditional skips removed because sometimes it
is hard to tell if everything passed
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20465
Yes, the tests are being run with python3. I do prefer to have these
conditional skips removed because sometimes it is hard to tell if everything
passed or was just skipped. But since pandas
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/20465
So, jenkins jobs run those tests with python3? If so, I feel better because
those tests are not completely skipped in Jenkins. If it is hard to make them
run with python 2. Letâs have a log to
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20465
Yup, there was a related discussion already. See this
https://github.com/apache/spark/pull/19884#issuecomment-351916074 and
https://github.com/apache/spark/pull/19884#issuecomment-353068446. We
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20465
Looking back at when pyarrow was last upgraded in #19884, pandas and
pyarrow were upgraded on all workers for python 3, but there were maybe some
concerns or difficulties with upgrading for
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20465
```
ImportError: Pandas >= 0.19.2 must be installed on calling Python process;
however, your version was 0.16.0.
```
I guess the RISELab boxes will need some updates...
---
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20465
sure, that's ok, I think we can revisit later (ie. next release) if we want
to add an env switch or something to make them optional
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20465
> I think there are some values in having a way to run python tests
without Arrow?
I agree, but the more important thing is to make sure jenkins runs
everything, so that we can be
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/20465
@felixcheung jenkins is actually skipping those tests (see the failure of
this pr). It makes sense to provide a way to allow developers to not run those
tests. But, I'd prefer that we run those tests
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20465
hmm, I think there are some values in having a way to run python tests
without Arrow? I mean the test.py is not just for Jenkins but for everyone
consuming the Spark release... unless we are
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20465
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20465
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86910/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20465
**[Test build #86910 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86910/testReport)**
for PR 20465 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20465
**[Test build #86910 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86910/testReport)**
for PR 20465 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20465
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/460/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20465
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20465
cc @yhuai @icexelloss @BryanCutler @ueshin @shaneknapp
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
24 matches
Mail list logo
