sunchao commented on pull request #29843:
URL: https://github.com/apache/spark/pull/29843#issuecomment-712586049
> @sunchao, yes I think we can do that but would you mind creating a
separate PR to fix the test first though? Using python3 with my workaround fix
should be good enough.
Currently the github action tests pass without the `python3` change, and the
jenkins tests fail either w/ or w/o the `python3` change: in the latter case it
fails with error such as:
```
20/10/16 19:20:36 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0)
(amp-jenkins-worker-03.amp executor 1): org.apache.spark.SparkException:
Error from python worker:
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 104, in _run_module_as_main
loader, code, fname = _get_module_details(mod_name)
File "/usr/lib64/python2.6/runpy.py", line 79, in _get_module_details
loader = get_loader(mod_name)
File "/usr/lib64/python2.6/pkgutil.py", line 456, in get_loader
return find_loader(fullname)
File "/usr/lib64/python2.6/pkgutil.py", line 466, in find_loader
for importer in iter_importers(fullname):
File "/usr/lib64/python2.6/pkgutil.py", line 422, in iter_importers
__import__(pkg)
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/__init__.py",
line 53, in <module>
from pyspark.rdd import RDD, RDDBarrier
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/rdd.py", line
34, in <module>
from pyspark.java_gateway import local_connect_and_auth
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/java_gateway.py",
line 29, in <module>
from py4j.java_gateway import java_import, JavaGateway, JavaObject,
GatewayParameters
File
"/home/jenkins/workspace/SparkPullRequestBuilder@2/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
line 60
PY4J_TRUE = {"yes", "y", "t", "true"}
^
SyntaxError: invalid syntax
```
> Also, seems like we're going to split PR (?). The first one (this) is for
preparation, and second one is actually bumping up to Hadoop version to 3.2.1
(?). Would you mind clarifying the plan and what this PR proposes in the
description/title?
Yes that's right. The plan is to have a separate PR bumping Hadoop version
to 3.2.2 when that comes out (probably will be soon). There is a
[bug](https://issues.apache.org/jira/browse/HDFS-15191) in 3.2.1 which affects
wire compatibility between 3.2 clients and 2.x server.
I'll update the PR description soon. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
