[jira] [Commented] (SPARK-22376) run-tests.py fails at exec-sbt if run with Python 3
[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240072#comment-16240072 ] Apache Spark commented on SPARK-22376: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/19665 > run-tests.py fails at exec-sbt if run with Python 3 > --- > > Key: SPARK-22376 > URL: https://issues.apache.org/jira/browse/SPARK-22376 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 2.3.0 > Environment: OSX 10.12.6 Python 3.6.0 Anaconda 4.4 >Reporter: Joel Croteau >Priority: Minor > > Running ./dev/run-tests with python3 as the default gives this error at the > Building Spark stage: > {noformat} > > Building Spark > > [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: > -Phadoop-2.6 -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pyarn -Pmesos > -Pkinesis-asl test:package streaming-kafka-0-8-assembly/assembly > streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly > Traceback (most recent call last): > File "./dev/run-tests.py", line 622, in > main() > File "./dev/run-tests.py", line 593, in main > build_apache_spark(build_tool, hadoop_version) > File "./dev/run-tests.py", line 391, in build_apache_spark > build_spark_sbt(hadoop_version) > File "./dev/run-tests.py", line 344, in build_spark_sbt > exec_sbt(profiles_and_goals) > File "./dev/run-tests.py", line 293, in exec_sbt > if not sbt_output_filter.match(line): > TypeError: cannot use a string pattern on a bytes-like object > {noformat} > This is because in Python 3, the stdout member of a POpen object defaults to > returning a byte stream, and exec_sbt tries to read it as a text screen. This > can be fixed by specifying universal_newlines=True when creating the POpen > object. I notice that the hashbang at the start of run-tests.py says to run > with python2, so I am not sure how much of the rest of it is Python 3 > compatible, but this is the first error I've run into. It ran with python 3 > because run-tests runs run-tests.py using the default Python, which on my > system is Python 3. Not sure whether the better solution here is to try and > fix Python 3 compatibility in run-tests.py or set run-tests to use Python 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22376) run-tests.py fails at exec-sbt if run with Python 3
[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223355#comment-16223355 ] Sean Owen commented on SPARK-22376: --- If the change is still compatible with Python 2, yes go ahead. > run-tests.py fails at exec-sbt if run with Python 3 > --- > > Key: SPARK-22376 > URL: https://issues.apache.org/jira/browse/SPARK-22376 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 2.3.0 > Environment: OSX 10.12.6 Python 3.6.0 Anaconda 4.4 >Reporter: Joel Croteau >Priority: Minor > > Running ./dev/run-tests with python3 as the default gives this error at the > Building Spark stage: > {noformat} > > Building Spark > > [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: > -Phadoop-2.6 -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pyarn -Pmesos > -Pkinesis-asl test:package streaming-kafka-0-8-assembly/assembly > streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly > Traceback (most recent call last): > File "./dev/run-tests.py", line 622, in > main() > File "./dev/run-tests.py", line 593, in main > build_apache_spark(build_tool, hadoop_version) > File "./dev/run-tests.py", line 391, in build_apache_spark > build_spark_sbt(hadoop_version) > File "./dev/run-tests.py", line 344, in build_spark_sbt > exec_sbt(profiles_and_goals) > File "./dev/run-tests.py", line 293, in exec_sbt > if not sbt_output_filter.match(line): > TypeError: cannot use a string pattern on a bytes-like object > {noformat} > This is because in Python 3, the stdout member of a POpen object defaults to > returning a byte stream, and exec_sbt tries to read it as a text screen. This > can be fixed by specifying universal_newlines=True when creating the POpen > object. I notice that the hashbang at the start of run-tests.py says to run > with python2, so I am not sure how much of the rest of it is Python 3 > compatible, but this is the first error I've run into. It ran with python 3 > because run-tests runs run-tests.py using the default Python, which on my > system is Python 3. Not sure whether the better solution here is to try and > fix Python 3 compatibility in run-tests.py or set run-tests to use Python 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org