Eyal Allweil created DATAFU-167: ----------------------------------- Summary: Fix Scala Python Bridge support in Spark 2 minor version updates Key: DATAFU-167 URL: https://issues.apache.org/jira/browse/DATAFU-167 Project: DataFu Issue Type: Bug Affects Versions: 1.6.1 Environment: _emphasized text_ Reporter: Eyal Allweil Fix For: 1.7.0
The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up). This can be reproduced in testing by running the tests with the command {code:bash} ./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 -PscalaCompatVersion=2.11 {code} The error message is {noformat} AttributeError: 'GatewayParameters' object has no attribute 'auth_token' at org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137) {noformat} Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; it's possible/probable that using the auth_token parameter will both fix this problem and be better in general. -- This message was sent by Atlassian Jira (v8.20.10#820010)