[ https://issues.apache.org/jira/browse/DATAFU-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eyal Allweil updated DATAFU-167: -------------------------------- Language: scala python Labels: up-for-grabs (was: ) > Fix Scala Python Bridge support in Spark 2 minor version updates > ----------------------------------------------------------------- > > Key: DATAFU-167 > URL: https://issues.apache.org/jira/browse/DATAFU-167 > Project: DataFu > Issue Type: Bug > Affects Versions: 1.6.1 > Environment: _emphasized text_ > Reporter: Eyal Allweil > Priority: Major > Labels: up-for-grabs > Fix For: 1.7.0 > > > The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3, > doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up). > This can be reproduced in testing by running the tests with the command > {code:bash} > ./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3 > -PscalaCompatVersion=2.11 > {code} > The error message is > {noformat} > AttributeError: 'GatewayParameters' object has no attribute 'auth_token' > at > org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137) > {noformat} > > Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter; > it's possible/probable that using the auth_token parameter will both fix this > problem and be better in general. > > Please note that in order to test Spark 2.4.4, you need to upgrade the > scalatest version used. -- This message was sent by Atlassian Jira (v8.20.10#820010)