Eyal Allweil created DATAFU-167:
-----------------------------------
Summary: Fix Scala Python Bridge support in Spark 2 minor version
updates
Key: DATAFU-167
URL: https://issues.apache.org/jira/browse/DATAFU-167
Project: DataFu
Issue Type: Bug
Affects Versions: 1.6.1
Environment: _emphasized text_
Reporter: Eyal Allweil
Fix For: 1.7.0
The Scala Python Bridge, which works for versions 2.2.2, 2.3.2 and 2.4.3,
doesn't work for versions 2.2.3, 2.3.3, and 2.4.4 (and up).
This can be reproduced in testing by running the tests with the command
{code:bash}
./gradlew :datafu-spark:test -PscalaVersion=2.11 -PsparkVersion=2.2.3
-PscalaCompatVersion=2.11
{code}
The error message is
{noformat}
AttributeError: 'GatewayParameters' object has no attribute 'auth_token'
at
org.apache.spark.datafu.deploy.SparkPythonRunner.execFile(SparkPythonRunner.scala:137)
{noformat}
Currently our code runs using the PYSPARK_ALLOW_INSECURE_GATEWAY parameter;
it's possible/probable that using the auth_token parameter will both fix this
problem and be better in general.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
