Karl Svensson created ARROW-6958: ------------------------------------ Summary: [python] tutorial script for arrow in spark throws error Key: ARROW-6958 URL: https://issues.apache.org/jira/browse/ARROW-6958 Project: Apache Arrow Issue Type: Bug Components: Java, Python Affects Versions: 0.15.0 Environment: Ubuntu v 18. Cluster spun up on google dataproc - see startup for specs of cluster Reporter: Karl Svensson Fix For: 0.8.0 Attachments: arrow_error.txt, start-cluster-nl.ps1.txt
Running the arrow example for pyspark ([found here |[https://github.com/apache/spark/blob/master/examples/src/main/python/sql/arrow.py]]) causes a java.lang.IllegalArgumentException error. Running the same script with pyarrow v 0.8.0 causes the script to run correctly. Attached are the startup settings in google dataproc I'm using to create the cluster, as well as the output (with error text). It isn't immediately obvious to me what is causing the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)