[ https://issues.apache.org/jira/browse/SPARK-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davies Liu resolved SPARK-9643. ------------------------------- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 7950 [https://github.com/apache/spark/pull/7950] > Error serializing datetimes with timezones using Dataframes and Parquet > ----------------------------------------------------------------------- > > Key: SPARK-9643 > URL: https://issues.apache.org/jira/browse/SPARK-9643 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.4.1 > Reporter: Alex Angelini > Labels: upgrade > Fix For: 1.6.0 > > > Trying to serialize a DataFrame with a datetime column that includes a > timezone fails with the following error. > {code} > net.razorvine.pickle.PickleException: invalid pickle data for datetime; > expected 1 or 7 args, got 2 > at > net.razorvine.pickle.objects.DateTimeConstructor.createDateTime(DateTimeConstructor.java:69) > at > net.razorvine.pickle.objects.DateTimeConstructor.construct(DateTimeConstructor.java:32) > at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:701) > at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:171) > at net.razorvine.pickle.Unpickler.load(Unpickler.java:85) > at net.razorvine.pickle.Unpickler.loads(Unpickler.java:98) > at > org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:151) > at > org.apache.spark.api.python.SerDeUtil$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(SerDeUtil.scala:150) > at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.org$apache$spark$sql$execution$datasources$InsertIntoHadoopFsRelation$$writeRows$1(commands.scala:185) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:163) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:163) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:64) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > According to [~davies] timezone serialization is done directly in Spark and > not dependent on Pyrolite, but I was not able to prove that. > Upgrading to Pyrolite 4.9 fixed this issue > https://github.com/apache/spark/pull/7950 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org