Felix Cheung created SPARK-23314: ------------------------------------ Summary: Pandas grouped udf on dataset with timestamp column error Key: SPARK-23314 URL: https://issues.apache.org/jira/browse/SPARK-23314 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 2.3.0 Environment: data sample
adshex,flight_id,latitude,longitude,altitude,speed,track,squawk,type,timestamp,name,other_names1,other_names2,n_number,serial_number,mfr_mdl_code,mfr,model,year_mfr,type_aircraft,agency A72AA1,72791e8,33.2552,-117.91699,5499,111,137,4401,B350,2015-08-18T07:58:54Z,US DEPARTMENT OF HOMELAND SECURITY,US CUSTOMS & BORDER PROTECTION,OFFICE OF AIR & MARINE,561A,FM-36,4220012,HAWKER BEECHCRAFT CORP,B300C,2010,5,dhs A72AA1,72791e8,33.2659,-117.928,5500,109,138,4401,B350,2015-08-18T07:58:39Z,US DEPARTMENT OF HOMELAND SECURITY,US CUSTOMS & BORDER PROTECTION,OFFICE OF AIR & MARINE,561A,FM-36,4220012,HAWKER BEECHCRAFT CORP,B300C,2010,5,dhs A72AA1,72791e8,33.2741,-117.93599,5500,109,137,4401,B350,2015-08-18T07:58:28Z,US DEPARTMENT OF HOMELAND SECURITY,US CUSTOMS & BORDER PROTECTION,OFFICE OF AIR & MARINE,561A,FM-36,4220012,HAWKER BEECHCRAFT CORP,B300C,2010,5,dhs A72AA1,72791e8,33.28251,-117.945,5500,112,138,4401,B350,2015-08-18T07:58:13Z,US DEPARTMENT OF HOMELAND SECURITY,US CUSTOMS & BORDER PROTECTION,OFFICE OF AIR & MARINE,561A,FM-36,4220012,HAWKER BEECHCRAFT CORP,B300C,2010,5,dhs A72AA1,72791e8,33.29341,-117.95699,5500,102,134,4401,B350,2015-08-18T07:57:58Z,US DEPARTMENT OF HOMELAND SECURITY,US CUSTOMS & BORDER PROTECTION,OFFICE OF AIR & MARINE,561A,FM-36,4220012,HAWKER BEECHCRAFT CORP,B300C,2010,5,dhs >>> flights = spark.read.option("inferSchema", True).option("header", >>> True).option("dateFormat", "yyyy-MM-dd HH:mm:ss").csv("data*.csv") >>> from pyspark.sql.functions import pandas_udf, PandasUDFType >>> @pandas_udf(flights.schema, PandasUDFType.GROUPED_MAP) ... def subtract_mean_year_mfr(pdf): ... return pdf.assign(year_mfr=pdf.year_mfr - pdf.year_mfr.mean()) ... g = flights.groupby('mfr').apply(subtract_mean_year_mfr) >>> g = flights.groupby('mfr').apply(subtract_mean_year_mfr) >>> >>> g.count() [Stage 3:=====================================================> (195 + 5) / 200]18/02/01 19:17:26 ERROR Executor: Exception in task 7.0 in stage 3.0 (TID 205) org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 229, in main process() File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 224, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 257, in dump_stream batch = _create_batch(series, self._timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 235, in _create_batch arrs = [create_array(s, t) for s, t in series] File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 230, in create_array s = _check_series_convert_timestamps_internal(s.fillna(0), timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/types.py", line 1733, in _check_series_convert_timestamps_internal return s.dt.tz_localize(tz).dt.tz_convert('UTC') File "/usr/local/lib/python2.7/site-packages/pandas/core/accessor.py", line 115, in f return self._delegate_method(name, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/accessors.py", line 131, in _delegate_method result = method(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/util/_decorators.py", line 118, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 1858, in tz_localize errors=errors) File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:164) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:114) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithoutKey$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/02/01 19:17:26 WARN TaskSetManager: Lost task 7.0 in stage 3.0 (TID 205, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 229, in main process() File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 224, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 257, in dump_stream batch = _create_batch(series, self._timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 235, in _create_batch arrs = [create_array(s, t) for s, t in series] File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 230, in create_array s = _check_series_convert_timestamps_internal(s.fillna(0), timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/types.py", line 1733, in _check_series_convert_timestamps_internal return s.dt.tz_localize(tz).dt.tz_convert('UTC') File "/usr/local/lib/python2.7/site-packages/pandas/core/accessor.py", line 115, in f return self._delegate_method(name, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/accessors.py", line 131, in _delegate_method result = method(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/util/_decorators.py", line 118, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 1858, in tz_localize errors=errors) File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:164) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:114) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithoutKey$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/02/01 19:17:26 ERROR TaskSetManager: Task 7 in stage 3.0 failed 1 times; aborting job Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/dataframe.py", line 455, in count return int(self._jdf.count()) File "/Users/felixcheung/Uber/spark-chamber/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__ File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/utils.py", line 63, in deco 18/02/01 19:17:26 WARN TaskSetManager: Lost task 62.0 in stage 3.0 (TID 207, localhost, executor driver): TaskKilled (Stage cancelled) 18/02/01 19:17:26 WARN TaskSetManager: Lost task 143.0 in stage 3.0 (TID 215, localhost, executor driver): TaskKilled (Stage cancelled) 18/02/01 19:17:26 WARN TaskSetManager: Lost task 167.0 in stage 3.0 (TID 216, localhost, executor driver): TaskKilled (Stage cancelled) return f(*a, **kw) File "/Users/felixcheung/Uber/spark-chamber/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o65.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 3.0 failed 1 times, most recent failure: Lost task 7.0 in stage 3.0 (TID 205, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 229, in main process() File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 224, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 257, in dump_stream batch = _create_batch(series, self._timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 235, in _create_batch arrs = [create_array(s, t) for s, t in series] File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 230, in create_array s = _check_series_convert_timestamps_internal(s.fillna(0), timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/types.py", line 1733, in _check_series_convert_timestamps_internal return s.dt.tz_localize(tz).dt.tz_convert('UTC') File "/usr/local/lib/python2.7/site-packages/pandas/core/accessor.py", line 115, in f return self._delegate_method(name, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/accessors.py", line 131, in _delegate_method result = method(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/util/_decorators.py", line 118, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 1858, in tz_localize errors=errors) File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:164) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:114) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithoutKey$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1590) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1578) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1577) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1577) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1811) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1760) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1749) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2092) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) at org.apache.spark.rdd.RDD.collect(RDD.scala:938) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:297) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2770) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2769) at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3251) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3250) at org.apache.spark.sql.Dataset.count(Dataset.scala:2769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 229, in main process() File "/Users/felixcheung/Uber/spark-chamber/python/lib/pyspark.zip/pyspark/worker.py", line 224, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 257, in dump_stream batch = _create_batch(series, self._timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 235, in _create_batch arrs = [create_array(s, t) for s, t in series] File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/serializers.py", line 230, in create_array s = _check_series_convert_timestamps_internal(s.fillna(0), timezone) File "/Users/felixcheung/Uber/spark-chamber/python/pyspark/sql/types.py", line 1733, in _check_series_convert_timestamps_internal return s.dt.tz_localize(tz).dt.tz_convert('UTC') File "/usr/local/lib/python2.7/site-packages/pandas/core/accessor.py", line 115, in f return self._delegate_method(name, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/accessors.py", line 131, in _delegate_method result = method(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/util/_decorators.py", line 118, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/pandas/core/indexes/datetimes.py", line 1858, in tz_localize errors=errors) File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:164) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:114) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.agg_doAggregateWithoutKey$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Reporter: Felix Cheung Under SPARK-22216 When testing pandas_udf on group bys, I saw this error with the timestamp column. AmbiguousTimeError: Cannot infer dst time from Timestamp('2015-11-01 01:29:30'), try using the 'ambiguous' argument For detailed on repo, see Environment box -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org