[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-521464400 Yup. BTW `IntegratedUDFTestUtils` doesn't exist in master branch. I

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-521426162 Am fine with backporting but @zsxwing WDYT? ---

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-12 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-520685272 This fix will be included in Apache Spark 3.0. I think you should up

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-07-23 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-514437195 Merged to master. Thanks all. --

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-07-22 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-514013132 Thanks for review and approval, @zsxwing. Just FYI, there's s

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-07-22 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-514012160 Thanks for review and approval, @zsxwing. BTW, I understand t

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-07-22 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-513676624 retest this please

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-06-30 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-507101455 gentle ping .. :-) .. -

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-06-25 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-505662431 In case of `org.apache.spark.TaskContext`, looks it's manually set f

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-06-24 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-505260782 #24945 is merged. Let me rebase. -

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-06-24 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-505227175 It works after this PR as below: ```python from pyspark.sq

[GitHub] [spark] HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-06-23 Thread GitBox
HyukjinKwon commented on issue #24946: [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) URL: https://github.com/apache/spark/pull/24946#issuecomment-504873202 Ah .. `ContinuousCoalesceRDD` is dependent on the thread local not b