Re: Release Apache Spark 2.4.4

Can the following be included? [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) Thanks, Terry On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan wrote: > +1 > > On Wed, Aug 14, 2019 at 12:52

Re: Release Apache Spark 2.4.4

+1 On Wed, Aug 14, 2019 at 12:52 PM Holden Karau wrote: > +1 > Does anyone have any critical fixes they’d like to see in 2.4.4? > > On Tue, Aug 13, 2019 at 5:22 PM Sean Owen wrote: > >> Seems fine to me if there are enough valuable fixes to justify another >> release. If there are any other

Re: Release Apache Spark 2.4.4

+1 Does anyone have any critical fixes they’d like to see in 2.4.4? On Tue, Aug 13, 2019 at 5:22 PM Sean Owen wrote: > Seems fine to me if there are enough valuable fixes to justify another > release. If there are any other important fixes imminent, it's fine to > wait for those. > > > On Tue,

RE: Release Apache Spark 2.4.4

Thanks, Dongjoon! +1 Kazuaki Ishizaki, From: Hyukjin Kwon To: Takeshi Yamamuro Cc: Dongjoon Hyun , dev , User Date: 2019/08/14 09:21 Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 +1 2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님 이 작성: Hi, Thanks for your

Re: Release Apache Spark 2.4.4

Seems fine to me if there are enough valuable fixes to justify another release. If there are any other important fixes imminent, it's fine to wait for those. On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun wrote: > > Hi, All. > > Spark 2.4.3 was released three months ago (8th May). > As of today

Re: Release Apache Spark 2.4.4

+1 2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님이 작성: > Hi, > > Thanks for your notification, Dongjoon! > I put some links for the other committers/PMCs to access the info easily: > > A commit list in github from the last release: >

Re: Release Apache Spark 2.4.4

Hi, Thanks for your notification, Dongjoon! I put some links for the other committers/PMCs to access the info easily: A commit list in github from the last release: https://github.com/apache/spark/compare/5ac2014e6c118fbeb1fe8e5c8064c4a8ee9d182a...branch-2.4 A issue list in jira:

Re: Release Apache Spark 2.4.4

+1 On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun wrote: > > Hi, All. > > Spark 2.4.3 was released three months ago (8th May). > As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` > since 2.4.3. > > It would be great if we can have Spark 2.4.4. > Shall we start `2.4.4

Release Apache Spark 2.4.4

Hi, All. Spark 2.4.3 was released three months ago (8th May). As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` since 2.4.3. It would be great if we can have Spark 2.4.4. Shall we start `2.4.4 RC1` next Monday (19th August)? Last time, there was a request for K8s issue

[K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

Hi all, K8S resource manager dumps the config map to /opt/spark/conf/spark-defaults.conf and passes it to spark submit twice: • via env var SPARK_CONF_DIR=/opt/spark/conf/ • argument --properties-file /opt/spark/conf/spark-defaults.conf This prevents definition of user defined

Re: [DISCUSS] ViewCatalog interface for DSv2

Thanks for the feedback, Ryan! I can share the WIP copy of the SPIP if that makes sense. I can't find out a lot about view resolution and validation in SQL Spec Part1. Anybody with full SQL knowledge may chime in. Here are my understanding based on online manuals, docs, and other resources:

Re: [DISCUSS] ViewCatalog interface for DSv2

Thanks for working on this, John! I'd like to see a more complete write-up of what you're proposing. Without that, I don't think we can have a productive discussion about this. For example, I think you're proposing to keep the view columns to ensure that the same columns are produced by the view

[DISCUSS] ViewCatalog interface for DSv2

Catalog support has been added to DSv2 along with a table catalog interface. Here I'd like to propose a view catalog interface, for the following benefit: - Abstraction for view management thus allowing different view backends - Disassociation of view definition storage from Hive Metastore

Re: displaying "Test build" in PR

"Can one of the admins verify this patch?" is a corrected message, as Jenkins won't test your PR until an admin approves it. BTW I think "5 minutes" is a reasonable delay for PR testing. It usually takes days to review and merge a PR, so I don't think seeing test progress right after PR creation

Re: My curation of pending structured streaming PRs to review

General tips: - dev@ is not usually the right place to discuss _specific_ changes except once in a while to call attention - Ping the authors of the code being changed directly - Tighten the change if possible - Tests, reproductions, docs, etc help prove the change - Bugs are more important than

Re: displaying "Test build" in PR

Thank you for your email. I think a newb like me might want to see what's going on PR and see something useful. For example, "Request builder polls every 5 minutes and you will see the progress here in a few minutes". I guess we can add a more useful message on AmplabJenkins

Re: My curation of pending structured streaming PRs to review

Thanks, Jungtaek for curating this list. It covers a lot of important fixes and performance improvements in structured streaming. Hi Devs What is missing from process perspective from getting these PRs merged? Apart from this list, is there any other forum where we can request attention to such

Re: Ask for ARM CI for spark

Hi all, About the arm test of spark, recently we found two tests failed after the commit https://github.com/apache/spark/pull/23767: ReplayListenerSuite: - ... - End-to-end replay *** FAILED *** "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) -