Re: Shuffle TTLs

Thanks for bringing this up. Wouldn't it be better for the notebooks to control when these DFs/RDDs expire so they can do fine granular control? On Wed, Oct 16, 2024 at 7:51 AM Holden Karau wrote: > Hi Spark Devs, > > So back in Spark 1.X we had shuffle TTLs, but they did not take into > account

Re: Dev list policy on posting genAI hallucinations

FWIW - Mich - I've often found your responses "gpt" like and can often be a distraction. Now I don't know if that's your actual writing style or you were indeed using genai tools to generate the responses on your behalf. I don't think we should sanction you if that's your writing style. But if you

Re: [VOTE] Single-pass Analyzer for Catalyst

> > Thanks, > Dongjoon. > > > On 2024/09/30 17:51:24 Herman van Hovell wrote: > > +1 > > > > On Mon, Sep 30, 2024 at 8:29 AM Reynold Xin > > > wrote: > > > > > +1 > > > > > > On Mon, Sep 30, 2024 at 6:47 AM Vladimir Go

Re: [VOTE] Single-pass Analyzer for Catalyst

+1 On Mon, Sep 30, 2024 at 6:47 AM Vladimir Golubev wrote: > Hi all, > > I’d like to start a vote for a single-pass Analyzer for the Catalyst > project. This project will introduce a new analysis framework to the > Catalyst, which will eventually replace the fixed-point one. > > Please refer to

Re: [DISCUSS] [Spark SQL] Single-pass Analyzer SPIP

Great document! Thanks for writing it up. On Tue, Sep 10, 2024 at 10:00 AM Vladimir Golubev wrote: > Hey folks, following up on the recent single-pass Analyzer discussion. I > made a high-level proposal document for this idea: > https://docs.google.com/document/d/1dWxvrJV-0joGdLtWbvJ0uNyTocDMJ90

Re: [VOTE] Move Variant to Parquet

+1 On Mon, Sep 2, 2024 at 9:30 AM Gene Pang wrote: > Hi all, > > I’d like to start a vote for moving the Variant specification and library > to the Parquet project. This allows the Variant binary format and shredding > format to be more widely used by other interested projects and systems. > > P

Re: [VOTE] Deprecate SparkR

+1 On Wed, Aug 21, 2024 at 6:42 PM Shivaram Venkataraman < shivaram.venkatara...@gmail.com> wrote: > Hi all > > Based on the previous discussion thread [1], I hereby call a vote to > deprecate the SparkR module in Apache Spark with the upcoming Spark 4 > release and remove it in the next major re

Re: [DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

+1 on this too When I implemented "group by all", I introduced at least two subtle bugs that many reviewers weren't able to catch and those two bugs would not have been possible to introduce if we had a single pass analyzer. Single pass can make the whole framework more robust. On Tue, Aug 2

Re: [DISCUSS] Move Variant to Parquet?

As I said on dev@iceberg, it'd be really unfortunate if we end up with two or even more diverging specs for storing variants. It just adds more work for everybody to interop. Parquet would be a great home for this spec as a neutral project that almost all the other important projects in this space

Re: Welcoming a new PMC member

Congratulations Kent! On Wed, Aug 14, 2024 at 10:10 AM Kent Yao wrote: > Thank you all very much! > > Kent > > On 2024/08/13 17:12:54 Matei Zaharia wrote: > > Congrats and welcome Kent! > > > > > On Aug 13, 2024, at 7:27 AM, Wenchen Fan wrote: > > > > > > Congratulations! > > > > > > On Tue, Au

Re: [DISCUSS] Deprecating SparkR

+1 It’s actually great that projects outside Spark’s repo can be more successful than the projects inside. A testament to both Spark itself and Spark Connect! On Tue, Aug 13, 2024 at 10:00 AM Martin Grund wrote: > +1 > > On Tue, Aug 13, 2024 at 7:26 AM Ruifeng Zheng wrote: > >> +1 >> >> On Tue

Re: [VOTE] Using Github Issues for Spark-Connect-Go _only_ issues.

+1 On Mon, Aug 12, 2024 at 10:28 AM Mich Talebzadeh wrote: > +1 for me > > Mich Talebzadeh, > > Architect | Data Engineer | Data Science | Financial Crime > PhD Imperial College > London

Re: [DISCUSS] Using Github Issues for Spark-Connect-Go _only_ issues.

I'd love that too. But maybe we can start small and try it out with one project ... On Thu, Aug 8, 2024 at 7:16 AM Sean Owen wrote: > Oh nice if that has changed. Id personally prefer switching all of Spark > to GitHub issues for simplicity but maybe that's a big lift. And a separate > question.

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 On Mon, Jul 8, 2024 at 7:44 PM haydn wrote: > +1 > > On Mon, Jul 8, 2024 at 7:41 PM haydn wrote: > >> +1 >> >> On Mon, Jul 8, 2024 at 19:41 Takuya UESHIN >> wrote: >> >>> +1 >>> >>> On Mon, Jul 8, 2024 at 6:05 PM Yuanjian Li >>> wrote: >>> +1 Hyukjin Kwon 于2024年7月4日周四 16:54

Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

+1 On Wed, Jul 3, 2024 at 4:45 PM L. C. Hsieh wrote: > +1 > > On Wed, Jul 3, 2024 at 3:54 PM Dongjoon Hyun > wrote: > > > > +1 > > > > Dongjoon > > > > On Wed, Jul 3, 2024 at 10:58 Xinrong Meng wrote: > >> > >> +1 > >> > >> Thank you @Hyukjin Kwon ! > >> > >> On Wed, Jul 3, 2024 at 8:55 AM bo

Re: [FYI] SPARK-47993: Drop Python 3.8

+1 On Thu, Apr 25, 2024 at 9:01 AM Santosh Pingale wrote: > +1 > > On Thu, Apr 25, 2024, 5:41 PM Dongjoon Hyun > wrote: > >> FYI, there is a proposal to drop Python 3.8 because its EOL is October >> 2024. >> >> https://github.com/apache/spark/pull/46228 >> [SPARK-47993][PYTHON] Drop Python 3.8

Re: A proposal for creating a Knowledge Sharing Hub for Apache Spark Community

One of the problem in the past when something like this was brought up was that the ASF couldn't have officially blessed venues beyond the already approved ones. So that's something to look into. Now of course you are welcome to run unofficial things unblessed as long as they follow trademark r

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 On Mon, Mar 11 2024 at 7:38 PM, Jungtaek Lim < kabhwan.opensou...@gmail.com > wrote: > > +1 (non-binding), thanks Gengliang! > > > On Mon, Mar 11, 2024 at 5:46 PM Gengliang Wang < ltn...@gmail.com > wrote: > > > >> Hi all, >> >> I'd like to start the vote for SPIP: Structured Logging F

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 On Fri, Nov 24, 2023 at 10:19 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1 > > > Thanks, > Dongjoon. > > On Fri, Nov 24, 2023 at 7:14 PM Ye Zhou < zhouyejoe@ gmail. com ( > zhouye...@gmail.com ) > wrote: > > >> +1(non-binding) >> >> On Fri, Nov 24, 2023 at 11:16 Mridul M

Re: [DISCUSS] SPIP: ShuffleManager short name registration via SparkPlugin

Why do we need this? The reason data source APIs need it is because it will be used by very unsophisticated end users and used all the time (for each connection / query). Shuffle is something you set up once, presumably by fairly sophisticated admins / engineers. On Sat, Nov 04, 2023 at 2:42 PM

Re: Are DataFrame rows ordered without an explicit ordering clause?

It should be the same as SQL. Otherwise it takes away a lot of potential future optimization opportunities. On Mon, Sep 18 2023 at 8:47 AM, Nicholas Chammas < nicholas.cham...@gmail.com > wrote: > > I’ve always considered DataFrames to be logically equivalent to SQL tables > or queries. > >

Re: [VOTE][SPIP] Python Data Source API

+1! On Fri, Jul 7 2023 at 11:58 AM, Holden Karau < hol...@pigscanfly.ca > wrote: > > +1 > > > On Fri, Jul 7, 2023 at 9:55 AM huaxin gao < huaxin.ga...@gmail.com > wrote: > > > >> +1 >> >> >> On Fri, Jul 7, 2023 at 8:59 AM Mich Talebzadeh < mich.talebza...@gmail.com >> > wrote: >> >> >>>

Re: [DISCUSS] SPIP: Python Data Source API

Personally I'd love this, but I agree with some of the earlier comments that this should not be Python specific (meaning I should be able to implement a data source in Python and then make it usable across all languages Spark  supports). I think we should find a way to make this reusable beyond P

Re: [VOTE][SPIP] PySpark Test Framework

+1 This is a great idea. On Wed, Jun 21, 2023 at 8:29 AM, Holden Karau < hol...@pigscanfly.ca > wrote: > > I’d like to start with a +1, better Python testing tools integrated into > the project make sense. > > On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu < amandastephanieliu@ gmail. com > ( aman

Re: [DISCUSS] Deprecate DStream in 3.4

+1 On Thu, Jan 12, 2023 at 9:46 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1 for the proposal (guiding only without any code change). > > > Thanks, > Dongjoon. > > On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu < zsxwing@ gmail. com ( > zsxw...@gmail.com ) > wrote: > > >> +1 >

Re: How can I get the same spark context in two different python processes

Spark Connect :) (It’s work in progress) On Mon, Dec 12 2022 at 2:29 PM, Kevin Su < pings...@gmail.com > wrote: > > Hey there, How can I get the same spark context in two different python > processes? > Let’s say I create a context in Process A, and then I want to use python > subprocess B to g

Re: Re: [VOTE][SPIP] Spark Connect

+1 super excited about this. I think it'd make Spark a lot more usable in application development and cloud setting: (1) Makes it easier to embed in applications with thinner client dependencies. (2) Easier to isolate user code vs system code in the driver. (3) Opens up the potential to upgrade

Re: Stickers and Swag

Nice! Going to order a few items myself ... On Tue, Jun 14, 2022 at 7:54 PM, Gengliang Wang < ltn...@gmail.com > wrote: > > FYI now you can find the shopping information on https:/ / spark. apache. org/ > community ( https://spark.apache.org/community ) as well :) > > > > Gengliang > > > >

Re: Data correctness issue with Repartition + FetchFailure

This is why RoundRobinPartitioning shouldn't be used ... On Sat, Mar 12, 2022 at 12:08 PM, Jason Xu < jasonxu.sp...@gmail.com > wrote: > > Hi Spark community, > > I reported a data correctness issue in https:/ / issues. apache. org/ jira/ > browse/ SPARK-38388 ( https://issues.apache.org/jira/b

Re: Creating a memory-efficient AggregateFunction to calculate Median

tl;dr: there's no easy way to implement aggregate expressions that'd require multiple pass over data. It is simply not something that's supported and doing so would be very high cost. Would you be OK using approximate percentile? That's relatively cheap. On Mon, Dec 13, 2021 at 6:43 PM, Nichola

Re: spark binary map

Read up on Unsafe here: https://mechanical-sympathy.blogspot.com/ On Sat, Oct 16, 2021 at 12:41 AM, Rohan Bajaj < rohanbaja...@gmail.com > wrote: > > In 2015 Reynold Xin made improvements to Spark and it was basically moving > some structures that were on the java heap and movin

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 On Thu, Oct 07, 2021 at 11:54 PM, Yuming Wang < wgy...@gmail.com > wrote: > > +1 (non-binding). > > > On Fri, Oct 8, 2021 at 1:02 PM Dongjoon Hyun < dongjoon. hyun@ gmail. com ( > dongjoon.h...@gmail.com ) > wrote: > > >> +1 for Apache Spark 3.2.0 RC7. >> >> >> It looks good to me. I te

Re: [VOTE] SPIP: Support pandas API layer on PySpark

+1. Would open up a huge persona for Spark. On Fri, Mar 26 2021 at 11:30 AM, Bryan Cutler < cutl...@gmail.com > wrote: > > +1 (non-binding) > > > On Fri, Mar 26, 2021 at 9:49 AM Maciej < mszymkiew...@gmail.com > wrote: > > >> +1 (nonbinding) >> >> >> >> On 3/26/21 3:52 PM, Hyukjin Kwon wr

Re: [DISCUSS] Support pandas API layer on PySpark

I don't think we should deprecate existing APIs. Spark's own Python API is relatively stable and not difficult to support. It has a pretty large number of users and existing code. Also pretty easy to learn by data engineers. pandas API is a great for data science, but isn't that great for some

Re: [VOTE] Release Spark 3.1.1 (RC3)

+1 Correctness issues are serious! On Wed, Feb 24, 2021 at 11:08 AM, Mridul Muralidharan < mri...@gmail.com > wrote: > > That is indeed cause for concern. > +1 on extending the voting deadline until we finish investigation of this. > > > > > Regards, > Mridul > > > > On Wed, Feb 24, 2021

Re: Auto-closing PRs or How to get reviewers' attention

Enrico - do feel free to reopen the PRs or email people directly, unless you are told otherwise. On Thu, Feb 18, 2021 at 9:09 AM, Nicholas Chammas < nicholas.cham...@gmail.com > wrote: > > On Thu, Feb 18, 2021 at 10:34 AM Sean Owen < srowen@ gmail. com ( > sro...@gmail.com ) > wrote: > > >>

Re: [DISCUSS] Add RocksDB StateStore

Late +1 On Sat, Feb 13 2021 at 2:49 PM, Liang-Chi Hsieh < vii...@gmail.com > wrote: > > > > Hi devs, > > > > Thanks for all the inputs. I think overall there are positive inputs in > Spark community about having RocksDB state store as external module. Then > let's go forward with this direc

Re: [Spark SQL]: SQL, Python, Scala and R API Consistency

There's another thing that's not mentioned … it's primarily a problem for Scala. Due to static typing, we need a very large number of function overloads for the Scala version of each function, whereas in SQL/Python they are just one. There's a limit on how many functions we can add, and it also

Re: [VOTE] Standardize Spark Exception Messages SPIP

Exciting & look forward to this! (And a late +1 vote that probably won't be counted) On Mon, Nov 09, 2020 at 2:37 PM, Allison Wang < allison.w...@databricks.com > wrote: > > > > Thanks everyone for voting! With 11 +1s and no -1s, this vote passes. > > > > +1s: > Mridul Muralidharan > Ange

Re: I'm going to be out starting Nov 5th

Take care Holden and best of luck with everything! On Sat, Oct 31 2020 at 10:21 AM, Holden Karau < hol...@pigscanfly.ca > wrote: > > Hi Folks, > > > Just a heads up so folks working on decommissioning or other areas I've > been active in don't block on me, I'm going to be out for at least a we

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

The issue is memory overhead. Writing files create a lot of buffer (especially in columnar formats like Parquet/ORC). Even a few file handlers and buffers per task can OOM the entire process easily. On Fri, Sep 04, 2020 at 5:51 AM, XIMO GUANTER GONZALBEZ < joaquin.guantergonzal...@telefonica.co

Re: Welcoming some new Apache Spark committers

Welcome all! On Tue, Jul 14, 2020 at 10:36 AM, Matei Zaharia < matei.zaha...@gmail.com > wrote: > > > > Hi all, > > > > The Spark PMC recently voted to add several new committers. Please join me > in welcoming them to their new roles! The new committers are: > > > > - Huaxin Gao > - Jun

Re: [DISCUSS] Apache Spark 3.0.1 Release

+1 on doing a new patch release soon. I saw some of these issues when preparing the 3.0 release, and some of them are very serious. On Tue, Jun 23, 2020 at 8:06 AM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu > wrote: > > > > +1 Thanks Yuanjian -- I think it'll be great to have a 3.0.

Re: Removing references to slave (and maybe in the future master)

Thanks for doing this. I think this is a great thing to do. But we gotta be careful with API compatibility. On Thu, Jun 18, 2020 at 11:32 AM, Holden Karau < hol...@pigscanfly.ca > wrote: > > Hi Folks, > > > I've started working on cleaning up the Spark code to remove references to > slave sin

[ANNOUNCE] Apache Spark 3.0.0

Hi all, Apache Spark 3.0.0 is the first release of the 3.x line. It builds on many of the innovations from Spark 2.x, bringing new ideas as well as continuing long-term projects that have been in development. This release resolves more than 3400 tickets. We'd like to thank our contributors and

Re: [vote] Apache Spark 3.0 RC3

...@yahoo.com > wrote: > > Reynold, > > > What's the plan on pushing the official release binaries and source tar? > It would be nice to have the release artifacts now that it's available on > maven. > > > thanks, > Tom > > > On Monday, J

Re: [vote] Apache Spark 3.0 RC3

> release. > > > Thanks, > Dongjoon. > > > > On Tue, Jun 9, 2020 at 9:41 PM Matei Zaharia < matei. zaharia@ gmail. com ( > matei.zaha...@gmail.com ) > wrote: > > >> Congrats! Excited to see the release posted soon. >> >> >>>

Re: Revisiting the idea of a Spark 2.5 transitional release

g into a release at the time we cut the > branch. > > On Fri, Jun 12, 2020 at 10:28 PM Reynold Xin < rxin@ databricks. com ( > r...@databricks.com ) > wrote: > > >> I understand the argument to add JDK 11 support just to extend the EOL, >> but the other things seem ki

Re: Revisiting the idea of a Spark 2.5 transitional release

I understand the argument to add JDK 11 support just to extend the EOL, but the other things seem kind of arbitrary and are not supported by your arguments, especially DSv2 which is a massive change. DSv2 IIUC is not api stable yet and will continue to evolve in the 3.x line. Spark is designed in

Re: [vote] Apache Spark 3.0 RC3

I waited another day to account for the weekend. This vote passes with the following +1 votes and no -1 votes! I'll start the release prep later this week. +1: Reynold Xin (binding) Prashant Sharma (binding) Gengliang Wang Sean Owen (binding) Mridul Muralidharan (binding) Takeshi Yam

Re: [vote] Apache Spark 3.0 RC3

Apologies for the mistake. The vote is open till 11:59pm Pacific time on Mon June 9th. On Sat, Jun 6, 2020 at 1:08 PM Reynold Xin wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.0.0. > > The vote is open until [DUE DAY] and passes if a majority

[vote] Apache Spark 3.0 RC3

Please vote on releasing the following candidate as Apache Spark version 3.0.0. The vote is open until [DUE DAY] and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0 [ ] -1 Do not release this package because ... To lea

[VOTE] Apache Spark 3.0 RC2

Please vote on releasing the following candidate as Apache Spark version 3.0.0. The vote is open until Thu May 21 11:59pm Pacific time and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0 [ ] -1 Do not release this packa

Re: [DISCUSS] Java specific APIs design concern and choice

The con is much more than just more effort to maintain a parallel API. It puts the burden for all libraries and library developers to maintain a parallel API as well. That’s one of the primary reasons we moved away from this RDD vs JavaRDD approach in the old RDD API. On Tue, Apr 28, 2020 at 12:3

Re: Spark DAG scheduler

bdi...@husky.neu.edu > wrote: > > Is it correct to say, the nodes in the DAG are RDDs and the edges are > computations? > > > On Thu, Apr 16, 2020 at 6:21 PM Reynold Xin < rxin@ databricks. com ( > r...@databricks.com ) > wrote: > > >> The RDD is the DAG. >>

Re: Spark DAG scheduler

The RDD is the DAG. On Thu, Apr 16, 2020 at 3:16 PM, Mania Abdi < abdi...@husky.neu.edu > wrote: > > Hello everyone, > > I am implementing a caching mechanism for analytic workloads running on > top of Spark and I need to retrieve the Spark DAG right after it is > generated and the DAG schedule

Re: [VOTE] Apache Spark 3.0.0 RC1

The Apache Software Foundation requires voting before any release can be published. On Tue, Mar 31, 2020 at 11:27 PM, Stephen Coy < s...@infomedia.com.au.invalid > wrote: > > >> On 1 Apr 2020, at 5:20 pm, Sean Owen < srowen@ gmail. com ( >> sro...@gmail.com ) > wrote: >> >> It can be publish

[VOTE] Apache Spark 3.0.0 RC1

Please vote on releasing the following candidate as Apache Spark version 3.0.0. The vote is open until 11:59pm Pacific time Fri Apr 3 , and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0 [ ] -1 Do not release this pack

Re: Release Manager's official `branch-3.0` Assessment?

ll have the blockers that will fail the >>> RCs.  >>> >>> >>> >>> Cheers, >>> >>> >>> Xiao >>> >>> >>> >>> On Tue, Mar 24, 2020 at 6:56 PM Dongjoon Hyun < dongjoon. hyun@ gmail. com >&

Re: results of taken(3) not appearing in console window

bcc dev, +user You need to print out the result. Take itself doesn't print. You only got the results printed to the console because the Scala REPL automatically prints the returned value from take. On Thu, Mar 26, 2020 at 12:15 PM, Zahid Rahman < zahidr1...@gmail.com > wrote: > > I am running

Re: Release Manager's official `branch-3.0` Assessment?

I actually think we should start cutting RCs. We can cut RCs even with blockers. On Tue, Mar 24, 2020 at 12:51 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > Hi, All. > > First of all, always "Community Over Code"! > I wish you the best health and happiness. > > As we know, we are s

Re: FYI: The evolution on `CHAR` type behavior

default datasource as provider for CREATE TABLE > syntax", 2019/12/06 > >    https:/ / lists. apache. org/ thread. html/ > > 493f88c10169680191791f9f6962fd16cd0ffa3b06726e92ed04cbe1%40%3Cdev. > spark. apache. org%3E ( > https://lists.apache.org/thread.html/493f88c10169680191791f9f6962fd16cd0ffa3b06726e92ed04cbe1%40%3Cdev.

Re: FYI: The evolution on `CHAR` type behavior

You are joking when you said " informed widely and discussed in many ways twice" right? This thread doesn't even talk about char/varchar:  https://lists.apache.org/thread.html/493f88c10169680191791f9f6962fd16cd0ffa3b06726e92ed04cbe1%40%3Cdev.spark.apache.org%3E (Yes it talked about changing the

Re: FYI: The evolution on `CHAR` type behavior

periences with the non-negligible cases in on-prem. > > > > Bests, > Dongjoon. > > > On Mon, Mar 16, 2020 at 5:42 PM Reynold Xin < rxin@ databricks. com ( > r...@databricks.com ) > wrote: > > >> −User >> >> >> >> char barely sh

Re: FYI: The evolution on `CHAR` type behavior

te away > from the standard on this specific behavior. > > > Bests, > Dongjoon. > > On Mon, Mar 16, 2020 at 5:35 PM Reynold Xin < rxin@ databricks. com ( > r...@databricks.com ) > wrote: > > >> BTW I'm not opposing us sticking to SQL standard (I

Re: FYI: The evolution on `CHAR` type behavior

systems also deviate away from the standard on this specific behavior. On Mon, Mar 16, 2020 at 5:29 PM, Reynold Xin < r...@databricks.com > wrote: > > I looked up our usage logs (sorry I can't share this publicly) and trim > has at least four orders of magnitude higher usage

Re: FYI: The evolution on `CHAR` type behavior

( >>> dongjoon.h...@gmail.com ) > wrote: >>> >>> Hi, Reynold. >>> (And +Michael Armbrust) >>> >>> >>> If you think so, do you think it's okay that we change the return value >>> silently? Then, I'm won

Re: FYI: The evolution on `CHAR` type behavior

, >> >> >> 100% agree with Reynold. >> >> >> >> >> Regards, >> Gourav Sengupta >> >> >> On Mon, Mar 16, 2020 at 3:31 AM Reynold Xin < rxin@ databricks. com ( >> r...@databricks.com ) > wrote: >> >&g

Re: FYI: The evolution on `CHAR` type behavior

the banning was > the proposed alternative to reduce the potential issue. > > > Please give us your opinion since it's still PR. > > > Bests, > Dongjoon. > > On Sat, Mar 14, 2020 at 17:54 Reynold Xin < rxin@ databricks. com ( > r...@databricks.com ) &g

Re: FYI: The evolution on `CHAR` type behavior

I don’t understand this change. Wouldn’t this “ban” confuse the hell out of both new and old users? For old users, their old code that was working for char(3) would now stop working. For new users, depending on whether the underlying metastore char(3) is either supported but different from ansi S

Re: [VOTE] Amend Spark's Semantic Versioning Policy

+1 On Mon, Mar 09, 2020 at 3:53 PM, John Zhuge < jzh...@apache.org > wrote: > > +1 (non-binding) > > > On Mon, Mar 9, 2020 at 1:32 PM Michael Heuer < heuermh@ gmail. com ( > heue...@gmail.com ) > wrote: > > >> +1 (non-binding) >> >> >> I am disappointed however that this only mentions API

Re: [DISCUSS] Shall we mark spark streaming component as deprecated.

It's a good discussion to have though: should we deprecate dstream, and what do we need to do to make that happen? My experience working with a lot of Spark users is that in general I recommend them staying away from dstream, due to a lot of design and architectural issues. On Mon, Mar 02, 2020

Re: [DISCUSS] naming policy of Spark configs

This is really cool. We should also be more opinionated about how we specify time and intervals. On Wed, Feb 12, 2020 at 3:15 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > Thank you, Wenchen. > > > The new policy looks clear to me. +1 for the explicit policy. > > > So, are we go

Re: Spark 3.0 branch cut and code freeze on Jan 31?

Note that branch-3.0 was cut. Please focus on testing, polish, and let's get the release out! On Wed, Jan 29, 2020 at 3:41 PM, Reynold Xin < r...@databricks.com > wrote: > > Just a reminder - code freeze is coming this Fri ! > > > > There can always be ex

Re: Spark 3.0 branch cut and code freeze on Jan 31?

;>>> named output from CleanupAliases >>>>> SPARK-25640 Clarify/Improve EvalType for grouped aggregate and window >>>>> aggregate >>>>> SPARK-25531 new write APIs for data source v2 >>>>> SPARK-25547 Pluggable jdbc connection factory >>>>> SPARK-2

Re: [SQL] Is it worth it (and advisable) to implement native UDFs?

If your UDF itself is very CPU intensive, it probably won't make that much of difference, because the UDF itself will dwarf the serialization/deserialization overhead. If your UDF is cheap, it will help tremendously. On Mon, Jan 20, 2020 at 6:33 PM, < em...@yeikel.com > wrote: > > > > Hi, >

Re: Enabling push-based shuffle in Spark

Thanks for writing this up.  Usually when people talk about push-based shuffle, they are motivating it primarily to reduce the latency of short queries, by pipelining the map phase, shuffle phase, and the reduce phase (which this design isn't going to address). It's interesting you are targetin

Re: Adding Maven Central mirror from Google to the build?

This seems reasonable! On Tue, Jan 21, 2020 at 3:23 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1, I'm supporting the following proposal. > > > > this mirror as the primary repo in the build, falling back to Central if > needed. > > > Thanks, > Dongjoon. > > > > On Tue, Jan

Re: [DISCUSS] Support year-month and day-time Intervals

Introducing a new data type has high overhead, both in terms of internal complexity and users' cognitive load. Introducing two data types would have even higher overhead. I looked quickly and looks like both Redshift and Snowflake, two of the most recent SQL analytics successes, have only one i

Re: [SPARK-30296][SQL] Add Dataset diffing feature

Can this perhaps exist as an utility function outside Spark? On Tue, Jan 07, 2020 at 12:18 AM, Enrico Minack < m...@enrico.minack.dev > wrote: > > > > Hi Devs, > > > > I'd like to get your thoughts on this Dataset feature proposal. Comparing > datasets is a central operation when regressio

Spark 3.0 branch cut and code freeze on Jan 31?

We've pushed out 3.0 multiple times. The latest release window documented on the website ( http://spark.apache.org/versioning-policy.html ) says we'd code freeze and cut branch-3.0 early Dec. It looks like we are suffering a bit from the tragedy of the commons, that nobody is pushing for getting

Re: Spark 3.0 preview release 2?

If the cost is low, why don't we just do monthly previews until we code freeze? If it is high, maybe we should discuss and do it when there are people that volunteer On Sun, Dec 08, 2019 at 10:32 PM, Xiao Li < gatorsm...@gmail.com > wrote: > > > > I got many great feedbacks from the com

Re: Why Spark generates Java code and not Scala?

It’s mainly due to compilation speed. Scala compiler is known to be slow. Even javac is quite slow. We use Janino which is a simpler compiler to get faster compilation speed at runtime. Also for low level code we can’t use (due to perf concerns) any of the edges scala has over java, eg we can’t us

Re: [VOTE] SPARK 3.0.0-preview (RC1)

Does the description make sense? This is a preview release so there is no need to retarget versions. On Tue, Oct 29, 2019 at 7:01 PM Xingbo Jiang wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.0.0-preview. > > The vote is open until November 2 PST and passe

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

Just curious - did we discuss why this shouldn't be another Apache sister project? On Wed, Oct 16, 2019 at 10:21 AM, Sean Owen < sro...@gmail.com > wrote: > > > > We don't all have to agree on whether to add this -- there are like 10 > people with an opinion -- and I certainly would not veto

Re: branch-3.0 vs branch-3.0-preview (?)

Can we just tag master? On Wed, Oct 16, 2019 at 12:34 AM, Wenchen Fan < cloud0...@gmail.com > wrote: > > Does anybody remember what we did for 2.0 preview? Personally I'd like to > avoid cutting branch-3.0 right now, otherwise we need to merge PRs into > two branches in the following several mon

Re: [DISCUSS] Out of order optimizer rules?

l write up, but I think we should at least give some > up-to-date description on that JIRA entry. > > On Wed, Oct 2, 2019 at 3:13 PM Reynold Xin < r...@databricks.com > wrote: > > >> No there is no separate write up internally. >> >> On Wed, Oct 2

Re: [DISCUSS] Out of order optimizer rules?

gt;> >> Regarding the place in the optimizer rules, it's preferred to happen late >> in the optimization, and definitely after join reorder. >> >> >> Thanks, >> Maryann >> >> On Wed, Oct 2, 2019 at 12:20 PM Reynold Xin wrote: >> >>

Re: [DISCUSS] Out of order optimizer rules?

Whoever created the JIRA years ago didn't describe dpp correctly, but the linked jira in Hive was correct (which unfortunately is much more terse than any of the patches we have in Spark  https://issues.apache.org/jira/browse/HIVE-9152 ). Henry R's description was also correct. On Wed, Oct 02,

Re: Collections passed from driver to executors

A while ago we changed it so the task gets broadcasted too, so I think the two are fairly similar. On Mon, Sep 23, 2019 at 8:17 PM, Dhrubajyoti Hati < dhruba.w...@gmail.com > wrote: > > I was wondering if anyone could help with this question. > > On Fri, 20 Sep, 2019, 11:52 AM Dhrubajyoti Hat

Re: [DISCUSS] Spark 2.5 release

lRow, > but I think we would want to carefully consider whether that is the right > decision. And in any case, we would be able to keep 2.5 and 3.0 > compatible, which is the main goal. > > On Sat, Sep 21, 2019 at 2:28 PM Reynold Xin < r...@databricks.com > wrote: > > &

Re: [DISCUSS] Spark 2.5 release

gt;> Sounds like we agree, then. We will use it for 3.0, but there are known >>>> problems with it. >>>> >>>> Thinking we’d have dsv2 working in both 3.x (which will change and >>>> progress towards more stable, but will have to break certain APIs)

Re: [DISCUSS] Spark 2.5 release

> UnsafeRow > was part of the original proposal. > > > > In any case, the goal for 3.0 was not to replace the use of InternalRow , > it was to get the majority of SQL working on top of the interface added > after 2.4. That’s done and stable, so I think a 2.5 release

Re: [DISCUSS] Spark 2.5 release

lity would be a problem in maintaining compatibility > between the 2.5 version and the 3.0 version. If we find that we need to > make API changes (other than additions) then we can make those in the 3.1 > release. Because the goals we set for the 3.0 release have been reached > with the

Re: [DISCUSS] Spark 2.5 release

DSv2 is far from stable right? All the actual data types are unstable and you guys have completely ignored that. We'd need to work on that and that will be a breaking change. If the goal is to make DSv2 work across 3.x and 2.x, that seems too invasive of a change to backport once you consider th

Re: Thoughts on Spark 3 release, or a preview release

+1! Long due for a preview release. On Thu, Sep 12, 2019 at 5:26 PM, Holden Karau < hol...@pigscanfly.ca > wrote: > > I like the idea from the PoV of giving folks something to start testing > against and exploring so they can raise issues with us earlier in the > process and we have more time to

Re: [VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

Having three modes is a lot. Why not just use ansi mode as default, and legacy for backward compatibility? Then over time there's only the ANSI mode, which is standard compliant and easy to understand. We also don't need to invent a standard just for Spark. On Thu, Sep 05, 2019 at 12:27 AM, Wen

Re: JDK11 Support in Apache Spark

rote: >>> >>> >>>> maybe in the future, but not right now as the hadoop 2.7 build is broken. >>>> >>>> >>>> also, i busted dev/ run-tests. py ( http://dev/run-tests.py ) in my changes >>>> to support java11 in PRBs: >>>> https:/ / github. com/ apache/ spark/ pull/ 25585 ( >>>> https://github.com/apache/spark/pull/25585 ) >>>> >>>> >>>> >>>> quick fix, testing now. >>>> >>>> On Mon, Aug 26, 2019 at 10:23 AM Reynold Xin < rxin@ databricks. com ( >>>> r...@databricks.com ) > wrote: >>>> >>>> >>>>> Would it be possible to have one build that works for both? >>>>> >>>> >>>> >>> >>> >> >> > >

Re: JDK11 Support in Apache Spark

Would it be possible to have one build that works for both? On Mon, Aug 26, 2019 at 10:22 AM Dongjoon Hyun wrote: > Thank you all! > > Let me add more explanation on the current status. > > - If you want to run on JDK8, you need to build on JDK8 > - If you want to run on JDK11, you need

Re: DataSourceV2 : Transactional Write support

We can also just write using one partition, which will be sufficient for most use cases. On Mon, Aug 5, 2019 at 7:48 PM Matt Cheah wrote: > There might be some help from the staging table catalog as well. > > > > -Matt Cheah > > > > *From: *Wenchen Fan > *Date: *Monday, August 5, 2019 at 7:40 P

  1   2   3   4   5   6   7   8   9   10   >