Re: Build error: python/lib/pyspark.zip is not a ZIP archive

2020-01-10 Thread Jeff Evans
Actually, there is a really trivial fix for that (an existing file not being deleted when packaging). Opened SPARK-30489 for it. On Fri, Jan 10, 2020 at 3:52 PM Jeff Evans wrote: > Thanks for the tip. Fixed by simply removing python/lib/pyspark.zip > (since it's apparently generated), and

Re: Build error: python/lib/pyspark.zip is not a ZIP archive

2020-01-10 Thread Jeff Evans
Thanks for the tip. Fixed by simply removing python/lib/pyspark.zip (since it's apparently generated), and rebuilding. I guess clean does not remove it. On Fri, Jan 10, 2020 at 3:50 PM Sean Owen wrote: > Sounds like you might have some corrupted file locally. I don't see > any of the

Re: Build error: python/lib/pyspark.zip is not a ZIP archive

2020-01-10 Thread Sean Owen
Sounds like you might have some corrupted file locally. I don't see any of the automated test builders failing. Nuke your local assembly build and try again? On Fri, Jan 10, 2020 at 3:49 PM Jeff Evans wrote: > > Greetings, > > I'm getting an error when building, on latest master (2bd873181 as of

Build error: python/lib/pyspark.zip is not a ZIP archive

2020-01-10 Thread Jeff Evans
Greetings, I'm getting an error when building, on latest master (2bd873181 as of this writing). Full build command I'm running is: ./build/mvn -DskipTests clean package [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (create-tmp-dir) on project

Re: [DISCUSS] Support year-month and day-time Intervals

2020-01-10 Thread Reynold Xin
Introducing a new data type has high overhead, both in terms of internal complexity and users' cognitive load. Introducing two data types would have even higher overhead. I looked quickly and looks like both Redshift and Snowflake, two of the most recent SQL analytics successes, have only one

Re: [DISCUSS] Support year-month and day-time Intervals

2020-01-10 Thread Dongjoon Hyun
Thank you for clarification. Bests, Dongjoon. On Fri, Jan 10, 2020 at 10:07 AM Kent Yao wrote: > Hi Dongjoon, > > Yes, As we want make CalenderIntervalType deprecated and so far, we just > find > 1. The make_interval function that produces legacy CalenderIntervalType > values, > 2. `interval`

Re: [DISCUSS] Support year-month and day-time Intervals

2020-01-10 Thread Kent Yao
Hi Dongjoon,Yes, As we want make CalenderIntervalType deprecated and so far, we just find1. The make_interval function that produces legacy CalenderIntervalType values, 2. `interval` -> CalenderIntervalType support in the parserThanks

Re: [DISCUSS] Support year-month and day-time Intervals

2020-01-10 Thread Dongjoon Hyun
Hi, Kent. Thank you for the proposal. Does your proposal need to revert something from the master branch? I'm just asking because it's not clear in the proposal document. Bests, Dongjoon. On Fri, Jan 10, 2020 at 5:31 AM Dr. Kent Yao wrote: > Hi, Devs > > I’d like to propose to add two new

unsubscribe

2020-01-10 Thread vijendra rana
unsubscribe

Re: unsubscribe

2020-01-10 Thread steve goodwin
unsubscribe

[DISCUSS] Support year-month and day-time Intervals

2020-01-10 Thread Dr. Kent Yao
Hi, Devs I’d like to propose to add two new interval types which are year-month and day-time intervals for better ANSI support and future improvements. We will keep the current CalenderIntervalType but mark it as deprecated until we find the right time to remove it completely. The backward

Re: Revisiting Python / pandas UDF (new proposal)

2020-01-10 Thread Hyukjin Kwon
Hi all, I made a PR - https://github.com/apache/spark/pull/27165 Please have a look when you guys fine some times. I addressed another point (by Maciej), "A couple of less-intuitive pandas UDF types" together because the more I look, the more I felt I should deal with it together with the