[jira] [Resolved] (LIVY-547) Livy kills session after livy.server.session.timeout even if the session is active

2019-08-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-547. -- Resolution: Fixed Fix Version/s: 0.7.0 > Livy kills session after livy.server.session.time

[jira] [Commented] (LIVY-547) Livy kills session after livy.server.session.timeout even if the session is active

2019-08-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903616#comment-16903616 ] Saisai Shao commented on LIVY-547: -- Issue resolved by pull request 190 https://github.com/apache/incubator

Re: Any plan to support update, delete and others

2019-08-08 Thread Saisai Shao
the latest features available in DataSourceV2, > and will continue to. In fact, we're adding features to DSv2 based on what > we've built internally at Netflix to support Iceberg. > > On Wed, Aug 7, 2019 at 7:03 PM Saisai Shao wrote: > >> Thanks a lot Ryan, that would be very helpful!

Re: Two newbie question about Iceberg

2019-08-08 Thread Saisai Shao
issing automatically? > > On Wed, Aug 7, 2019 at 7:13 PM Saisai Shao wrote: > >> Thanks guys for your reply. >> >> I didn't do anything special, I don't even have a configured Hive. I just >> simply put the iceberg (assembly) jar into Spark and start a local Spark >&g

Re: Iceberg in Spark 3.0.0

2019-08-07 Thread Saisai Shao
IMHO I agree that we should have a branch to track the changes for Spark 3.0.0. Spark 3.0.0 has several changes regarding to DataSource V2, it would be better to evaluate the changes and do the design by also considering 3.0 changes. My two cents :) Best regards, Saisai Edgar Rodriguez

Re: Two newbie question about Iceberg

2019-08-07 Thread Saisai Shao
version of Hive would avoid this problem? What version > are you linking with? > > > > On Tue, Aug 6, 2019 at 8:59 PM Saisai Shao > wrote: > > Hi team, > > > > I just met some issues when trying Iceberg with quick start guide. Not > sure if it is proper to send this to

Re: Any plan to support update, delete and others

2019-08-07 Thread Saisai Shao
y to call this yet. Same for > MERGE > INTO, open source Spark doesn’t support the operation yet. We’re also > working on building support into Spark as we go. > > I hope that helps! > > On Wed, Aug 7, 2019 at 4:25 AM Saisai Shao wrote: > >> Hi team, >> >&

Any plan to support update, delete and others

2019-08-07 Thread Saisai Shao
Hi team, Delta lake project recently announced version 0.3.0, which added several new features in API level, like update, delete, merge, vacuum, etc. May I ask is there any plan to add such features in Iceberg? Thanks Saisai

Two newbie question about Iceberg

2019-08-06 Thread Saisai Shao
Hi team, I just met some issues when trying Iceberg with quick start guide. Not sure if it is proper to send this to @dev mail list (seems there's no user mail list). One issue is that seems current Iceberg cannot run with embedded metastore. It will throw an exception. Is this an on-purpose

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-06 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900749#comment-16900749 ] Saisai Shao commented on LIVY-616: -- I see your point. Let me check the design and code then. Thanks

[jira] [Commented] (LIVY-616) Livy Server discovery

2019-08-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900602#comment-16900602 ] Saisai Shao commented on LIVY-616: -- [~oshevchenko] you can either push an empty commit or reopen the PR

[jira] [Assigned] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28475: --- Assignee: Nick Karpov > Add regex MetricFilter to GraphiteS

[jira] [Updated] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28475: Priority: Minor (was: Major) > Add regex MetricFilter to GraphiteS

[jira] [Resolved] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28475. - Resolution: Resolved Fix Version/s: 3.0.0 > Add regex MetricFilter to GraphiteS

[jira] [Commented] (SPARK-28475) Add regex MetricFilter to GraphiteSink

2019-08-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898749#comment-16898749 ] Saisai Shao commented on SPARK-28475: - This is resolved via https://github.com/apache/spark/pull

[jira] [Closed] (LIVY-597) Upgrade Livy guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed LIVY-597. Resolution: Duplicate > Upgrade Livy guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: Saisai Shao (was: jiewang) > Remove Guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: jiewang (was: Saisai Shao) > Remove Guava depende

[jira] [Assigned] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-587: Assignee: jiewang > Remove Guava dependency > --- > >

[jira] [Resolved] (LIVY-587) Remove Guava dependency

2019-07-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-587. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 181 [https://github.com

[jira] [Assigned] (LIVY-613) Livy can't handle the java.sql.Date type correctly

2019-07-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-613: Assignee: wyp > Livy can't handle the java.sql.Date type correc

[jira] [Resolved] (LIVY-613) Livy can't handle the java.sql.Date type correctly

2019-07-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-613. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 186 [https://github.com

[jira] [Reopened] (LIVY-575) Implement missing metadata operations

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened LIVY-575: -- > Implement missing metadata operations > - > >

[jira] [Updated] (LIVY-609) LDAP auth for Livy thriftserver

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-609: - Priority: Major (was: Minor) > LDAP auth for Livy thriftser

[jira] [Updated] (LIVY-609) LDAP auth for Livy thriftserver

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-609: - Fix Version/s: (was: 0.7.0) > LDAP auth for Livy thriftser

[jira] [Resolved] (LIVY-610) optimization for windows environment build.

2019-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-610. -- Resolution: Fixed Issue resolved by pull request 184 [https://github.com/apache/incubator-livy/pull/184

[jira] [Commented] (LIVY-575) Implement missing metadata operations

2019-07-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887773#comment-16887773 ] Saisai Shao commented on LIVY-575: -- Hi [~mgaido], we have 3 colleagues planning to work on Livy project

[jira] [Assigned] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28106: --- Assignee: angerszhu > Spark SQL add jar with wrong hdfs path, SparkContext still

[jira] [Resolved] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28106. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24909 [https

[jira] [Updated] (SPARK-28106) Spark SQL add jar with wrong hdfs path, SparkContext still add it to jar path ,and cause Task Failed

2019-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28106: Component/s: Spark Core > Spark SQL add jar with wrong hdfs path, SparkContext still

[jira] [Resolved] (LIVY-582) python test_create_new_session_without_default_config test fails consistently

2019-07-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-582. -- Resolution: Fixed Fix Version/s: 0.7.0 Issue resolved by pull request 180 [https://github.com

[jira] [Assigned] (LIVY-582) python test_create_new_session_without_default_config test fails consistently

2019-07-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-582: Assignee: Yiheng Wang > python test_create_new_session_without_default_config test fa

[jira] [Assigned] (LIVY-603) upgrade build spark version to 2.4.3

2019-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-603: Assignee: Yiheng Wang > upgrade build spark version to 2.

[jira] [Resolved] (LIVY-603) upgrade build spark version to 2.4.3

2019-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-603. -- Resolution: Fixed Issue resolved by pull request 179 [https://github.com/apache/incubator-livy/pull/179

[jira] [Commented] (LIVY-587) Remove Guava dependency

2019-07-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882007#comment-16882007 ] Saisai Shao commented on LIVY-587: -- I would suggest to remove guava rather than upgrade guava version

[jira] [Comment Edited] (LIVY-587) Remove Guava dependency

2019-07-10 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882007#comment-16882007 ] Saisai Shao edited comment on LIVY-587 at 7/10/19 12:40 PM: I would suggest

[jira] [Assigned] (SPARK-28202) [Core] [Test] Avoid noises of system props in SparkConfSuite

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-28202: --- Assignee: ShuMing Li > [Core] [Test] Avoid noises of system props in SparkConfSu

[jira] [Resolved] (SPARK-28202) [Core] [Test] Avoid noises of system props in SparkConfSuite

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-28202. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24998 [https

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2019-07-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876070#comment-16876070 ] Saisai Shao commented on SPARK-25299: - Better to post a pdf version [~mcheah] :). > Use rem

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2019-06-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874703#comment-16874703 ] Saisai Shao commented on SPARK-25299: - Votes were passed, so what is our plan for code submission

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

2019-06-16 Thread Saisai Shao
+1 (binding) Thanks Saisai Imran Rashid 于2019年6月15日周六 上午3:46写道: > +1 (binding) > > I think this is a really important feature for spark. > > First, there is already a lot of interest in alternative shuffle storage > in the community. There is already a lot of interest in alternative > shuffle

[jira] [Updated] (SPARK-25299) Use remote storage for persisting shuffle data

2019-06-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-25299: Labels: SPIP (was: ) > Use remote storage for persisting shuffle d

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

2019-06-13 Thread Saisai Shao
I think maybe we could start a vote on this SPIP. This has been discussed for a while, and the current doc is pretty complete as for now. Also we saw lots of demands in the community about building their own shuffle storage. Thanks Saisai Imran Rashid 于2019年6月11日周二 上午3:27写道: > I would be

[jira] [Created] (SPARK-27996) Spark UI redirect will be failed behind the https reverse proxy

2019-06-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-27996: --- Summary: Spark UI redirect will be failed behind the https reverse proxy Key: SPARK-27996 URL: https://issues.apache.org/jira/browse/SPARK-27996 Project: Spark

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

2019-06-10 Thread Saisai Shao
I'm currently working with MemVerge on the Splash project (one implementation of remote shuffle storage) and followed this ticket for a while. I would like to be a shepherd if no one else volunteered to be. Best regards, Saisai Matt Cheah 于2019年6月6日周四 上午8:33写道: > Hi everyone, > > > > I wanted

Re: Support for Livy with Scala 2.12

2019-06-05 Thread Saisai Shao
tion that was done an year ago and it > looks like it is mostly Pom changes. If that’s not correct than I might > need help to perform the upgrade. > > Do you have bandwidth to make this change? > > > > > From: Saisai Shao mailto:sai.sai.s...@gmail.com>>

Re: Support for Livy with Scala 2.12

2019-06-04 Thread Saisai Shao
.12 module? Is that just a > change in POM files or code change is required? > > We have release planned for July to upgrade Jupyter and Livy to utilize > spark 2.4.2. This is blocking us from upgrade. > From: Saisai Shao mailto:sai.sai.s...@gmail.com>> > Date: Monday, Jun 0

Re: Support for Livy with Scala 2.12

2019-06-03 Thread Saisai Shao
Like what we did before to support both Scala 2.10 and 2.11 in Livy, I think we should also have a new module to support 2.12. 于2019年6月4日周二 上午7:40写道: > Yes, the spark binary we downloaded is built with default Scala 2.12. We > want to use databricks delta which I think only support Scala 2.12.

[jira] [Commented] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844750#comment-16844750 ] Saisai Shao commented on SPARK-15348: - No, it doesn't support hive ACID, it has its own mechanism

[jira] [Commented] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844597#comment-16844597 ] Saisai Shao commented on SPARK-15348: - I think delta lake project is exactly what you want. > H

[jira] [Comment Edited] (SPARK-15348) Hive ACID

2019-05-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844597#comment-16844597 ] Saisai Shao edited comment on SPARK-15348 at 5/21/19 7:38 AM: -- I think

Re: Dynamic metric names

2019-05-06 Thread Saisai Shao
I remembered there was a PR about doing similar thing ( https://github.com/apache/spark/pull/18406). From my understanding, this seems like a quite specific requirement, it may requires code change to support your needs. Thanks Saisai Sergey Zhemzhitsky 于2019年5月4日周六 下午4:44写道: > Hello Spark

Re: Livy-0.6 release?

2019-03-12 Thread Saisai Shao
I can also help to release a new version. My only concern is that how mature the thrift module is, shall we enable it by default or leave it disabled? Thanks Saisai Jeff Zhang 于2019年3月12日周二 上午10:54写道: > Thanks Marcelo, I can help to test it in zeppelin side which use livy as > one interpreter.

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-06 Thread Saisai Shao
Do we have other block/critical issues for Spark 2.4.1 or waiting something to be fixed? I roughly searched the JIRA, seems there's no block/critical issues marked for 2.4.1. Thanks Saisai shane knapp 于2019年3月7日周四 上午4:57写道: > i'll be popping in to the sig-big-data meeting on the 20th to talk

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-05 Thread Saisai Shao
Hi DB, I saw that we already have 6 RCs, but the vote I can search by now was RC2, were they all canceled? Thanks Saisai DB Tsai 于2019年2月22日周五 上午4:51写道: > I am cutting a new rc4 with fix from Felix. Thanks. > > Sincerely, > > DB Tsai >

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-01-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750872#comment-16750872 ] Saisai Shao commented on SPARK-24615: - I'm really sorry about the delay. Due to some changes on my

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10

2019-01-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734072#comment-16734072 ] Saisai Shao commented on SPARK-26512: - This seems like a Netty version problem, netty-3.9.9

[jira] [Commented] (SPARK-26512) Spark 2.4.0 is not working with Hadoop 2.8.3 in windows 10?

2019-01-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733740#comment-16733740 ] Saisai Shao commented on SPARK-26512: - Please list the problems you saw, any log or exception. We

[jira] [Updated] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-26457: Priority: Minor (was: Major) > Show hadoop configurations in HistoryServer environment

[jira] [Resolved] (SPARK-26516) zeppelin with spark on mesos: environment variable setting

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-26516. - Resolution: Invalid > zeppelin with spark on mesos: environment variable sett

[jira] [Updated] (SPARK-26513) Trigger GC on executor node idle

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-26513: Fix Version/s: (was: 3.0.0) > Trigger GC on executor node i

[jira] [Commented] (SPARK-26516) zeppelin with spark on mesos: environment variable setting

2019-01-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732587#comment-16732587 ] Saisai Shao commented on SPARK-26516: - Questions should go to user@spark mail list. Also

Re: Apache Spark 2.2.3 ?

2019-01-01 Thread Saisai Shao
Agreed to have a new branch-2.3 release, as we already accumulated several fixes. Thanks Saisai Xiao Li 于2019年1月2日周三 下午1:32写道: > Based on the commit history, > https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 > contains more critical fixes. Maybe the priority

[jira] [Commented] (SPARK-20415) SPARK job hangs while writing DataFrame to HDFS

2018-12-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16730084#comment-16730084 ] Saisai Shao commented on SPARK-20415: - Have you tried latest version of Spark, does this problem

[jira] [Commented] (SPARK-25299) Use remote storage for persisting shuffle data

2018-12-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729357#comment-16729357 ] Saisai Shao commented on SPARK-25299: - [~jealous] Can we have a doc about this proposed solution

Re: about LIVY-424

2018-11-11 Thread Saisai Shao
If you're hitting the problem of LIVY-424, this is exactly a Scala REPL issue, not a Livy issue, it is hard to fix in the Livy side. lk_hadoop 于2018年11月12日周一 上午9:37写道: > hi,all: > I meet this issue https://issues.apache.org/jira/browse/LIVY-424 > , anybody know how to resolve it? >

Re: What's a blocker?

2018-10-24 Thread Saisai Shao
Just my two cents of the past experience. As a release manager of Spark 2.3.2, I felt significantly delay during the release by block issues. Vote was failed several times by one or two "block issue". I think during the RC time, each "block issue" should be carefully evaluated by the related PMCs

[jira] [Assigned] (LIVY-526) Upgrade jetty version

2018-10-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-526: Assignee: Arun Mahadevan > Upgrade jetty version > - > >

[jira] [Resolved] (LIVY-526) Upgrade jetty version

2018-10-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-526. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 120 [https://github.com

Re: welcome a new batch of committers

2018-10-07 Thread Saisai Shao
Congratulations to all! Jacek Laskowski 于2018年10月7日周日 上午1:12写道: > Wow! That's a nice bunch of contributors. Congrats to all new committers. > I've had tough times to follow all the contributions, but with this crew > it's gonna be nearly impossible. > > Pozdrawiam, > Jacek Laskowski > >

[jira] [Updated] (SPARK-25501) Kafka delegation token support

2018-09-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-25501: Labels: SPIP (was: ) > Kafka delegation token supp

Re: SPIP: Support Kafka delegation token in Structured Streaming

2018-09-29 Thread Saisai Shao
I like this proposal. Since Kafka already provides delegation token mechanism, we can also leverage Spark's delegation token framework to add Kafka as a built-in support. BTW I think there's no much difference in support structured streaming and DStream, maybe we can set both as goal. Thanks

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-27 Thread Saisai Shao
Only "without-hadoop" profile has 2.12 binary, is it expected? Thanks Saisai Wenchen Fan 于2018年9月28日周五 上午11:08写道: > I'm adding my own +1, since all the problems mentioned in the RC1 voting > email are all resolved. And there is no blocker issue for 2.4.0 AFAIK. > > On Fri, Sep 28, 2018 at

[ANNOUNCE] Announcing Apache Spark 2.3.2

2018-09-26 Thread Saisai Shao
We are happy to announce the availability of Spark 2.3.2! Apache Spark 2.3.2 is a maintenance release, based on the branch-2.3 maintenance branch of Spark. We strongly recommend all 2.3.x users to upgrade to this stable release. To download Spark 2.3.2, head over to the download page:

[ANNOUNCE] Announcing Apache Spark 2.3.2

2018-09-26 Thread Saisai Shao
We are happy to announce the availability of Spark 2.3.2! Apache Spark 2.3.2 is a maintenance release, based on the branch-2.3 maintenance branch of Spark. We strongly recommend all 2.3.x users to upgrade to this stable release. To download Spark 2.3.2, head over to the download page:

[jira] [Updated] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-09-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-25206: Labels: Parquet correctness known_issue (was: Parquet correctness) > wrong records are retur

[VOTE][RESULT] Spark 2.3.2 (RC6)

2018-09-23 Thread Saisai Shao
The vote passes. Thanks to all who helped with the release! I'll follow up later with a release announcement once everything is published. +1 (* = binding): Sea Owen* Wenchen Fan* Saisai Shao Dongjoon Hyun Takeshi Yamamuro John Zhuge Xiao Li* Denny Lee Ryan Blue Michael Heuer +0: None -1

[jira] [Assigned] (LIVY-510) Remove support for jdk7

2018-09-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-510: Assignee: Marcelo Vanzin > Remove support for j

[jira] [Resolved] (LIVY-510) Remove support for jdk7

2018-09-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-510. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 111 [https://github.com

***UNCHECKED*** Re: [VOTE] SPARK 2.3.2 (RC6)

2018-09-19 Thread Saisai Shao
: >> false) *** FAILED *** >> >> Thank you, Saisai. >> >> Bests, >> Dongjoon. >> >> On Mon, Sep 17, 2018 at 6:48 PM Saisai Shao >> wrote: >> >>> +1 from my own side. >>> >>> Thanks >>> Saisai >>>

***UNCHECKED*** Re: Re: [VOTE] SPARK 2.3.2 (RC6)

2018-09-19 Thread Saisai Shao
** FAILED *** >> >> Thank you, Saisai. >> >> Bests, >> Dongjoon. >> >> On Mon, Sep 17, 2018 at 6:48 PM Saisai Shao >> wrote: >> >>> +1 from my own side. >>> >>> Thanks >>> Saisai >>> >>> W

Re: [VOTE] SPARK 2.3.2 (RC6)

2018-09-17 Thread Saisai Shao
ld from source with most profiles passed for me. >> On Mon, Sep 17, 2018 at 8:17 AM Saisai Shao >> wrote: >> > >> > Please vote on releasing the following candidate as Apache Spark >> version 2.3.2. >> > >> > The vote is open until September 21 PST a

Re: [VOTE] SPARK 2.4.0 (RC1)

2018-09-17 Thread Saisai Shao
Hi Wenchen, I think you need to set SPHINXPYTHON to python3 before building the docs, to workaround the doc issue ( https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-docs/_site/api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegression ). Here is the notes for release page:

[VOTE] SPARK 2.3.2 (RC6)

2018-09-17 Thread Saisai Shao
Please vote on releasing the following candidate as Apache Spark version 2.3.2. The vote is open until September 21 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.3.2 [ ] -1 Do not release this package because ...

Re: [DISCUSS] Getting rid of old stuff

2018-09-13 Thread Saisai Shao
+1, To support J8, Spark 2.2+ might be the only option. I'm not sure if those vendors will continue to support Spark 2.2-, but IMHO for the community release, I think moving forward would be a better choice. Thanks Saisai Jeff Zhang 于2018年9月14日周五 上午9:39写道: > +1 > > > Alex Bozarth

[jira] [Commented] (LIVY-507) Livy 0.5 incubating. Dumps output of command run in the browser.

2018-09-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611602#comment-16611602 ] Saisai Shao commented on LIVY-507: -- Frankly saying, some users they just want the opposite, so it is hard

[jira] [Updated] (LIVY-507) Livy 0.5 incubating. Dumps output of command run in the browser.

2018-09-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-507: - Fix Version/s: (was: 0.5.0) > Livy 0.5 incubating. Dumps output of command run in the brow

[jira] [Created] (LIVY-508) Support custom auth filter for LivyServer

2018-09-11 Thread Saisai Shao (JIRA)
Saisai Shao created LIVY-508: Summary: Support custom auth filter for LivyServer Key: LIVY-508 URL: https://issues.apache.org/jira/browse/LIVY-508 Project: Livy Issue Type: Improvement

[jira] [Updated] (LIVY-508) Support custom auth filter for LivyServer

2018-09-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated LIVY-508: - Description: Current Livy only supports Spnego auth support, we may have different requirements for auth

[jira] [Assigned] (LIVY-493) Add UTs to the thriftserver module

2018-09-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-493: Assignee: Marco Gaido > Add UTs to the thriftserver mod

[jira] [Assigned] (LIVY-491) Add thriftserver module

2018-09-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned LIVY-491: Assignee: Marco Gaido > Add thriftserver module > --- > >

[jira] [Resolved] (LIVY-493) Add UTs to the thriftserver module

2018-09-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-493. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 104 [https://github.com

[jira] [Resolved] (LIVY-492) Base implementation Livy thriftserver

2018-09-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-492. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 104 [https://github.com

[jira] [Resolved] (LIVY-491) Add thriftserver module

2018-09-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/LIVY-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-491. -- Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 104 [https://github.com

Re: [VOTE] SPARK 2.3.2 (RC5)

2018-09-06 Thread Saisai Shao
Hi, PMC members asked me to hold a bit while they're dealing with some other things. Please wait for a bit while. Thanks Saisai zzc <441586...@qq.com> 于2018年9月6日周四 下午4:27写道: > Hi Saisai: > Spark 2.4 was cut, and is there any new process on 2.3.2? > > > > -- > Sent from:

[jira] [Assigned] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24615: --- Assignee: (was: Saisai Shao) > Accelerator-aware task scheduling for Sp

[jira] [Assigned] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

2018-08-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-25183: --- Assignee: Steve Loughran > Spark HiveServer2 registers shutdown hook with

[jira] [Resolved] (SPARK-25183) Spark HiveServer2 registers shutdown hook with JVM, not ShutdownHookManager; race conditions can arise

2018-08-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-25183. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22186 [https

[jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598115#comment-16598115 ] Saisai Shao commented on SPARK-25206: - I see, if it is not going to be merged, let's close this JIRA

[jira] [Commented] (SPARK-25135) insert datasource table may all null when select from view on parquet

2018-08-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598111#comment-16598111 ] Saisai Shao commented on SPARK-25135: - What's the ETA of this issue [~yumwang]? > insert datasou

<    1   2   3   4   5   6   7   8   9   10   >