[jira] [Resolved] (SPARK-26189) Fix the doc of unionAll in SparkR

2018-11-30 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-26189. -- Resolution: Fixed Assignee: Huaxin Gao Fix Version/s: 3.0.0

Re: [VOTE] Accept donation of Rust Parquet implementation

2018-11-30 Thread Felix Cheung
+1!! From: Andy Grove Sent: Friday, November 30, 2018 4:26:21 PM To: dev@arrow.apache.org Subject: Re: [VOTE] Accept donation of Rust Parquet implementation +1 and great to see this happening! On Fri, Nov 30, 2018 at 4:51 PM Wes McKinney wrote: > Dear all, >

Re: [VOTE] Apache Crail 1.1-incubating (rc8)

2018-11-30 Thread Felix Cheung
+1 (binding) a few comments below, checked: filename signature & hash DISCLAIMER, LICENSE, NOTICE build from src no binary src files have headers (see below) comments, not blocker for release IMO: 1. CREDITS file is a bit non-standard in an ASF release - this is generally not included as it is

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-11-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701505#comment-16701505 ] Felix Cheung commented on SPARK-21291: -- hmm, ok > R bucketBy partitionBy

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-11-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700743#comment-16700743 ] Felix Cheung commented on SPARK-21291: -- I think we need to reopen this Jira since bucketBy

Re: [VOTE] Accept the Iceberg project for incubation

2018-11-13 Thread Felix Cheung
+1 (non binding) awesome to see this is taken forward to the incubator and looking forward to collaborate with the community! On Tue, Nov 13, 2018 at 9:09 AM Ryan Blue wrote: > +1 (binding) > > On Tue, Nov 13, 2018 at 9:06 AM Ryan Blue wrote: > > > The discuss thread seems to have reached

[jira] [Commented] (SPARK-24255) Require Java 8 in SparkR description

2018-11-12 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684668#comment-16684668 ] Felix Cheung commented on SPARK-24255: -- [~shivaram] I'm thinking if this is handling all version

[jira] [Resolved] (SPARK-26010) SparkR vignette fails on CRAN on Java 11

2018-11-12 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-26010. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 3.0.0

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-11 Thread Felix Cheung
I opened a PR on the vignettes fix to skip eval. From: Shivaram Venkataraman Sent: Wednesday, November 7, 2018 7:26 AM To: Felix Cheung Cc: Sean Owen; Shivaram Venkataraman; Wenchen Fan; Matei Zaharia; dev Subject: Re: [CRAN-pretest-archived] CRAN submission

[jira] [Created] (SPARK-26010) SparkR vignette fails on Java 11

2018-11-11 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-26010: Summary: SparkR vignette fails on Java 11 Key: SPARK-26010 URL: https://issues.apache.org/jira/browse/SPARK-26010 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-26010) SparkR vignette fails on CRAN on Java 11

2018-11-11 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-26010: - Summary: SparkR vignette fails on CRAN on Java 11 (was: SparkR vignette fails on Java 11

[jira] [Commented] (SPARK-25995) sparkR should ensure user args are after the argument used for the port

2018-11-10 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682762#comment-16682762 ] Felix Cheung commented on SPARK-25995: -- sparkR is just taking the whole string as-is [https

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Felix Cheung
Considering the timing for Spark 3.0, > deprecating lower versions, bumping up R to 3.4 might be reasonable > option. > > Adding Shane as well. > > If we ended up with not upgrading it, I will forward this email to CRAN > sysadmin to discuss further anyway. > > > &

Re: DataSourceV2 capability API

2018-11-09 Thread Felix Cheung
One question is where will the list of capability strings be defined? From: Ryan Blue Sent: Thursday, November 8, 2018 2:09 PM To: Reynold Xin Cc: Spark Dev List Subject: Re: DataSourceV2 capability API Yes, we currently use traits that have methods. Something

Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame

2018-11-09 Thread Felix Cheung
Very cool! From: Hyukjin Kwon Sent: Thursday, November 8, 2018 10:29 AM To: dev Subject: Arrow optimization in conversion from R DataFrame to Spark DataFrame Hi all, I am trying to introduce R Arrow optimization by reusing PySpark Arrow optimization. It

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-08 Thread Felix Cheung
_20181105_165757/Windows/00check.log > and > https://win-builder.r-project.org/incoming_pretest/SparkR_2.4.0_20181105_165757/Debian/00check.log, > the tests run in 1s. > On Tue, Nov 6, 2018 at 1:29 PM Felix Cheung wrote: > > > > I’d rather not mess with 2.4.0 at this point. On CRA

Re: [Vote] call a vote for IoTDB incubation proposal

2018-11-07 Thread Felix Cheung
+1 cool project On Wed, Nov 7, 2018 at 1:02 AM Gosling Von wrote: > +1 > > Good luck ~ > > Von Gosling > > > > 在 2018年11月7日,下午3:46,hxd 写道: > > > > Hi, > > Sorry for the previous mail with bad format. > > I'd like to call a VOTE to accept IoTDB project, a database for managing > large amounts

Re: Test and support only LTS JDK release?

2018-11-06 Thread Felix Cheung
Is there a list of LTS release that I can reference? From: Ryan Blue Sent: Tuesday, November 6, 2018 1:28 PM To: sn...@snazy.de Cc: Spark Dev List; cdelg...@apple.com Subject: Re: Test and support only LTS JDK release? +1 for supporting LTS releases. On Tue,

Re: Make Scala 2.12 as default Scala version in Spark 3.0

2018-11-06 Thread Felix Cheung
So to clarify, only scala 2.12 is supported in Spark 3? From: Ryan Blue Sent: Tuesday, November 6, 2018 1:24 PM To: d_t...@apple.com Cc: Sean Owen; Spark Dev List; cdelg...@apple.com Subject: Re: Make Scala 2.12 as default Scala version in Spark 3.0 +1 to Scala

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-06 Thread Felix Cheung
. Need to investigate but worse case test_package can run with 0 test. From: Sean Owen Sent: Tuesday, November 6, 2018 10:51 AM To: Shivaram Venkataraman Cc: Felix Cheung; Wenchen Fan; Matei Zaharia; dev Subject: Re: [CRAN-pretest-archived] CRAN submission SparkR

Re: Java 11 support

2018-11-06 Thread Felix Cheung
+1 for Spark 3, definitely Thanks for the updates From: Sean Owen Sent: Tuesday, November 6, 2018 9:11 AM To: Felix Cheung Cc: dev Subject: Re: Java 11 support I think that Java 9 support basically gets Java 10, 11 support. But the jump from 8 to 9

Java 11 support

2018-11-06 Thread Felix Cheung
Speaking of, get we work to support Java 11? That will fix all the problems below. From: Felix Cheung Sent: Tuesday, November 6, 2018 8:57 AM To: Wenchen Fan Cc: Matei Zaharia; Sean Owen; Spark dev list; Shivaram Venkataraman Subject: Re: [CRAN-pretest-archived

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-06 Thread Felix Cheung
We have not been able to publish to CRAN for quite some time (since 2.3.0 was archived - the cause is Java 11) I think it’s ok to announce the release of 2.4.0 From: Wenchen Fan Sent: Tuesday, November 6, 2018 8:51 AM To: Felix Cheung Cc: Matei Zaharia; Sean

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-06 Thread Felix Cheung
some ideas. Matei > On Nov 5, 2018, at 9:09 PM, Felix Cheung wrote: > > I don¡Št know what the cause is yet. > > The test should be skipped because of this check > https://github.com/apache/spark/blob/branch-2.4/R/pkg/inst/tests/testthat/test_basic.R#L21 > > And this >

Re: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Felix Cheung
: callJStatic("org.apache.spark.ml.r.GeneralizedLinearRegressionWrapper", "fit", formula, The earlier release was achived because of Java 11+ too so this unfortunately isn’t new. From: Sean Owen Sent: Monday, November 5, 2018 7:22 PM To: Felix Cheung

Fwd: [CRAN-pretest-archived] CRAN submission SparkR 2.4.0

2018-11-05 Thread Felix Cheung
FYI. SparkR submission failed. It seems to detect Java 11 correctly with vignettes but not skipping tests as would be expected. Error: processing vignette ‘sparkr-vignettes.Rmd’ failed with diagnostics: Java version 8 is required for this package; found version: 11.0.1 Execution halted *

[jira] [Commented] (SPARK-25923) SparkR UT Failure (checking CRAN incoming feasibility)

2018-11-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674216#comment-16674216 ] Felix Cheung commented on SPARK-25923: -- thanks - what's the exchange required with CRAN admin

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-01 Thread Felix Cheung
Thanks for being this up and much appreciate with keeping on top of this at all times. Can upgrading R able to fix the issue. Is this perhaps not necessarily malform but some new format for new versions perhaps? Anyway we should consider upgrading R version if that fixes the problem. As an

Re: [VOTE] SPARK 2.4.0 (RC5)

2018-10-31 Thread Felix Cheung
+1 Checked R doc and all R API changes From: Denny Lee Sent: Wednesday, October 31, 2018 9:13 PM To: Chitral Verma Cc: Wenchen Fan; dev@spark.apache.org Subject: Re: [VOTE] SPARK 2.4.0 (RC5) +1 On Wed, Oct 31, 2018 at 12:54 PM Chitral Verma

[jira] [Resolved] (SPARK-25859) add scala/java/python example and doc for PrefixSpan

2018-10-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25859. -- Resolution: Fixed Assignee: Huaxin Gao Fix Version/s: 2.4.0

[jira] [Resolved] (SPARK-16693) Remove R deprecated methods

2018-10-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-16693. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 3.0.0 > Remov

[jira] [Commented] (SPARK-12172) Consider removing SparkR internal RDD APIs

2018-10-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665908#comment-16665908 ] Felix Cheung commented on SPARK-12172: -- sounds good > Consider removing SparkR internal RDD A

[jira] [Resolved] (SPARK-15545) R remove non-exported unused methods, like jsonRDD

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-15545. -- Resolution: Duplicate > R remove non-exported unused methods, like json

[jira] [Updated] (SPARK-15545) R remove non-exported unused methods, like jsonRDD

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-15545: - Affects Version/s: 2.3.2 External issue ID: SPARK-12172 > R remove non-exported unu

[jira] [Comment Edited] (SPARK-12172) Consider removing SparkR internal RDD APIs

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664631#comment-16664631 ] Felix Cheung edited comment on SPARK-12172 at 10/26/18 4:11 AM: ok

[jira] [Commented] (SPARK-12172) Consider removing SparkR internal RDD APIs

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664631#comment-16664631 ] Felix Cheung commented on SPARK-12172: -- ok, what's our option for spark.lapply? > Consi

[jira] [Commented] (SPARK-16611) Expose several hidden DataFrame/RDD functions

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664628#comment-16664628 ] Felix Cheung commented on SPARK-16611: -- ping - we are going to consider removing RDD methods

[jira] [Commented] (SPARK-16611) Expose several hidden DataFrame/RDD functions

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664630#comment-16664630 ] Felix Cheung commented on SPARK-16611: -- see SPARK-12172 > Expose several hidden DataFrame/

[jira] [Commented] (SPARK-16693) Remove R deprecated methods

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664626#comment-16664626 ] Felix Cheung commented on SPARK-16693: -- rebuilt this on spark 3.0.0 > Remove R deprecated meth

[jira] [Updated] (SPARK-16693) Remove R deprecated methods

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-16693: - Description: For methods deprecated in Spark 2.0.0, we should remove them in 2.1.0 -> 3.

Re: DataSourceV2 hangouts sync

2018-10-25 Thread Felix Cheung
Yes please! From: Ryan Blue Sent: Thursday, October 25, 2018 1:10 PM To: Spark Dev List Subject: DataSourceV2 hangouts sync Hi everyone, There's been some great discussion for DataSourceV2 in the last few months, but it has been difficult to resolve some of

[jira] [Resolved] (SPARK-24572) "eager execution" for R shell, IDE

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-24572. -- Resolution: Fixed Assignee: Weiqiang Zhuang Fix Version/s: 3.0.0

[jira] [Resolved] (SPARK-24516) PySpark Bindings for K8S - make Python 3 the default

2018-10-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-24516. -- Resolution: Fixed Assignee: Ilan Filonenko Fix Version/s: 3.0.0

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658377#comment-16658377 ] Felix Cheung edited comment on SPARK-22947 at 10/21/18 8:53 PM: so

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658377#comment-16658377 ] Felix Cheung commented on SPARK-22947: -- so what's our take on this? it seems quite useful for time

[jira] [Resolved] (SPARK-24207) PrefixSpan: R API

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-24207. -- Resolution: Fixed Fix Version/s: 3.0.0 > PrefixSpan: R

[jira] [Assigned] (SPARK-24207) PrefixSpan: R API

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-24207: Assignee: Huaxin Gao > PrefixSpan: R API > - > >

[jira] [Commented] (SPARK-25634) New Metrics in External Shuffle Service to help identify abusing application

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658343#comment-16658343 ] Felix Cheung commented on SPARK-25634: -- how about off-heap and netty buffer usage? > New Metr

[jira] [Resolved] (SPARK-25675) [Spark Job History] Job UI page does not show pagination with one page

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25675. -- Resolution: Fixed Fix Version/s: 3.0.0 > [Spark Job History] Job UI page does not s

[jira] [Assigned] (SPARK-25675) [Spark Job History] Job UI page does not show pagination with one page

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-25675: Assignee: Shivu Sondur > [Spark Job History] Job UI page does not show paginat

[jira] [Updated] (SPARK-25730) Kubernetes scheduler tries to read pod details that it just deleted

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-25730: - Affects Version/s: (was: 2.5.0) > Kubernetes scheduler tries to read pod deta

[jira] [Assigned] (SPARK-25730) Kubernetes scheduler tries to read pod details that it just deleted

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-25730: Assignee: Mike Kaplinskiy > Kubernetes scheduler tries to read pod details that it j

[jira] [Resolved] (SPARK-25730) Kubernetes scheduler tries to read pod details that it just deleted

2018-10-21 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25730. -- Resolution: Fixed Fix Version/s: 3.0.0 > Kubernetes scheduler tries to read

Re: Zeppelin add hadoop submarine(machine learning framework) interpreter

2018-10-20 Thread Felix Cheung
Very cool! From: Jeff Zhang Sent: Friday, October 19, 2018 7:14 AM To: dev@zeppelin.apache.org Subject: Re: Zeppelin add hadoop submarine(machine learning framework) interpreter Thanks xun. This would be a great addon for zeppelin to support deep learning. I

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

2018-10-16 Thread Felix Cheung
I’m in favor of it. If you check the PR it’s a few isolated script changes and all test-only changes. Should have low impact on release but much better integration test coverage. From: Erik Erlandson Sent: Tuesday, October 16, 2018 8:20 AM To: dev Subject:

Re: SparkR issue

2018-10-14 Thread Felix Cheung
1 seems like its spending a lot of time in R (slicing the data I guess?) and not with Spark 2 could you write it into a csv file locally and then read it from Spark? From: ayan guha Sent: Monday, October 8, 2018 11:21 PM To: user Subject: SparkR issue Hi We

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

2018-10-07 Thread Felix Cheung
Jars and libraries only accessible locally at the driver is fairly limited? Don’t you want the same on all executor? From: Yinan Li Sent: Friday, October 5, 2018 11:25 AM To: Stavros Kontopoulos Cc: rve...@dotnetrdf.org; dev Subject: Re: [DISCUSS][K8S] Local

Re: Spark SQL parser and DDL

2018-10-07 Thread Felix Cheung
Sounds like a good idea? Would this be a step in the direction of supporting variation of the SQL dialect, too? From: Ryan Blue Sent: Thursday, October 4, 2018 8:56 AM To: Spark Dev List Subject: Spark SQL parser and DDL Hi everyone, I’ve been working on

Re: [DISCUSS] Syntax for table DDL

2018-10-02 Thread Felix Cheung
I think it has been an important “selling point” that Spark is “mostly compatible“ with Hive DDL. I have see a lot of teams suffering from switching between Presto and Hive dialects. So one question I have is, we are at a point of switch from Hive compatible to ANSI SQL, say? Perhaps a more

Re: On Scala 2.12.7

2018-10-01 Thread Felix Cheung
Although like you said, spark support for scala 2.12 is beta anyway then shouldn’t we get it to a working state by basing on 2.12.7? There shouldn’t be a stability issue since it is not officially “supported” From: Wenchen Fan Sent: Monday, October 1, 2018

Re: can Spark 2.4 work on JDK 11?

2018-09-29 Thread Felix Cheung
Not officially. We have seen problem with JDK 10 as well. It will be great if you or someone would like to contribute to get it to work.. From: kant kodali Sent: Tuesday, September 25, 2018 2:31 PM To: user @spark Subject: can Spark 2.4 work on JDK 11? Hi All,

[jira] [Updated] (SPARK-25572) SparkR tests failed on CRAN on Java 10

2018-09-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-25572: - Description: follow up to SPARK-24255 from 2.3.2 release we can see that CRAN doesn't seem

[jira] [Commented] (SPARK-25572) SparkR tests failed on CRAN on Java 10

2018-09-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633129#comment-16633129 ] Felix Cheung commented on SPARK-25572: -- [~cloud_fan] while not a blocker, it would be great

[jira] [Commented] (SPARK-25572) SparkR tests failed on CRAN on Java 10

2018-09-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633130#comment-16633130 ] Felix Cheung commented on SPARK-25572: -- [~shivaram] > SparkR tests failed on CRAN on Java

[jira] [Resolved] (SPARK-25572) SparkR tests failed on CRAN on Java 10

2018-09-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25572. -- Resolution: Fixed Fix Version/s: 2.5.0 2.4.1 Target

[jira] [Updated] (SPARK-25572) SparkR tests failed on CRAN on Java 10

2018-09-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-25572: - Summary: SparkR tests failed on CRAN on Java 10 (was: SparkR to skip tests because Java 10

[jira] [Created] (SPARK-25572) SparkR to skip tests because Java 10

2018-09-28 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-25572: Summary: SparkR to skip tests because Java 10 Key: SPARK-25572 URL: https://issues.apache.org/jira/browse/SPARK-25572 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-09-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630410#comment-16630410 ] Felix Cheung commented on SPARK-21291: -- Wait. I don’t think saveAsTable is the same thing? >

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-09-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628721#comment-16628721 ] Felix Cheung commented on SPARK-21291: -- The PR did not have bucketBy? > R bucketBy partitio

Re: spark.lapply

2018-09-26 Thread Felix Cheung
It looks like the native R process is terminated from buffer overflow. Do you know how much data is involved? From: Junior Alvarez Sent: Wednesday, September 26, 2018 7:33 AM To: user@spark.apache.org Subject: spark.lapply Hi! I’m using spark.lapply() in

Re: SPIP: support decimals with negative scale in decimal operation

2018-09-23 Thread Felix Cheung
DISCUSS thread is good to have... From: Marco Gaido Sent: Friday, September 21, 2018 3:31 AM To: Wenchen Fan Cc: dev Subject: Re: SPIP: support decimals with negative scale in decimal operation Hi Wenchen, Thank you for the clarification. I agree that this is

Re: 2.4.0 Blockers, Critical, etc

2018-09-21 Thread Felix Cheung
I think the point is we actually need to do these validation before completing the release... From: Wenchen Fan Sent: Friday, September 21, 2018 12:02 AM To: Sean Owen Cc: Spark dev list Subject: Re: 2.4.0 Blockers, Critical, etc Sean thanks for checking them!

Re: Mentors wanted for Apache Dubbo (incubating)

2018-09-20 Thread Felix Cheung
Very cool project. Would love to act as mentor (went through incubator with a project myself) but I’m not on IPMC. Would try to help in other ways. Keep it up! On Thu, Sep 20, 2018 at 7:34 PM Huxing Zhang wrote: > Hi community, > > The Apache Dubbo project now has two active mentors, and is

Re: [Feedback Requested] SPARK-25299: Using Distributed Storage for Persisting Shuffle Data

2018-09-20 Thread Felix Cheung
Hi +baibing3 +huangtao6 Came across your presentation on Alluxio - including shuffling - would you be interested in this? From: Matt Cheah Sent: Tuesday, September 4, 2018 2:54 PM To: Yuanjian Li Cc: Spark dev list Subject: Re: [Feedback Requested]

Re: [DISCUSS] PySpark Window UDF

2018-09-20 Thread Felix Cheung
Definitely! numba numbers are amazing From: Wes McKinney Sent: Saturday, September 8, 2018 7:46 AM To: Li Jin Cc: dev@spark.apache.org Subject: Re: [DISCUSS] PySpark Window UDF hi Li, These results are very cool. I'm excited to see you continuing to push this

[jira] [Resolved] (SPARK-23648) extend hint syntax to support any expression for R

2018-09-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-23648. -- Resolution: Fixed Assignee: Huaxin Gao Fix Version/s: 2.5.0 > extend h

Re: [VOTE] SPARK 2.4.0 (RC1)

2018-09-18 Thread Felix Cheung
If we could work on this quickly - it might get on to future RCs. From: Stavros Kontopoulos Sent: Monday, September 17, 2018 2:35 PM To: Yinan Li Cc: Xiao Li; eerla...@redhat.com; van...@cloudera.com.invalid; Sean Owen; Wenchen Fan; dev Subject: Re: [VOTE]

[jira] [Commented] (SPARK-24572) "eager execution" for R shell, IDE

2018-09-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617120#comment-16617120 ] Felix Cheung commented on SPARK-24572: -- thanks! very close -  showDF doesn't return anything so we

[jira] [Comment Edited] (SPARK-21291) R bucketBy partitionBy API

2018-09-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617118#comment-16617118 ] Felix Cheung edited comment on SPARK-21291 at 9/17/18 6:07 AM: --- I think

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-09-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617118#comment-16617118 ] Felix Cheung commented on SPARK-21291: -- I think it should be like this one   [https

Re: Should python-2 be supported in Spark 3.0?

2018-09-16 Thread Felix Cheung
I don’t think we should remove any API even in a major release without deprecating it first... From: Mark Hamstra Sent: Sunday, September 16, 2018 12:26 PM To: Erik Erlandson Cc: u...@spark.apache.org; dev Subject: Re: Should python-2 be supported in Spark 3.0?

Re: Should python-2 be supported in Spark 3.0?

2018-09-16 Thread Felix Cheung
I don’t think we should remove any API even in a major release without deprecating it first... From: Mark Hamstra Sent: Sunday, September 16, 2018 12:26 PM To: Erik Erlandson Cc: user@spark.apache.org; dev Subject: Re: Should python-2 be supported in Spark 3.0?

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-09-13 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613994#comment-16613994 ] Felix Cheung commented on SPARK-21291: -- No, you wouldn’t return a writer in R. I will reply

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-09-10 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610089#comment-16610089 ] Felix Cheung commented on SPARK-23200: -- probably need someone to rebuild on the current config

[jira] [Commented] (SPARK-22632) Fix the behavior of timestamp values for R's DataFrame to respect session timezone

2018-09-10 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610083#comment-16610083 ] Felix Cheung commented on SPARK-22632: -- mismatch between R and JVM time zone could be an issue

Re: Branch 2.4 is cut

2018-09-10 Thread Felix Cheung
I’m a bit concern about what Arun is summarizing? We are building on DSv2 and already have to rewrite for bunch of changes in master/2.4, increasing in cost for dev work and release management. If we are saying more changes are coming in 3.0, do we have more info on what value the current

[jira] [Resolved] (SPARK-25117) Add EXEPT ALL and INTERSECT ALL support in R.

2018-09-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25117. -- Resolution: Fixed Assignee: Dilip Biswal > Add EXEPT ALL and INTERSECT ALL support i

[jira] [Updated] (SPARK-25117) Add EXEPT ALL and INTERSECT ALL support in R.

2018-09-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-25117: - Fix Version/s: 2.4.0 > Add EXEPT ALL and INTERSECT ALL support i

[jira] [Resolved] (SPARK-25007) Add array_intersect / array_except /array_union / array_shuffle to SparkR

2018-09-02 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25007. -- Resolution: Fixed Fix Version/s: 2.4.0 > Add array_intersect / array_exc

[jira] [Assigned] (SPARK-25007) Add array_intersect / array_except /array_union / array_shuffle to SparkR

2018-09-02 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-25007: Assignee: Huaxin Gao > Add array_intersect / array_except /array_union / array_shuf

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599380#comment-16599380 ] Felix Cheung commented on SPARK-24434: -- .. and let's make sure any discussion are summarized

[jira] [Assigned] (SPARK-25275) require memberhip in wheel to run 'su' (in dockerfiles)

2018-08-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-25275: Assignee: Erik Erlandson > require memberhip in wheel to run 'su' (in dockerfi

[jira] [Assigned] (SPARK-24433) Add Spark R support

2018-08-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-24433: Assignee: Ilan Filonenko > Add Spark R supp

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598300#comment-16598300 ] Felix Cheung commented on SPARK-24434: -- so [~onursatici] is there a reason you open a PR even

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598297#comment-16598297 ] Felix Cheung commented on SPARK-24434: -- [~skonto] - hi what's happening? Henry is right, with Spark

Re: [DISCUSS] move away from python doctests

2018-08-31 Thread Felix Cheung
+1 on what Li said. And +1 on getting more coverage in unit tests - however often times we omit python unit tests deliberately if the python “wrapper” is trivial. This is what I’ve learned over the years from the previous pyspark maintainers. Admittedly gaps are there.

Re: SPIP: Executor Plugin (SPARK-24918)

2018-08-31 Thread Felix Cheung
+1 From: Mridul Muralidharan Sent: Wednesday, August 29, 2018 1:27:27 PM To: dev@spark.apache.org Subject: Re: SPIP: Executor Plugin (SPARK-24918) +1 I left a couple of comments in NiharS's PR, but this is very useful to have in spark ! Regards, Mridul On Fri,

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

2018-08-16 Thread Felix Cheung
I fully support merging that SparkR support on k8s. If Ilan and other are willing to manually validate the RC I’m happy to voucher for it (I’m not 100% sure i have capacity to test it that way but certainly will try) Also +1 on revisiting Jenkins builds. tbh I’m not sure we depend on them too

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-16 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582009#comment-16582009 ] Felix Cheung commented on SPARK-24918: -- I'd tend to agree with opt in - too many mistakes with copy

Re: [DISCUSS] ZEPPELIN-2619. Save note in [Title].zpln instead of [NOTEID]/note.json

2018-08-13 Thread Felix Cheung
Perhaps one concern is users having characters in note name that are invalid for file name/file path? From: Mohit Jaggi Sent: Sunday, August 12, 2018 6:02 PM To: users@zeppelin.apache.org Cc: dev Subject: Re: [DISCUSS] ZEPPELIN-2619. Save note in [Title].zpln

<    1   2   3   4   5   6   7   8   9   10   >