[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470777#comment-16470777 ] Felix Cheung commented on SPARK-23206: -- yes, for use network and disk IO stats. We have been

Re: Revisiting Online serving of Spark models?

Huge +1 on this! From: holden.ka...@gmail.com on behalf of Holden Karau Sent: Thursday, May 10, 2018 9:39:26 AM To: Joseph Bradley Cc: dev Subject: Re: Revisiting Online serving of Spark models? On Thu, May 10,

[jira] [Created] (SPARK-24207) PrefixSpan: R API

Felix Cheung created SPARK-24207: Summary: PrefixSpan: R API Key: SPARK-24207 URL: https://issues.apache.org/jira/browse/SPARK-24207 Project: Spark Issue Type: Sub-task Components

[jira] [Commented] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466920#comment-16466920 ] Felix Cheung commented on SPARK-23780: -- I suppose if you load googleVis first and then SparkR

[jira] [Updated] (SPARK-24195) sc.addFile for local:/ path is broken

[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24195: - Description: In changing SPARK-6300 https://github.com/apache/spark/commit

[jira] [Updated] (SPARK-24195) sc.addFile for local:/ path is broken

[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24195: - Affects Version/s: 1.3.1 1.4.1 1.5.2

[jira] [Created] (SPARK-24195) sc.addFile for local:/ path is broken

Felix Cheung created SPARK-24195: Summary: sc.addFile for local:/ path is broken Key: SPARK-24195 URL: https://issues.apache.org/jira/browse/SPARK-24195 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465433#comment-16465433 ] Felix Cheung commented on SPARK-23291: -- I don't disagree the behavior issue. (ah, so someone did run

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:38 PM: --- actually

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:35 PM: --- actually

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung commented on SPARK-23291: -- actually, I'm not sure we should backport this to a x.x.1

[jira] [Comment Edited] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465307#comment-16465307 ] Felix Cheung edited comment on SPARK-23291 at 5/6/18 10:34 PM: --- actually

Re: SparkR test failures in PR builder

This is resolved. Please see https://issues.apache.org/jira/browse/SPARK-24152 From: Kazuaki Ishizaki Sent: Wednesday, May 2, 2018 4:51:11 PM To: dev Cc: Joseph Bradley; Hossein Falaki Subject: Re: SparkR test failures in PR builder I am

[jira] [Comment Edited] (SPARK-24152) SparkR CRAN feasibility check server problem

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461983#comment-16461983 ] Felix Cheung edited comment on SPARK-24152 at 5/3/18 6:29 AM: -- ok good

[jira] [Updated] (SPARK-24152) SparkR CRAN feasibility check server problem

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24152: - Summary: SparkR CRAN feasibility check server problem (was: Flaky Test: SparkR) > SparkR C

[jira] [Commented] (SPARK-24152) SparkR CRAN feasibility check server problem

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461993#comment-16461993 ] Felix Cheung commented on SPARK-24152: -- (I updated the bug title - it's not really flaky

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461983#comment-16461983 ] Felix Cheung edited comment on SPARK-24152 at 5/3/18 6:26 AM: -- ok good

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461983#comment-16461983 ] Felix Cheung edited comment on SPARK-24152 at 5/3/18 6:26 AM: -- ok good

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461983#comment-16461983 ] Felix Cheung commented on SPARK-24152: -- ok good. in the event this reoccurs persistently, option 1

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461974#comment-16461974 ] Felix Cheung commented on SPARK-24152: -- Is this still a problem? > Flaky Test: Spa

Re: all calculations finished, but "VCores Used" value remains at its max

Zeppelin keeps the Spark job alive. This is likely a better question for the Zeppelin project. From: Valery Khamenya Sent: Tuesday, May 1, 2018 4:30:24 AM To: user@spark.apache.org Subject: all calculations finished, but "VCores Used" value

Re: zeppelin 0.8 tar file

0.8 is not released yet. From: Soheil Pourbafrani Sent: Sunday, April 29, 2018 9:18:10 AM To: users@zeppelin.apache.org Subject: zeppelin 0.8 tar file Is there any pre-compiled tar file of Zeppelin 0.8 to download?

[jira] [Commented] (SPARK-23954) Converting spark dataframe containing int64 fields to R dataframes leads to impredictable errors.

[ https://issues.apache.org/jira/browse/SPARK-23954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457390#comment-16457390 ] Felix Cheung commented on SPARK-23954: -- yap, please see discussion in SPARK-12360 in particular

Re: [DISCUSS] Adjust test logs for CI

bothersome for another developer. > >> > >> So it would be better to have a consensus to do our best to reduce my > >> logs, > >> especially made by test cases. > >> > >> How do you think of it? > >> > >> JL > >

Re: Problem running Kubernetes example v2.2.0-kubernetes-0.5.0

You might want to check with the spark-on-k8s Or try using kubernetes from the official spark 2.3.0 release. (Yes we don't have an official docker image though but you can build with the script) From: Rico Bergmann Sent: Wednesday, April

Re: [Julia] Does Spark.jl work in Zeppelin's existing Spark/livy.spark interpreters?

Actually, I’m not sure we support Julia as a language in the Spark interpreter. As far as I understand this, this is Julia -> Spark so we would need support for this added to enable Java (Zeppelin) -> Julia -> Spark From: Jongyoul Lee

Re: [DISCUSS] Adjust test logs for CI

Is there a way to do this via enable/disable component for logging in log4j? From: Jongyoul Lee Sent: Sunday, April 22, 2018 7:01:54 AM To: dev Subject: [DISCUSS] Adjust test logs for CI Hello contributors, I wonder how you guys think of

Re: [discuss][data source v2] remove type parameter in DataReader/WriterFactory

Is it required for DataReader to support all known DataFormat? Hopefully, not, as assumed by the 'throw' in the interface. Then specifically how are we going to express capability of the given reader of its supported format(s), or specific support for each of "real-time data in row format, and

Re: [Structured Streaming Query] Calculate Running Avg from Kafka feed using SQL query

Instead of write to console you need to write to memory for it to be queryable .format("memory") .queryName("tableName") https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks From: Aakash Basu

Re: Hadoop 3 support

What would be the strategy with hive? Cherry pick patches? Update to more “modern” versions (like 2.3?) I know of a few critical schema evolution fixes that we could port to hive 1.2.1-spark _ From: Steve Loughran Sent: Tuesday, April 3,

[jira] [Created] (ZEPPELIN-3385) PySpark interpreter should handle .. for autocomplete

Felix Cheung created ZEPPELIN-3385: -- Summary: PySpark interpreter should handle .. for autocomplete Key: ZEPPELIN-3385 URL: https://issues.apache.org/jira/browse/ZEPPELIN-3385 Project: Zeppelin

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425206#comment-16425206 ] Felix Cheung commented on SPARK-23285: -- fixed in 2.3, updating. > Allow spark.executor.co

[jira] [Updated] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23285: - Fix Version/s: 2.4.0 > Allow spark.executor.cores to be fractio

[jira] [Comment Edited] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425206#comment-16425206 ] Felix Cheung edited comment on SPARK-23285 at 4/4/18 8:53 AM: -- fixed in 2.4

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425202#comment-16425202 ] Felix Cheung commented on SPARK-23680: -- also, please check fixed version and target version when

[jira] [Updated] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23680: - Target Version/s: 2.4.0 (was: 2.3.1, 2.4.0) > entrypoint.sh does not accept arbitrary U

[jira] [Updated] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23680: - Fix Version/s: 2.4.0 > entrypoint.sh does not accept arbitrary UIDs, returning as an er

[jira] [Assigned] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-23680: Assignee: Ricardo Martinelli de Oliveira > entrypoint.sh does not accept arbitrary U

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425200#comment-16425200 ] Felix Cheung commented on SPARK-23680: -- try now. i also had to add rmartine - seemed like

Re: [Spark R] Proposal: Exposing RBackend in RRunner

Auto reference counting should already be handled by SparkR already. Can you elaborate on which object and how that would be used? From: Jeremy Liu <jeremy.jl@gmail.com> Sent: Thursday, March 29, 2018 8:23:58 AM To: Reynold Xin Cc: Felix Cheun

Re: [Spark R] Proposal: Exposing RBackend in RRunner

I think the difference is py4j is a public library whereas the R backend is specific to SparkR. Can you elaborate what you need JVMObjectTracker for? We have provided R convenient APIs to call into JVM: sparkR.callJMethod for example _ From: Jeremy Liu

Re: [Spark R]: Linear Mixed-Effects Models in Spark R

If your data can be split into groups and you can call into your favorite R package on each group of data (in parallel): https://spark.apache.org/docs/latest/sparkr.html#run-a-given-function-on-a-large-dataset-grouping-by-input-columns-and-using-gapply-or-gapplycollect

[jira] [Commented] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412918#comment-16412918 ] Felix Cheung commented on SPARK-23780: -- though there are other methods   [https://www.rforge.net

[jira] [Comment Edited] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412458#comment-16412458 ] Felix Cheung edited comment on SPARK-23780 at 3/24/18 6:53 AM: --- here

[jira] [Commented] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412458#comment-16412458 ] Felix Cheung commented on SPARK-23780: -- here [https://github.com/mages/googleVis/blob/master/R

[jira] [Commented] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412457#comment-16412457 ] Felix Cheung commented on SPARK-23780: -- hmm, I think the cause of this is the incompatibility

[jira] [Commented] (SPARK-23497) Sparklyr Applications doesn't disconnect spark driver in client mode

[ https://issues.apache.org/jira/browse/SPARK-23497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410858#comment-16410858 ] Felix Cheung commented on SPARK-23497: -- you should probably follow up with sparklyr/rstudio

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410856#comment-16410856 ] Felix Cheung commented on SPARK-23650: -- can you clarify where you see "R environment i

Re: Toward an "API" for spark images used by the Kubernetes back-end

I like being able to customize the docker image itself - but I realize this thread is more about “API” for the stock image. Environment is nice. Probably we need a way to set custom spark config (as a file??) From: Holden Karau Sent:

Re: "IPython is available, use IPython for PySparkInterpreter"

I think that's a good point - perhaps this shouldn't be a warning. From: Ruslan Dautkhanov Sent: Monday, March 19, 2018 11:10:48 AM To: users Subject: "IPython is available, use IPython for PySparkInterpreter" We're getting " IPython is

Re: Build zeppelin 0.8 with spark 2.3

Are you running with branch-0.8? I think there is a recent change in master for this. From: Felix Cheung <felixcheun...@hotmail.com> Sent: Monday, March 19, 2018 9:49:10 AM To: dev@zeppelin.apache.org; dev@zeppelin.apache.org Subject: Re: Build zeppel

Re: Build zeppelin 0.8 with spark 2.3

Spark 2.3 does not support Scala 2.10. There should be a script to switch Zeppelin to build for Scala 2.11 only... From: Xiaohui Liu Sent: Sunday, March 18, 2018 9:20:13 PM To: dev@zeppelin.apache.org Subject: Build zeppelin 0.8 with spark 2.3

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404198#comment-16404198 ] Felix Cheung commented on SPARK-23650: -- Is there a reason for the broadcast? Could you instead

Re: Custom metrics sink

There is a proposal to expose them. See SPARK-14151 From: Christopher Piggott Sent: Friday, March 16, 2018 1:09:38 PM To: user@spark.apache.org Subject: Custom metrics sink Just for fun, i want to make a stupid program that makes different

Re: Changing how we compute release hashes

+1 there From: Sean Owen <sro...@gmail.com> Sent: Friday, March 16, 2018 9:51:49 AM To: Felix Cheung Cc: rb...@netflix.com; Nicholas Chammas; Spark dev list Subject: Re: Changing how we compute release hashes I think the issue with that is that OS X doesn'

Re: Changing how we compute release hashes

Instead of using gpg to create the sha512 hash file we could just change to using sha512sum? That would output the right format that is in turns verifiable. From: Ryan Blue Sent: Friday, March 16, 2018 8:31:45 AM To: Nicholas Chammas

[jira] [Comment Edited] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401525#comment-16401525 ] Felix Cheung edited comment on SPARK-23650 at 3/16/18 7:16 AM: --- do you mean

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401525#comment-16401525 ] Felix Cheung commented on SPARK-23650: -- do you mean this?   RRunner: Times: boot = 0.010 s, init

Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0

+1 From: Till Rohrmann Sent: Thursday, March 15, 2018 5:45:14 AM To: dev@flink.apache.org Subject: Re: [DISCUSS] Not marking Jira issues as resolved in 1.5.0 as resolved in 1.6.0 +1 for marking bugs as fixed 1.5.0 only On Thu, Mar 15,

Re: How to start practicing Python Spark Streaming in Linux?

It’s best to start with Structured Streaming https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#tab_python_0 https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#tab_python_0 _ From: Aakash Basu

Re: Too many open files on Bucketing sink

I have seen this before as well. My workaround was to limit the number of parallelism but it is the unfortunate effect of limiting the number of processing tasks also (and so slowing things down) Another alternative is to have bigger buckets (and smaller number of buckets) Not sure if there

[jira] [Commented] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398175#comment-16398175 ] Felix Cheung commented on SPARK-23618: -- I think this is because the user isn't in the user role list

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398174#comment-16398174 ] Felix Cheung commented on SPARK-23650: -- I see one RRunner - do you have more of the log? > S

[jira] [Commented] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

[ https://issues.apache.org/jira/browse/SPARK-23632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397213#comment-16397213 ] Felix Cheung commented on SPARK-23632: -- could you explain how you think these environment variables

[jira] [Commented] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397211#comment-16397211 ] Felix Cheung commented on SPARK-23618: -- [~foxish] - Jira has a different user role system, I've

Re: [VOTE] Accept Pinot into Apache Incubator

+1 On Sun, Mar 11, 2018 at 5:34 AM Willem Jiang wrote: > +1 (binding) > > > Willem Jiang > > Blog: http://willemjiang.blogspot.com (English) > http://jnn.iteye.com (Chinese) > Twitter: willemjiang > Weibo: 姜宁willem > > On Sun, Mar 11, 2018 at 7:51 PM, Pierre

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396607#comment-16396607 ] Felix Cheung commented on SPARK-23650: -- which system/platform are you running on? > Slow SparkR

[jira] [Commented] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

[ https://issues.apache.org/jira/browse/SPARK-23632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396606#comment-16396606 ] Felix Cheung commented on SPARK-23632: -- well, if download of packages is taking that long

Re: [DISCUSS] Apache Pinot Incubator Proposal

Hi Kishore - do you need one more mentor? On Tue, Feb 13, 2018 at 12:10 AM kishore g wrote: > Hello, > > I would like to propose Pinot as an Apache Incubator project. The proposal > is available as a draft at https://wiki.apache.org/incubator/PinotProposal. > I > have also

[jira] [Commented] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

[ https://issues.apache.org/jira/browse/SPARK-23632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392507#comment-16392507 ] Felix Cheung commented on SPARK-23632: -- To clarify, are you running into problem because the package

[jira] [Updated] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23291: - Affects Version/s: 2.1.2 2.2.0 2.3.0 > Spa

[jira] [Resolved] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-23291. -- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.4.0

IPMC join request

Hi all, I'd like to join IPMC, initially to help mentor Dr Elephant as incubator project but also looking forward to help mentor other Apache incubator projects. I am PPMC/PMC of Apache Zeppelin (since incubation to TLP) and PMC of Apache Spark, Release Manager for releases. Thanks! Felix

[jira] [Assigned] (SPARK-22430) Unknown tag warnings when building R docs with Roxygen 6.0.1

[ https://issues.apache.org/jira/browse/SPARK-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-22430: Assignee: Rekha Joshi > Unknown tag warnings when building R docs with Roxygen 6.

[jira] [Resolved] (SPARK-22430) Unknown tag warnings when building R docs with Roxygen 6.0.1

[ https://issues.apache.org/jira/browse/SPARK-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-22430. -- Resolution: Fixed Fix Version/s: 2.4.0 Target Version/s: 2.4.0 > Unknown

Re: Question on Spark-kubernetes integration

For pyspark specifically IMO should be very high on the list to port back... As for roadmap - should be sharing more soon. From: lucas.g...@gmail.com <lucas.g...@gmail.com> Sent: Friday, March 2, 2018 9:41:46 PM To: user@spark.apache.org Cc: Felix Cheung S

Re: Question on Spark-kubernetes integration

That's in the plan. We should be sharing a bit more about the roadmap in future releases shortly. In the mean time this is in the official documentation on what is coming: https://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work This supports started as a fork of the Apache

Re: Welcoming some new committers

Congrats and welcome! From: Dongjoon Hyun Sent: Friday, March 2, 2018 4:27:10 PM To: Spark dev list Subject: Re: Welcoming some new committers Congrats to all! Bests, Dongjoon. On Fri, Mar 2, 2018 at 4:13 PM, Wenchen Fan

Re: Using bundler for Jekyll?

Also part of the problem is that the latest news panel is static on each page, so any new link added changes hundreds of files? From: holden.ka...@gmail.com on behalf of Holden Karau Sent: Thursday, March 1, 2018

Re: Help needed in R documentation generation

; Sent: Tuesday, February 27, 2018 10:26:23 AM To: Felix Cheung Cc: Mihály Tóth; Mihály Tóth; dev@spark.apache.org Subject: Re: Help needed in R documentation generation I followed Misi's instructions: - click on https://dist.apache.org/repos/dist/dev/spark/v2.3.0-rc5-docs/_site/api/R/index.html -

Re: Help needed in R documentation generation

sut...@gmail.com> Sent: Tuesday, February 27, 2018 9:13:18 AM To: Felix Cheung Cc: Mihály Tóth; dev@spark.apache.org Subject: Re: Help needed in R documentation generation Hi, Earlier, at https://spark.apache.org/docs/latest/api/R/index.html I see 1. sin as a title 2. description describe

Re: Help needed in R documentation generation

ctions. This sounds like a bug in the documentation of Spark R, does'nt it? Shall I file a Jira about it? Locally I ran SPARK_HOME/R/create-docs.sh and it returned successfully. Unfortunately with the result mentioned above. Best Regards, Misi From: Felix Cheung <f

Re: Spark on K8s - using files fetched by init-container?

Yes you were pointing to HDFS on a loopback address... From: Jenna Hoole Sent: Monday, February 26, 2018 1:11:35 PM To: Yinan Li; user@spark.apache.org Subject: Re: Spark on K8s - using files fetched by init-container? Oh, duh. I

Re: [VOTE] Spark 2.3.0 (RC5)

+1 Tested R: install from package, CRAN tests, manual tests, help check, vignettes check Filed this https://issues.apache.org/jira/browse/SPARK-23461 This is not a regression so not a blocker of the release. Tested this on win-builder and r-hub. On r-hub on multiple platforms everything

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377913#comment-16377913 ] Felix Cheung commented on SPARK-23206: -- [~elu] Hi Edwina, we are interesting in this as well. We

Re: Help needed in R documentation generation

8 8:06:59 AM To: Felix Cheung Cc: dev@spark.apache.org Subject: Re: Help needed in R documentation generation I see. When I click on such a selected function, like 'sin' the page falls apart and does not tell anything about sin function. How is it supposed to work when all functions link to th

Re: Help needed in R documentation generation

This is recent change. The html file column_math_functions.html should have the right help content. What is the problem you are experiencing? From: Mihály Tóth Sent: Sunday, February 25, 2018 10:42:50 PM To: dev@spark.apache.org Subject:

Re: Github pull requests

Re JIRA - the merge PR script in Spark closes the JIRA automatically.. _ From: Julian Hyde Sent: Wednesday, February 21, 2018 8:46 PM Subject: Re: Github pull requests To: Jonas Pfefferle Cc: , Patrick Stuedi

Re: [graphframes]how Graphframes Deal With BidirectionalRelationships

No it does not support bi directional edges as of now. _ From: xiaobo <guxiaobo1...@qq.com> Sent: Tuesday, February 20, 2018 4:35 AM Subject: Re: [graphframes]how Graphframes Deal With BidirectionalRelationships To: Felix Cheung <felixcheun...@hotmail.co

Re: [VOTE] Spark 2.3.0 (RC4)

not be in the release) Thanks! _ From: Shivaram Venkataraman <shiva...@eecs.berkeley.edu> Sent: Tuesday, February 20, 2018 2:24 AM Subject: Re: [VOTE] Spark 2.3.0 (RC4) To: Felix Cheung <felixcheun...@hotmail.com> Cc: Sean Owen <sro...@gmail.com>, dev <

Re: [VOTE] Spark 2.3.0 (RC4)

Felix Cheung <felixcheun...@hotmail.com> Cc: dev <dev@spark.apache.org> Maybe I misunderstand, but I don't see any .iml file in the 4 results on that page? it looks reasonable. On Mon, Feb 19, 2018 at 8:02 PM Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotma

Re: [VOTE] Spark 2.3.0 (RC4)

Any idea with sql func docs search result returning broken links as below? From: Felix Cheung <felixcheun...@hotmail.com> Sent: Sunday, February 18, 2018 10:05:22 AM To: Sameer Agarwal; Sameer Agarwal Cc: dev Subject: Re: [VOTE] Spark 2.3.0 (RC4) Quick que

Re: [graphframes]how Graphframes Deal With Bidirectional Relationships

Generally that would be the approach. But since you have effectively double the number of edges this will likely affect the scale your job will run. From: xiaobo Sent: Monday, February 19, 2018 3:22:02 AM To: user@spark.apache.org Subject:

[jira] [Updated] (SPARK-23461) vignettes should include model predictions for some ML models

[ https://issues.apache.org/jira/browse/SPARK-23461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23461: - Description: eg.  Linear Support Vector Machine (SVM) Classifier h4. Logistic Regression Tree

Re: [VOTE] Spark 2.3.0 (RC4)

Quick questions: is there search link for sql functions quite right? https://dist.apache.org/repos/dist/dev/spark/v2.3.0-rc4-docs/_site/api/sql/search.html?q=app this file shouldn't be included? https://dist.apache.org/repos/dist/dev/spark/v2.3.0-rc4-bin/spark-parent_2.11.iml

Re: Does Pyspark Support Graphx?

Hi - I’m maintaining it. As of now there is an issue with 2.2 that breaks personalized page rank, and that’s largely the reason there isn’t a release for 2.2 support. There are attempts to address this issue - if you are interested we would love for your help.

[jira] [Created] (SPARK-23461) vignettes should include model predictions for some ML models

Felix Cheung created SPARK-23461: Summary: vignettes should include model predictions for some ML models Key: SPARK-23461 URL: https://issues.apache.org/jira/browse/SPARK-23461 Project: Spark

[jira] [Commented] (SPARK-23435) R tests should support latest testthat

[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368351#comment-16368351 ] Felix Cheung commented on SPARK-23435: -- Working on this. Debugging a problem. > R tests sho

[jira] [Assigned] (SPARK-23435) R tests should support latest testthat

[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-23435: Assignee: Felix Cheung > R tests should support latest testt

[jira] [Created] (SPARK-23435) R tests should support latest testthat

Felix Cheung created SPARK-23435: Summary: R tests should support latest testthat Key: SPARK-23435 URL: https://issues.apache.org/jira/browse/SPARK-23435 Project: Spark Issue Type: Bug

<    3   4   5   6   7   8   9   10   11   12   >