Re: Cutting the RC for Spark 2.2.1 release

Thanks Dongjoon! I will track that. From: Dongjoon Hyun <dongjoon.h...@gmail.com> Sent: Wednesday, November 8, 2017 7:41:20 PM To: Holden Karau Cc: Felix Cheung; dev@spark.apache.org Subject: Re: Cutting the RC for Spark 2.2.1 release It's great,

Cutting the RC for Spark 2.2.1 release

Hi! As we are closing down on the few known issues I think we are ready to tag and cut the 2.2.1 release. If you are aware of any issue that you think should go into this release please feel free to ping me and mark the JIRA as targeting 2.2.1. I will be scrubbing JIRA in the next few days.

Re: Kicking off the process around Spark 2.2.1

canfly.ca> Sent: Thursday, November 2, 2017 12:47:13 PM To: Reynold Xin Cc: Felix Cheung; Sean Owen; dev@spark.apache.org Subject: Re: Kicking off the process around Spark 2.2.1 I agree, except in this case we probably want some of the fixes that are going into the maintenance release t

[jira] [Resolved] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-22281. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.3.0

[jira] [Resolved] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-22327. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.3.0

Re: Zeppelin 0.7.2 integration with Presto 0.184

Great. Could someone open a JIRA on this? Unless the policy is changing, this can be a blocker for Presto for 0.8 release

[jira] [Commented] (SPARK-22430) Unknown tag warnings when building R docs with Roxygen 6.0.1

[ https://issues.apache.org/jira/browse/SPARK-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237719#comment-16237719 ] Felix Cheung commented on SPARK-22430: -- I am seeing it too. I think we can just remove the tag

Re: Kicking off the process around Spark 2.2.1

For the 2.2.1, we are still working through a few bugs. Hopefully it won't be long. From: Kevin Grealish <kevin...@microsoft.com> Sent: Thursday, November 2, 2017 9:51:56 AM To: Felix Cheung; Sean Owen; Holden Karau Cc: dev@spark.apache.org Subject: RE: K

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236103#comment-16236103 ] Felix Cheung commented on SPARK-22344: -- Yes to both. If SPARK_HOME is set before calling

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233762#comment-16233762 ] Felix Cheung commented on SPARK-22344: -- Maybe just delete the directory returned from sparkCachePath

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233751#comment-16233751 ] Felix Cheung commented on SPARK-22344: -- Hmm yes we do have to know if it has just found

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226197#comment-16226197 ] Felix Cheung commented on SPARK-22344: -- Yes I think we should do just that. Might need to delete

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225770#comment-16225770 ] Felix Cheung commented on SPARK-22344: -- Kinda. We have some alternate code paths and the packageName

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224139#comment-16224139 ] Felix Cheung commented on SPARK-22344: -- Kinda we don't have any uninstall feature though

[jira] [Comment Edited] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223228#comment-16223228 ] Felix Cheung edited comment on SPARK-22344 at 10/28/17 4:26 AM: I think

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223228#comment-16223228 ] Felix Cheung commented on SPARK-22344: -- I think the PR addresses other cases, but what do we want

Re: Kicking off the process around Spark 2.2.1

2017 4:39:15 AM To: Holden Karau Cc: Felix Cheung; dev@spark.apache.org Subject: Re: Kicking off the process around Spark 2.2.1 It would be reasonably consistent with the timing of other x.y.1 releases, and more release managers sounds useful, yeah. Note also that in theory the code freeze for

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220083#comment-16220083 ] Felix Cheung commented on SPARK-22344: -- why is hive there when enableHiveSupport should be off? what

[jira] [Comment Edited] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220080#comment-16220080 ] Felix Cheung edited comment on SPARK-22344 at 10/26/17 7:25 AM

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220080#comment-16220080 ] Felix Cheung commented on SPARK-22344: -- this is what I see on a clean machine tracking access/create

Re: CRAN SparkR package removed?

Yes - unfortunately something was found after it was published and made available publicly. We have a JIRA on this and are working on the best course of action. _ From: Holden Karau > Sent: Wednesday, October 25,

[jira] [Updated] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22344: - Affects Version/s: 2.3.0 1.6.3 2.2.0 > Preven

[jira] [Commented] (SPARK-21616) SparkR 2.3.0 migration guide, release note

[ https://issues.apache.org/jira/browse/SPARK-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218104#comment-16218104 ] Felix Cheung commented on SPARK-21616: -- True, I don't know if we are tracking changes to programming

[jira] [Commented] (SPARK-21616) SparkR 2.3.0 migration guide, release note

[ https://issues.apache.org/jira/browse/SPARK-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216439#comment-16216439 ] Felix Cheung commented on SPARK-21616: -- SPARK-17902 > SparkR 2.3.0 migration guide, release n

[jira] [Commented] (SPARK-21208) Ability to "setLocalProperty" from sc, in sparkR

[ https://issues.apache.org/jira/browse/SPARK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216438#comment-16216438 ] Felix Cheung commented on SPARK-21208: -- hi - any taker on this? > Ability to "setLocal

[jira] [Comment Edited] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214751#comment-16214751 ] Felix Cheung edited comment on SPARK-22281 at 10/23/17 7:30 AM: ok, I

[jira] [Comment Edited] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214751#comment-16214751 ] Felix Cheung edited comment on SPARK-22281 at 10/23/17 7:10 AM: ok, I

[jira] [Commented] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214751#comment-16214751 ] Felix Cheung commented on SPARK-22281: -- ok, I have a solution for both. it turns out the fix for glm

[jira] [Updated] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22281: - Description: cAs discussed here http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE

[jira] [Commented] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214431#comment-16214431 ] Felix Cheung commented on SPARK-22281: -- tried a few things. If we remove the {code} @param {code

[jira] [Updated] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22327: - Description: with warning * checking CRAN incoming feasibility ... WARNING Maintainer: 'Shivaram

[jira] [Comment Edited] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213798#comment-16213798 ] Felix Cheung edited comment on SPARK-22327 at 10/21/17 7:45 AM

[jira] [Commented] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213798#comment-16213798 ] Felix Cheung commented on SPARK-22327: -- in contrast, this is from master * checking CRAN incoming

[jira] [Comment Edited] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213798#comment-16213798 ] Felix Cheung edited comment on SPARK-22327 at 10/21/17 7:45 AM

[jira] [Updated] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22327: - Description: with warning * checking CRAN incoming feasibility ... WARNING Maintainer: 'Shivaram

[jira] [Updated] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22327: - Description: with warning Insufficient package version (submitted: 2.0.3, existing: 2.1.2) We

[jira] [Commented] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213781#comment-16213781 ] Felix Cheung commented on SPARK-22327: -- https://amplab.cs.berkeley.edu/jenkins/job

[jira] [Updated] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22327: - Affects Version/s: 2.3.0 > R CRAN check fails on non-latest branc

[jira] [Updated] (SPARK-22327) R CRAN check fails on non-latest branches

[ https://issues.apache.org/jira/browse/SPARK-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22327: - Affects Version/s: 2.2.1 > R CRAN check fails on non-latest branc

[jira] [Created] (SPARK-22327) R CRAN check fails on non-latest branches

Felix Cheung created SPARK-22327: Summary: R CRAN check fails on non-latest branches Key: SPARK-22327 URL: https://issues.apache.org/jira/browse/SPARK-22327 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17608) Long type has incorrect serialization/deserialization

[ https://issues.apache.org/jira/browse/SPARK-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205379#comment-16205379 ] Felix Cheung commented on SPARK-17608: -- any taker on this? > Long type has incorrect serializat

[jira] [Updated] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22281: - Description: cAs discussed here http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE

[jira] [Commented] (SPARK-22281) Handle R method breaking signature changes

[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205313#comment-16205313 ] Felix Cheung commented on SPARK-22281: -- And here for all r-devel WARN https://cran.r-project.org/web

[jira] [Created] (SPARK-22281) Handle R method breaking signature changes

Felix Cheung created SPARK-22281: Summary: Handle R method breaking signature changes Key: SPARK-22281 URL: https://issues.apache.org/jira/browse/SPARK-22281 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19700) Design an API for pluggable scheduler implementations

[ https://issues.apache.org/jira/browse/SPARK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200659#comment-16200659 ] Felix Cheung commented on SPARK-19700: -- Not that I'm aware of - I agree it is very important to take

[jira] [Commented] (SPARK-17275) Flaky test: org.apache.spark.deploy.RPackageUtilsSuite.jars that don't exist are skipped and print warning

[ https://issues.apache.org/jira/browse/SPARK-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196380#comment-16196380 ] Felix Cheung commented on SPARK-17275: -- perhaps we should close this? it's been a year... > Fl

Re: [VOTE] Spark 2.1.2 (RC4)

Thanks Nick, Hyukjin. Yes this seems to be a longer standing issue on RHEL with respect to forking. From: Nick Pentreath Sent: Friday, October 6, 2017 6:16:53 AM To: Hyukjin Kwon Cc: dev Subject: Re: [VOTE] Spark 2.1.2 (RC4) Ah yes - I

[jira] [Commented] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194141#comment-16194141 ] Felix Cheung commented on SPARK-22202: -- [~holden.ka...@gmail.com] actually, I think for R we would

[jira] [Updated] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22202: - Priority: Minor (was: Major) > Release tgz content differences for python an

[jira] [Commented] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193239#comment-16193239 ] Felix Cheung commented on SPARK-22202: -- [~holden.ka...@gmail.com] would you be concerned

[jira] [Commented] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193238#comment-16193238 ] Felix Cheung commented on SPARK-22202: -- Yes, exactly. > Release tgz content differences for pyt

Re: Nightly builds for master branch failed

Thanks Shane! From: shane knapp <skn...@berkeley.edu> Sent: Thursday, October 5, 2017 9:14:54 AM To: Felix Cheung Cc: Liwei Lin; Spark dev list Subject: Re: Nightly builds for master branch failed yep, it was a corrupted jar on amp-jenkins-worker-01. i g

Re: [VOTE] Spark 2.1.2 (RC4)

+1 Tested SparkR package manually on multiple platforms and checked different Hadoop release jar. And previously tested the last RC on different R releases (see the last RC vote thread) I found some differences in bin release jars created by the different options when running the

[jira] [Updated] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22202: - Description: As a follow up to SPARK-22167, currently we are running different profiles/steps

Re: Nightly builds for master branch failed

Hmm, sounds like some sort of corruption of the maven directory on the Jenkins box... From: Liwei Lin Sent: Wednesday, October 4, 2017 6:52:54 PM To: Spark dev list Subject: Nightly builds for master branch failed

Re: Disabling Closed -> Reopened transition for non-committers

To be sure, this is only for JIRA and not for github PR, right? If then +1 but I think the access control on JIRA does not necessarily match the committer list, and is manually maintained, last I hear. From: Sean Owen Sent: Wednesday,

[jira] [Updated] (SPARK-22202) Release tgz content differences for python and R

[ https://issues.apache.org/jira/browse/SPARK-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22202: - Description: As a follow up to SPARK-22167, currently we are running different profiles/steps

[jira] [Created] (SPARK-22202) Release tgz content differences for python and R

Felix Cheung created SPARK-22202: Summary: Release tgz content differences for python and R Key: SPARK-22202 URL: https://issues.apache.org/jira/browse/SPARK-22202 Project: Spark Issue Type

[jira] [Commented] (SPARK-22167) Spark Packaging w/R distro issues

[ https://issues.apache.org/jira/browse/SPARK-22167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189954#comment-16189954 ] Felix Cheung commented on SPARK-22167: -- There are likely 2 stages to this. More pressing might

[jira] [Comment Edited] (SPARK-22063) Upgrade lintr to latest commit sha1 ID

[ https://issues.apache.org/jira/browse/SPARK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187574#comment-16187574 ] Felix Cheung edited comment on SPARK-22063 at 10/2/17 9:16 AM: --- surely, I

[jira] [Commented] (SPARK-22063) Upgrade lintr to latest commit sha1 ID

[ https://issues.apache.org/jira/browse/SPARK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187574#comment-16187574 ] Felix Cheung commented on SPARK-22063: -- surely, I think we could even start with something simple

[jira] [Commented] (SPARK-22167) Spark Packaging w/R distro issues

[ https://issues.apache.org/jira/browse/SPARK-22167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187156#comment-16187156 ] Felix Cheung commented on SPARK-22167: -- I think I'd propose a change on this part of the release

Re: [VOTE] Spark 2.1.2 (RC2)

-1 (Sorry) spark-2.1.2-bin-hadoop2.7.tgz is missing the R directory, not sure why yet. Tested on multiple platform as source package, (against 2.1.1 jar) seemed fine except this WARNING on R-devel * checking for code/documentation mismatches ... WARNING Codoc mismatches from documentation

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180261#comment-16180261 ] Felix Cheung commented on SPARK-15799: -- I commented on the PR. I don't think there is any code

Re: using R with Spark

et.net/> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData<http://twitter.com/BobLovesData> From: Georg Heiler [mailto:georg.kf.hei...@gmail.com] Sent: Sunday, September 24, 2017 3:39 PM To: Felix Cheung <felixcheun...@hot

Re: using R with Spark

If you google it you will find posts or info on how to connect it to different cloud and hadoop/spark vendors. From: Georg Heiler <georg.kf.hei...@gmail.com> Sent: Sunday, September 24, 2017 1:39:09 PM To: Felix Cheung; Adaryl Wakefield; user@spark.apac

Re: using R with Spark

Both are free to use; you can use sparklyr from the R shell without RStudio (but you probably want an IDE) From: Adaryl Wakefield Sent: Sunday, September 24, 2017 11:19:24 AM To: user@spark.apache.org Subject: using R with Spark

Re: graphframes on cluster

Could you include the code where it fails? Generally the best way to use gf is to use the --packages options with spark-submit command From: Imran Rajjad Sent: Wednesday, September 20, 2017 5:47:27 AM To: user @spark Subject: graphframes on

[jira] [Commented] (SPARK-18131) Support returning Vector/Dense Vector from backend

[ https://issues.apache.org/jira/browse/SPARK-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169579#comment-16169579 ] Felix Cheung commented on SPARK-18131: -- bump. I think this is a real big problem - results from

[jira] [Commented] (SPARK-21802) Make sparkR MLP summary() expose probability column

[ https://issues.apache.org/jira/browse/SPARK-21802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169575#comment-16169575 ] Felix Cheung commented on SPARK-21802: -- yes if this is from the prediction (with rawPrediction etc

[jira] [Commented] (SPARK-21802) Make sparkR MLP summary() expose probability column

[ https://issues.apache.org/jira/browse/SPARK-21802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169487#comment-16169487 ] Felix Cheung commented on SPARK-21802: -- Can you clarify where you see it? I just ran against

Re: [VOTE] Spark 2.1.2 (RC1)

Yes ;) From: Xiao Li <gatorsm...@gmail.com> Sent: Friday, September 15, 2017 2:22:03 PM To: Holden Karau Cc: Ryan Blue; Denny Lee; Felix Cheung; Sean Owen; dev@spark.apache.org Subject: Re: [VOTE] Spark 2.1.2 (RC1) Sorry, this release candidate is

Re: [VOTE] Spark 2.1.2 (RC1)

+1 tested SparkR package on Windows, r-hub, Ubuntu. _ From: Sean Owen > Sent: Thursday, September 14, 2017 3:12 PM Subject: Re: [VOTE] Spark 2.1.2 (RC1) To: Holden Karau >,

Re: Official Docker image build and release process

Anyone? From: Felix Cheung <felixcheun...@hotmail.com> Sent: Thursday, September 14, 2017 8:45:07 AM To: dev@zeppelin.apache.org Subject: Official Docker image build and release process Hi! Where do we have information on this? Thanks

Official Docker image build and release process

Hi! Where do we have information on this? Thanks

Re: 2.1.2 maintenance release?

maintenance release? To: Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>>, Holden Karau <hol...@pigscanfly.ca<mailto:hol...@pigscanfly.ca>>, Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>>, dev <dev@spark.apache.org<mail

Re: Cloudera Data Science Workbench and Zeppelin

Having used it myself, it looks to be a different technology at several different levels, so I'm not sure it is based on Zeppelin in any way. From: Mich Talebzadeh Sent: Sunday, September 10, 2017 1:07:55 AM To:

[jira] [Commented] (SPARK-20684) expose createOrReplaceGlobalTempView/createGlobalTempView and dropGlobalTempView in SparkR

[ https://issues.apache.org/jira/browse/SPARK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160420#comment-16160420 ] Felix Cheung commented on SPARK-20684: -- I"m making this primary JIRA for tracking this

[jira] [Updated] (SPARK-20684) expose createOrReplaceGlobalTempView/createGlobalTempView and dropGlobalTempView in SparkR

[ https://issues.apache.org/jira/browse/SPARK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-20684: - Summary: expose createOrReplaceGlobalTempView/createGlobalTempView and dropGlobalTempView

[jira] [Reopened] (SPARK-20684) expose createGlobalTempView and dropGlobalTempView in SparkR

[ https://issues.apache.org/jira/browse/SPARK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reopened SPARK-20684: -- > expose createGlobalTempView and dropGlobalTempView in Spa

Re: Queries with streaming sources must be executed with writeStream.start()

What is newDS? If it is a Streaming Dataset/DataFrame (since you have writeStream there) then there seems to be an issue preventing toJSON to work. From: kant kodali Sent: Saturday, September 9, 2017 4:04:33 PM To: user @spark Subject:

Re: How to convert Row to JSON in Java?

toJSON on Dataset/DataFrame? From: kant kodali Sent: Saturday, September 9, 2017 4:15:49 PM To: user @spark Subject: How to convert Row to JSON in Java? Hi All, How to convert Row to JSON in Java? It would be nice to have .toJson() method

[jira] [Updated] (SPARK-21128) Running R tests multiple times failed due to pre-exiting "spark-warehouse" / "metastore_db"

[ https://issues.apache.org/jira/browse/SPARK-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-21128: - Target Version/s: 2.2.1, 2.3.0 (was: 2.3.0) Fix Version/s: 2.2.1 > Running R te

Re: 2.1.2 maintenance release?

+1 on both 2.1.2 and 2.2.1 And would try to help and/or wrangle the release if needed. (Note: trying to backport a few changes to branch-2.1 right now) From: Sean Owen Sent: Friday, September 8, 2017 12:05:28 AM To: Holden Karau; dev

Re: Putting Kafka 0.8 behind an (opt-in) profile

+1 From: Cody Koeninger Sent: Tuesday, September 5, 2017 8:12:07 AM To: Sean Owen Cc: dev Subject: Re: Putting Kafka 0.8 behind an (opt-in) profile +1 to going ahead and giving a deprecation warning now On Tue, Sep 5, 2017 at 6:39 AM, Sean

[jira] [Comment Edited] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152973#comment-16152973 ] Felix Cheung edited comment on SPARK-21727 at 9/4/17 11:08 PM: --- precisely

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152973#comment-16152973 ] Felix Cheung commented on SPARK-21727: -- precisely. as far as I can tell, everything should "

Re: sparkR 3rd library

Can you include the code you call spark.lapply? From: patcharee Sent: Sunday, September 3, 2017 11:46:40 PM To: spar >> user@spark.apache.org Subject: sparkR 3rd library Hi, I am using spark.lapply to execute an existing R script in

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151589#comment-16151589 ] Felix Cheung commented on SPARK-21727: -- any taker of this change? > Operating on an ArrayT

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151588#comment-16151588 ] Felix Cheung commented on SPARK-21727: -- That is true. I think the documentation is unclear

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151437#comment-16151437 ] Felix Cheung commented on SPARK-21727: -- hmm.. I think that's what the error message is saying {code

[jira] [Commented] (SPARK-12157) Support numpy types as return values of Python UDFs

[ https://issues.apache.org/jira/browse/SPARK-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151336#comment-16151336 ] Felix Cheung commented on SPARK-12157: -- any more thought on this? I think we should at least

Re: [EXTERNAL] Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled

Yap I was able to get this to work with a custom bucketer. A custom bucketer can use the clock given ("processing time") or it can use a timestamp from the data ("event time") for the bucketing path. From: Raja.Aravapalli Sent:

Re: [VOTE][SPIP] SPARK-21190: Vectorized UDFs in Python

+1 on this and like the suggestion of type in string form. Would it be correct to assume there will be data type check, for example the returned pandas data frame column data types match what are specified. We have seen quite a bit of issues/confusions with that in R. Would it make sense to

Re: Updates on migration guides

+1 think we do migration guide changes for ML and R in separate JIRA/PR/commit but we definition should have it updated before the release. From: linguin@gmail.com Sent: Wednesday, August 30, 2017 8:27:17 AM To: Dongjoon Hyun Cc: Xiao

[jira] [Assigned] (SPARK-21801) SparkR unit test randomly fail on trees

[ https://issues.apache.org/jira/browse/SPARK-21801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-21801: Assignee: Felix Cheung > SparkR unit test randomly fail on tr

[jira] [Resolved] (SPARK-21801) SparkR unit test randomly fail on trees

[ https://issues.apache.org/jira/browse/SPARK-21801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-21801. -- Resolution: Fixed Fix Version/s: 2.3.0 > SparkR unit test randomly fail on tr

Re: [DISCUSS] Release 0.7.3

+1 From: Park Hoon <1am...@gmail.com> Sent: Tuesday, August 29, 2017 8:37:16 AM To: dev@zeppelin.apache.org Subject: Re: Re: [DISCUSS] Release 0.7.3 +1 On Wed, 30 Aug 2017 at 00:21 moon soo Lee wrote: > +1 > > On Tue, Aug 29, 2017 at 5:05 AM

[jira] [Resolved] (SPARK-21805) disable R vignettes code on Windows

[ https://issues.apache.org/jira/browse/SPARK-21805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-21805. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138099#comment-16138099 ] Felix Cheung commented on SPARK-15799: -- [~shivaram] might have more updates. The CRAN submission

<    7   8   9   10   11   12   13   14   15   16   >