[jira] [Updated] (SPARK-21409) Expose state store memory usage in SQL metrics and progress updates

[ https://issues.apache.org/jira/browse/SPARK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21409: Fix Version/s: (was: 3.0.0) 2.3.0 > Expose state store memory usage in SQL metrics

[jira] [Updated] (SPARK-22278) Expose current event time watermark and current processing time in GroupState

[ https://issues.apache.org/jira/browse/SPARK-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22278: Fix Version/s: (was: 3.0.0) 2.3.0 > Expose current event time watermark and current

[jira] [Created] (SPARK-23070) Bump previousSparkVersion in MimaBuild.scala to be 2.2.0

Xiao Li created SPARK-23070: --- Summary: Bump previousSparkVersion in MimaBuild.scala to be 2.2.0 Key: SPARK-23070 URL: https://issues.apache.org/jira/browse/SPARK-23070 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23038) Update docker/spark-test (JDK/OS)

[ https://issues.apache.org/jira/browse/SPARK-23038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23038: - Fix Version/s: 2.4.0 > Update docker/spark-test (JDK/OS) > - > >

[jira] [Resolved] (SPARK-23038) Update docker/spark-test (JDK/OS)

[ https://issues.apache.org/jira/browse/SPARK-23038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-23038. -- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-23069) R doc for describe missing text

[ https://issues.apache.org/jira/browse/SPARK-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325467#comment-16325467 ] Apache Spark commented on SPARK-23069: -- User 'felixcheung' has created a pull request for this

[jira] [Assigned] (SPARK-23069) R doc for describe missing text

[ https://issues.apache.org/jira/browse/SPARK-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23069: Assignee: (was: Apache Spark) > R doc for describe missing text >

[jira] [Assigned] (SPARK-23069) R doc for describe missing text

[ https://issues.apache.org/jira/browse/SPARK-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23069: Assignee: Apache Spark > R doc for describe missing text >

[jira] [Updated] (SPARK-23069) R doc for describe missing text

[ https://issues.apache.org/jira/browse/SPARK-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23069: - Issue Type: Documentation (was: Bug) > R doc for describe missing text >

[jira] [Resolved] (SPARK-23063) Changes to publish the spark-kubernetes package

[ https://issues.apache.org/jira/browse/SPARK-23063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-23063. -- Resolution: Fixed Assignee: Anirudh Ramanathan Fix Version/s: 2.3.0 > Changes

[jira] [Created] (SPARK-23069) R doc for describe missing text

Felix Cheung created SPARK-23069: Summary: R doc for describe missing text Key: SPARK-23069 URL: https://issues.apache.org/jira/browse/SPARK-23069 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325465#comment-16325465 ] Felix Cheung commented on SPARK-23065: -- I have checked the doc, looks good, but I have a few small

[jira] [Resolved] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-23065. -- Resolution: Fixed Fix Version/s: 2.3.0 Target Version/s: 2.3.0 > R API doc

[jira] [Created] (SPARK-23068) R doc build error does not fail build

Felix Cheung created SPARK-23068: Summary: R doc build error does not fail build Key: SPARK-23068 URL: https://issues.apache.org/jira/browse/SPARK-23068 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325438#comment-16325438 ] Wenchen Fan commented on SPARK-22980: - I feel it's not a problem, we never promise `pandas_udf` and

[jira] [Comment Edited] (SPARK-23041) Inconsistent `drop`ing of columns in dataframes

[ https://issues.apache.org/jira/browse/SPARK-23041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325434#comment-16325434 ] Hyukjin Kwon edited comment on SPARK-23041 at 1/14/18 3:38 AM: --- Seems this

[jira] [Commented] (SPARK-23041) Inconsistent `drop`ing of columns in dataframes

[ https://issues.apache.org/jira/browse/SPARK-23041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325434#comment-16325434 ] Hyukjin Kwon commented on SPARK-23041: -- Seems this reproducer throws an error as below by

[jira] [Resolved] (SPARK-22959) Configuration to select the modules for daemon and worker in PySpark

[ https://issues.apache.org/jira/browse/SPARK-22959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22959. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20151

[jira] [Assigned] (SPARK-22959) Configuration to select the modules for daemon and worker in PySpark

[ https://issues.apache.org/jira/browse/SPARK-22959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22959: Assignee: Hyukjin Kwon > Configuration to select the modules for daemon and worker in

[jira] [Assigned] (SPARK-22885) ML test for StructuredStreaming: spark.ml.tuning

[ https://issues.apache.org/jira/browse/SPARK-22885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22885: Assignee: (was: Apache Spark) > ML test for StructuredStreaming: spark.ml.tuning >

[jira] [Commented] (SPARK-22885) ML test for StructuredStreaming: spark.ml.tuning

[ https://issues.apache.org/jira/browse/SPARK-22885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325402#comment-16325402 ] Apache Spark commented on SPARK-22885: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-22885) ML test for StructuredStreaming: spark.ml.tuning

[ https://issues.apache.org/jira/browse/SPARK-22885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22885: Assignee: Apache Spark > ML test for StructuredStreaming: spark.ml.tuning >

[jira] [Commented] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325400#comment-16325400 ] Felix Cheung commented on SPARK-23065: -- Something was definitely cached - not sure what since I have

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325399#comment-16325399 ] Hyukjin Kwon commented on SPARK-22980: -- cc [~cloud_fan] too > Using pandas_udf when inputs are not

[jira] [Commented] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325396#comment-16325396 ] Sameer Agarwal commented on SPARK-23065: No, the jekyll error didn't fail the doc build. We

[jira] [Updated] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-23065: --- Attachment: Screen Shot 2018-01-13 at 3.15.48 PM.png Screen Shot 2018-01-13

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325371#comment-16325371 ] Hyukjin Kwon commented on SPARK-22980: -- They are already well explained in documentation. It expects

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325358#comment-16325358 ] Hyukjin Kwon commented on SPARK-22980: -- The newly added description explains the results diff above.

[jira] [Created] (SPARK-23067) Allow for easier debugging of the docker container

Anirudh Ramanathan created SPARK-23067: -- Summary: Allow for easier debugging of the docker container Key: SPARK-23067 URL: https://issues.apache.org/jira/browse/SPARK-23067 Project: Spark

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325356#comment-16325356 ] Hyukjin Kwon commented on SPARK-22980: -- [~smilegator], I am confused of what you suggest to fix.

[jira] [Assigned] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22980: Assignee: (was: Apache Spark) > Using pandas_udf when inputs are not Pandas's Series

[jira] [Assigned] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22980: Assignee: Apache Spark > Using pandas_udf when inputs are not Pandas's Series or

[jira] [Commented] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325322#comment-16325322 ] Xiao Li commented on SPARK-22980: - length is just an example. The newly added description does not help.

[jira] [Updated] (SPARK-22980) Using pandas_udf when inputs are not Pandas's Series or DataFrame

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22980: Summary: Using pandas_udf when inputs are not Pandas's Series or DataFrame (was: Clarify the length of

[jira] [Reopened] (SPARK-22980) Clarify the length of each series is of each batch within scalar Pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-22980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-22980: - Assignee: (was: Hyukjin Kwon) > Clarify the length of each series is of each batch within scalar

[jira] [Resolved] (SPARK-22790) add a configurable factor to describe HadoopFsRelation's size

[ https://issues.apache.org/jira/browse/SPARK-22790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22790. - Resolution: Fixed Assignee: Nan Zhu Fix Version/s: 2.3.0 > add a configurable factor to

[jira] [Resolved] (SPARK-23036) Add withGlobalTempView for testing

[ https://issues.apache.org/jira/browse/SPARK-23036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23036. - Resolution: Fixed Assignee: xubo245 Fix Version/s: 2.3.0 > Add withGlobalTempView for

[jira] [Commented] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325229#comment-16325229 ] Felix Cheung commented on SPARK-23065: -- Was the Jekyll error failed the doc build? Just want to make

[jira] [Resolved] (SPARK-22870) Dynamic allocation should allow 0 idle time

[ https://issues.apache.org/jira/browse/SPARK-22870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22870. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20080

[jira] [Assigned] (SPARK-22870) Dynamic allocation should allow 0 idle time

[ https://issues.apache.org/jira/browse/SPARK-22870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22870: - Assignee: Yuming Wang > Dynamic allocation should allow 0 idle time >

[jira] [Assigned] (SPARK-23043) Upgrade json4s-jackson to 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23043: - Assignee: Takako Shimamoto Priority: Minor (was: Major) > Upgrade json4s-jackson to 3.5.3

[jira] [Resolved] (SPARK-23043) Upgrade json4s-jackson to 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-23043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23043. --- Resolution: Fixed Fix Version/s: 2.4.0 Resolved by https://github.com/apache/spark/pull/20233

[jira] [Commented] (SPARK-23066) Master Page increase master start-up time.

[ https://issues.apache.org/jira/browse/SPARK-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325168#comment-16325168 ] Sean Owen commented on SPARK-23066: --- This should be in logs, not in the crowded UI. > Master Page

[jira] [Commented] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325166#comment-16325166 ] Sean Owen commented on SPARK-23060: --- I see. This might otherwise be 'pipe' but that's not possible. The

[jira] [Assigned] (SPARK-23039) Fix the bug in alter table set location.

[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23039: Assignee: Apache Spark > Fix the bug in alter table set location. >

[jira] [Assigned] (SPARK-23039) Fix the bug in alter table set location.

[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23039: Assignee: (was: Apache Spark) > Fix the bug in alter table set location. >

[jira] [Commented] (SPARK-23039) Fix the bug in alter table set location.

[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325143#comment-16325143 ] Apache Spark commented on SPARK-23039: -- User 'xubo245' has created a pull request for this issue:

[jira] [Commented] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325142#comment-16325142 ] Gianmarco Donetti commented on SPARK-23060: --- Yeah exactly. There already is a pipe method for

[jira] [Updated] (SPARK-23039) Fix the bug in alter table set location.

[ https://issues.apache.org/jira/browse/SPARK-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated SPARK-23039: Description: TOBO work: Fix the bug in alter table set location.

[jira] [Commented] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325110#comment-16325110 ] Hyukjin Kwon commented on SPARK-23060: -- [~gianmarco.donetti], I think you refer Pandas's pipe -

[jira] [Commented] (SPARK-22923) Non-equi join(theta join) should use sort merge join

[ https://issues.apache.org/jira/browse/SPARK-22923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325074#comment-16325074 ] Marco Gaido commented on SPARK-22923: - I dob't think SortMergeJoinExec can be used, since the

[jira] [Commented] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325068#comment-16325068 ] Sameer Agarwal commented on SPARK-23065: Thanks for verifying this [~felixcheung]! While the

[jira] [Assigned] (SPARK-23065) R API doc empty in Spark 2.3.0 RC1

[ https://issues.apache.org/jira/browse/SPARK-23065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal reassigned SPARK-23065: -- Assignee: Sameer Agarwal > R API doc empty in Spark 2.3.0 RC1 >

[jira] [Assigned] (SPARK-23066) Master Page increase master start-up time.

[ https://issues.apache.org/jira/browse/SPARK-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23066: Assignee: Apache Spark > Master Page increase master start-up time. >

[jira] [Commented] (SPARK-23066) Master Page increase master start-up time.

[ https://issues.apache.org/jira/browse/SPARK-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325049#comment-16325049 ] Apache Spark commented on SPARK-23066: -- User 'guoxiaolongzte' has created a pull request for this

[jira] [Assigned] (SPARK-23066) Master Page increase master start-up time.

[ https://issues.apache.org/jira/browse/SPARK-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23066: Assignee: (was: Apache Spark) > Master Page increase master start-up time. >

[jira] [Updated] (SPARK-23066) Master Page increase master start-up time.

[ https://issues.apache.org/jira/browse/SPARK-23066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-23066: --- Summary: Master Page increase master start-up time. (was: Master Page ncrease master

[jira] [Commented] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325045#comment-16325045 ] Apache Spark commented on SPARK-23060: -- User 'gianmarcodonetti' has created a pull request for this

[jira] [Assigned] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23060: Assignee: (was: Apache Spark) > RDD's apply function > > >

[jira] [Assigned] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23060: Assignee: Apache Spark > RDD's apply function > > >

[jira] [Comment Edited] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325043#comment-16325043 ] Gianmarco Donetti edited comment on SPARK-23060 at 1/13/18 8:42 AM:

[jira] [Comment Edited] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325043#comment-16325043 ] Gianmarco Donetti edited comment on SPARK-23060 at 1/13/18 8:42 AM:

[jira] [Commented] (SPARK-23060) RDD's apply function

[ https://issues.apache.org/jira/browse/SPARK-23060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325043#comment-16325043 ] Gianmarco Donetti commented on SPARK-23060: --- The reason of this need of mine, is the absence of

[jira] [Created] (SPARK-23066) Master Page ncrease master start-up time.

guoxiaolongzte created SPARK-23066: -- Summary: Master Page ncrease master start-up time. Key: SPARK-23066 URL: https://issues.apache.org/jira/browse/SPARK-23066 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-23048) Update mllib docs to replace OneHotEncoder with OneHotEncoderEstimator

[ https://issues.apache.org/jira/browse/SPARK-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23048: Assignee: Apache Spark > Update mllib docs to replace OneHotEncoder with

[jira] [Assigned] (SPARK-23048) Update mllib docs to replace OneHotEncoder with OneHotEncoderEstimator

[ https://issues.apache.org/jira/browse/SPARK-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23048: Assignee: (was: Apache Spark) > Update mllib docs to replace OneHotEncoder with

[jira] [Commented] (SPARK-23048) Update mllib docs to replace OneHotEncoder with OneHotEncoderEstimator

[ https://issues.apache.org/jira/browse/SPARK-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325029#comment-16325029 ] Apache Spark commented on SPARK-23048: -- User 'viirya' has created a pull request for this issue: