[jira] [Resolved] (SPARK-30329) add iterator/foreach methods for Vectors

[ https://issues.apache.org/jira/browse/SPARK-30329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30329. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26982

[jira] [Resolved] (SPARK-30321) log weightSum in Algo that has weights support

[ https://issues.apache.org/jira/browse/SPARK-30321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30321. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26972

[jira] [Assigned] (SPARK-30321) log weightSum in Algo that has weights support

[ https://issues.apache.org/jira/browse/SPARK-30321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30321: Assignee: Huaxin Gao > log weightSum in Algo that has weights support >

[jira] [Resolved] (SPARK-30380) Refactor RandomForest.findSplits

[ https://issues.apache.org/jira/browse/SPARK-30380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30380. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27040

[jira] [Assigned] (SPARK-30380) Refactor RandomForest.findSplits

[ https://issues.apache.org/jira/browse/SPARK-30380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30380: Assignee: zhengruifeng > Refactor RandomForest.findSplits >

[jira] [Resolved] (SPARK-30378) FM support getter of training prams

[ https://issues.apache.org/jira/browse/SPARK-30378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30378. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27044

[jira] [Assigned] (SPARK-30378) FM support getter of training prams

[ https://issues.apache.org/jira/browse/SPARK-30378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-30378: Assignee: Huaxin Gao > FM support getter of training prams >

[jira] [Resolved] (SPARK-30358) ML expose predictRaw and predictProbability

[ https://issues.apache.org/jira/browse/SPARK-30358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30358. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27015

[jira] [Comment Edited] (SPARK-30392) Documentation for the date_trunc function is incorrect

[ https://issues.apache.org/jira/browse/SPARK-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005924#comment-17005924 ] Ankit Raj Boudh edited comment on SPARK-30392 at 12/31/19 4:44 AM: ---

[jira] [Commented] (SPARK-30392) Documentation for the date_trunc function is incorrect

[ https://issues.apache.org/jira/browse/SPARK-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005924#comment-17005924 ] Ankit Raj Boudh commented on SPARK-30392: - [~avanname] , changes are already present in below

[jira] [Created] (SPARK-30394) Skip collecting stats in DetermineTableStats rule when hive table is convertible to datasource tables

liupengcheng created SPARK-30394: Summary: Skip collecting stats in DetermineTableStats rule when hive table is convertible to datasource tables Key: SPARK-30394 URL:

[jira] [Resolved] (SPARK-30379) Avoid OOM when using collection accumulator

[ https://issues.apache.org/jira/browse/SPARK-30379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30379. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27038

[jira] [Commented] (SPARK-30392) Documentation for the date_trunc function is incorrect

[ https://issues.apache.org/jira/browse/SPARK-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005901#comment-17005901 ] Ankit Raj Boudh commented on SPARK-30392: - I will raise PR for this. > Documentation for the

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Attachment: kinesisexceedreadlimit.png > Too much ProvisionedThroughputExceededException while recover

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Attachment: (was: kinesisexceedreadlimit.png) > Too much ProvisionedThroughputExceededException while

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Attachment: kinesisexceedreadlimit.png > Too much ProvisionedThroughputExceededException while recover

[jira] [Resolved] (SPARK-30215) Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex

[ https://issues.apache.org/jira/browse/SPARK-30215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang resolved SPARK-30215. --- Resolution: Not A Problem > Remove PrunedInMemoryFileIndex and merge its functionality into >

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Resolved] (SPARK-24970) Spark Kinesis streaming application fails to recover from streaming checkpoint due to ProvisionedThroughputExceededException

[ https://issues.apache.org/jira/browse/SPARK-24970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-24970. -- Resolution: Not A Problem > Spark Kinesis streaming application fails to recover from

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Description: I have a spark application which consume from Kinesis with 6 shards. Data was produced to

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Attachment: sparkuiwhilecheckpointrecoveryerror.png > Too much ProvisionedThroughputExceededException

[jira] [Updated] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen updated SPARK-30393: Attachment: kinesisusagewhilecheckpointrecoveryerror.png > Too much

[jira] [Created] (SPARK-30393) Too much ProvisionedThroughputExceededException while recover from checkpoint

Stephen created SPARK-30393: --- Summary: Too much ProvisionedThroughputExceededException while recover from checkpoint Key: SPARK-30393 URL: https://issues.apache.org/jira/browse/SPARK-30393 Project: Spark

[jira] [Commented] (SPARK-30132) Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses

[ https://issues.apache.org/jira/browse/SPARK-30132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005841#comment-17005841 ] Sean R. Owen commented on SPARK-30132: -- https://github.com/scala/bug/issues/11840 > Scala 2.13

[jira] [Commented] (SPARK-30225) "Stream is corrupted at" exception on reading disk-spilled data of a shuffle operation

[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005838#comment-17005838 ] Marcelo Masiero Vanzin commented on SPARK-30225: The changes in SPARK-23366 may not

[jira] [Assigned] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-18105: -- Assignee: Marcelo Masiero Vanzin (was: Davies Liu) > LZ4 failed to

[jira] [Assigned] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-18105: -- Assignee: (was: Marcelo Masiero Vanzin) > LZ4 failed to

[jira] [Updated] (SPARK-30392) Documentation for the date_trunc function is incorrect

[ https://issues.apache.org/jira/browse/SPARK-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashley updated SPARK-30392: --- Attachment: Date_trunc.PNG > Documentation for the date_trunc function is incorrect >

[jira] [Updated] (SPARK-30392) Documentation for the date_trunc function is incorrect

[ https://issues.apache.org/jira/browse/SPARK-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashley updated SPARK-30392: --- Description: The documentation for the date_trunc function includes a few sample SELECT statements to show

[jira] [Created] (SPARK-30392) Documentation for the date_trunc function is incorrect

Ashley created SPARK-30392: -- Summary: Documentation for the date_trunc function is incorrect Key: SPARK-30392 URL: https://issues.apache.org/jira/browse/SPARK-30392 Project: Spark Issue Type:

[jira] [Commented] (SPARK-28264) Revisiting Python / pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005500#comment-17005500 ] Maciej Szymkiewicz commented on SPARK-28264: Thanks [~hyukjin.kwon]. In general I think

[jira] [Created] (SPARK-30391) ExecutorAllocationManager requestTotalExecutors in removeExecutors may not be needed

Thomas Graves created SPARK-30391: - Summary: ExecutorAllocationManager requestTotalExecutors in removeExecutors may not be needed Key: SPARK-30391 URL: https://issues.apache.org/jira/browse/SPARK-30391

[jira] [Resolved] (SPARK-30339) Avoid to fail twice in function lookup

[ https://issues.apache.org/jira/browse/SPARK-30339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30339. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26994

[jira] [Assigned] (SPARK-30339) Avoid to fail twice in function lookup

[ https://issues.apache.org/jira/browse/SPARK-30339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30339: Assignee: Zhenhua Wang > Avoid to fail twice in function lookup >

[jira] [Resolved] (SPARK-30185) Implement Dataset.tail API

[ https://issues.apache.org/jira/browse/SPARK-30185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30185. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26809

[jira] [Created] (SPARK-30390) Avoid double caching in mllib.KMeans#runWithWeights.

Aman Omer created SPARK-30390: - Summary: Avoid double caching in mllib.KMeans#runWithWeights. Key: SPARK-30390 URL: https://issues.apache.org/jira/browse/SPARK-30390 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30387) Improve YarnClientSchedulerBackend log message

[ https://issues.apache.org/jira/browse/SPARK-30387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30387: - Priority: Trivial (was: Minor) This is too trivial for a JIRA > Improve

[jira] [Assigned] (SPARK-30361) Monitoring URL do not redact information about environment

[ https://issues.apache.org/jira/browse/SPARK-30361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30361: Assignee: Ajith S > Monitoring URL do not redact information about environment >

[jira] [Resolved] (SPARK-30361) Monitoring URL do not redact information about environment

[ https://issues.apache.org/jira/browse/SPARK-30361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30361. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27018

[jira] [Updated] (SPARK-30361) Monitoring URL do not redact information about environment

[ https://issues.apache.org/jira/browse/SPARK-30361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30361: - Priority: Minor (was: Critical) > Monitoring URL do not redact information about environment >

[jira] [Issue Comment Deleted] (SPARK-29776) rpad and lpad should return NULL when padstring parameter is empty

[ https://issues.apache.org/jira/browse/SPARK-29776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-29776: Comment: was deleted (was: [~hyukjin.kwon] , as per the discussion SPARK-29776 and

[jira] [Commented] (SPARK-30389) Add jar should not allow to add other format except jar file

[ https://issues.apache.org/jira/browse/SPARK-30389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005312#comment-17005312 ] Ankit Raj Boudh commented on SPARK-30389: - i will raise PR for this > Add jar should not allow

[jira] [Created] (SPARK-30389) Add jar should not allow to add other format except jar file

ABHISHEK KUMAR GUPTA created SPARK-30389: Summary: Add jar should not allow to add other format except jar file Key: SPARK-30389 URL: https://issues.apache.org/jira/browse/SPARK-30389

[jira] [Created] (SPARK-30388) mark all running map stages of finished job as finished

Xuesen Liang created SPARK-30388: Summary: mark all running map stages of finished job as finished Key: SPARK-30388 URL: https://issues.apache.org/jira/browse/SPARK-30388 Project: Spark

[jira] [Updated] (SPARK-30385) WebUI occasionally throw IOException on stop()

[ https://issues.apache.org/jira/browse/SPARK-30385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-30385: - Description: While using ./bin/spark-shell, recently, I occasionally see IOException when I try to quit:

[jira] [Updated] (SPARK-30385) WebUI occasionally throw IOException on stop()

[ https://issues.apache.org/jira/browse/SPARK-30385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-30385: - Description: While using ./bin/spark-shell, recently, I occasionally see IOException when I try to quit:

[jira] [Created] (SPARK-30387) Improve YarnClientSchedulerBackend log message

jobit mathew created SPARK-30387: Summary: Improve YarnClientSchedulerBackend log message Key: SPARK-30387 URL: https://issues.apache.org/jira/browse/SPARK-30387 Project: Spark Issue Type:

[jira] [Created] (SPARK-30386) Pushdown limit to outer join through projects

ulysses you created SPARK-30386: --- Summary: Pushdown limit to outer join through projects Key: SPARK-30386 URL: https://issues.apache.org/jira/browse/SPARK-30386 Project: Spark Issue Type:

[jira] [Created] (SPARK-30385) WebUI occasionally throw IOException on stop()

wuyi created SPARK-30385: Summary: WebUI occasionally throw IOException on stop() Key: SPARK-30385 URL: https://issues.apache.org/jira/browse/SPARK-30385 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-28264) Revisiting Python / pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005228#comment-17005228 ] Hyukjin Kwon commented on SPARK-28264: -- I came up with a new proposal. Please take a look guys if

[jira] [Updated] (SPARK-28264) Revisiting Python / pandas UDF

[ https://issues.apache.org/jira/browse/SPARK-28264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28264: - Description: In the past two years, the pandas UDFs are perhaps the most important changes to

[jira] [Assigned] (SPARK-30383) Remove meaning less tooltip from Executor Tab

[ https://issues.apache.org/jira/browse/SPARK-30383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30383: Assignee: Ankit Raj Boudh > Remove meaning less tooltip from Executor Tab >

[jira] [Resolved] (SPARK-30383) Remove meaning less tooltip from Executor Tab

[ https://issues.apache.org/jira/browse/SPARK-30383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30383. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27043

[jira] [Commented] (SPARK-30384) Needs to improve the Column name and tooltips for the Fair Scheduler Pool Table

[ https://issues.apache.org/jira/browse/SPARK-30384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005215#comment-17005215 ] Ankit Raj Boudh commented on SPARK-30384: - i will submit the PR. > Needs to improve the Column

[jira] [Updated] (SPARK-30384) Needs to improve the Column name and tooltips for the Fair Scheduler Pool Table

[ https://issues.apache.org/jira/browse/SPARK-30384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-30384: - Summary: Needs to improve the Column name and tooltips for the Fair Scheduler

[jira] [Created] (SPARK-30384) Needs to improve the Column name and tooltips for the Fair Scheduler Poos Table

ABHISHEK KUMAR GUPTA created SPARK-30384: Summary: Needs to improve the Column name and tooltips for the Fair Scheduler Poos Table Key: SPARK-30384 URL: https://issues.apache.org/jira/browse/SPARK-30384