[jira] [Created] (SPARK-27670) Add High available for Spark Hive thrift server.

2019-05-10 Thread jiaan.geng (JIRA)
jiaan.geng created SPARK-27670: -- Summary: Add High available for Spark Hive thrift server. Key: SPARK-27670 URL: https://issues.apache.org/jira/browse/SPARK-27670 Project: Spark Issue Type: New

[jira] [Assigned] (SPARK-27670) Add High available for Spark Hive thrift server.

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27670: Assignee: Apache Spark > Add High available for Spark Hive thrift server. > -

[jira] [Assigned] (SPARK-27670) Add High available for Spark Hive thrift server.

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27670: Assignee: (was: Apache Spark) > Add High available for Spark Hive thrift server. > --

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: houragg_filter.csv > In Spark2.4 Structured Streaming:The executor storage memory increa

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837078#comment-16837078 ] tommy duan commented on SPARK-27648: Hi [~gsomogyi] & [~kabhwan] I have filtered th

[jira] [Commented] (SPARK-27600) Unable to start Spark Hive Thrift Server when multiple hive server server share the same metastore

2019-05-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837081#comment-16837081 ] Hyukjin Kwon commented on SPARK-27600: -- Spark is upgrading Hive to 2.3.5. Will be f

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: image-2019-05-10-17-18-25-051.png > In Spark2.4 Structured Streaming:The executor storag

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837078#comment-16837078 ] tommy duan edited comment on SPARK-27648 at 5/10/19 9:41 AM: -

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837078#comment-16837078 ] tommy duan edited comment on SPARK-27648 at 5/10/19 9:40 AM: -

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837078#comment-16837078 ] tommy duan edited comment on SPARK-27648 at 5/10/19 9:43 AM: -

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837078#comment-16837078 ] tommy duan edited comment on SPARK-27648 at 5/10/19 9:49 AM: -

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-10 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: (was: image-2019-05-10-17-18-25-051.png) > In Spark2.4 Structured Streaming:The exec

[jira] [Updated] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-10 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-27664: Description: In short, This issue(i.e. degraded performance ) surfaces when the number of

[jira] [Updated] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-10 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-27664: Description: In short, This issue(i.e. degraded performance ) surfaces when the number of

[jira] [Created] (SPARK-27671) Analysis exception thrown when casting from a nested null in a struct

2019-05-10 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-27671: --- Summary: Analysis exception thrown when casting from a nested null in a struct Key: SPARK-27671 URL: https://issues.apache.org/jira/browse/SPARK-27671 Project:

[jira] [Created] (SPARK-27672) Add since info to string expressions

2019-05-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27672: Summary: Add since info to string expressions Key: SPARK-27672 URL: https://issues.apache.org/jira/browse/SPARK-27672 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-27671) Analysis exception thrown when casting from a nested null in a struct

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27671: Assignee: (was: Apache Spark) > Analysis exception thrown when casting from a nested

[jira] [Commented] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837182#comment-16837182 ] Apache Spark commented on SPARK-27664: -- User 'ScrapCodes' has created a pull reques

[jira] [Assigned] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27664: Assignee: (was: Apache Spark) > Performance issue with FileStatusCache, while reading

[jira] [Assigned] (SPARK-27671) Analysis exception thrown when casting from a nested null in a struct

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27671: Assignee: Apache Spark > Analysis exception thrown when casting from a nested null in a s

[jira] [Assigned] (SPARK-27664) Performance issue with FileStatusCache, while reading from object stores.

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27664: Assignee: Apache Spark > Performance issue with FileStatusCache, while reading from objec

[jira] [Assigned] (SPARK-27672) Add since info to string expressions

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27672: Assignee: Hyukjin Kwon (was: Apache Spark) > Add since info to string expressions >

[jira] [Assigned] (SPARK-27672) Add since info to string expressions

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27672: Assignee: Apache Spark (was: Hyukjin Kwon) > Add since info to string expressions >

[jira] [Assigned] (SPARK-27673) Add since info to random. regex, null expressions

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27673: Assignee: Apache Spark (was: Hyukjin Kwon) > Add since info to random. regex, null expre

[jira] [Assigned] (SPARK-27673) Add since info to random. regex, null expressions

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27673: Assignee: Hyukjin Kwon (was: Apache Spark) > Add since info to random. regex, null expre

[jira] [Created] (SPARK-27673) Add since info to random. regex, null expressions

2019-05-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27673: Summary: Add since info to random. regex, null expressions Key: SPARK-27673 URL: https://issues.apache.org/jira/browse/SPARK-27673 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837234#comment-16837234 ] Stavros Kontopoulos commented on SPARK-25075: - [~srowen] since Scala 2.11 ha

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837260#comment-16837260 ] Guillaume Massé commented on SPARK-25075: - Supporting both Scala 2.12 and Scala

[jira] [Commented] (SPARK-18484) case class datasets - ability to specify decimal precision and scale

2019-05-10 Thread Bill Schneider (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837269#comment-16837269 ] Bill Schneider commented on SPARK-18484: I agree with [~bonazzaf].  Where this i

[jira] [Created] (SPARK-27674) the hint should not be dropped after cache lookup

2019-05-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27674: --- Summary: the hint should not be dropped after cache lookup Key: SPARK-27674 URL: https://issues.apache.org/jira/browse/SPARK-27674 Project: Spark Issue Type: B

[jira] [Assigned] (SPARK-27674) the hint should not be dropped after cache lookup

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27674: Assignee: Apache Spark (was: Wenchen Fan) > the hint should not be dropped after cache l

[jira] [Assigned] (SPARK-27674) the hint should not be dropped after cache lookup

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27674: Assignee: Wenchen Fan (was: Apache Spark) > the hint should not be dropped after cache l

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837299#comment-16837299 ] Sean Owen commented on SPARK-25075: --- Given that 2.13 isn't out yet and most dependenci

[jira] [Assigned] (SPARK-27675) do not use MutableColumnarRow in ColumnarBatch

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27675: Assignee: Wenchen Fan (was: Apache Spark) > do not use MutableColumnarRow in ColumnarBat

[jira] [Assigned] (SPARK-27675) do not use MutableColumnarRow in ColumnarBatch

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27675: Assignee: Apache Spark (was: Wenchen Fan) > do not use MutableColumnarRow in ColumnarBat

[jira] [Resolved] (SPARK-27650) sepate the row iterator functionality from ColumnarBatch

2019-05-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27650. - Resolution: Won't Do > sepate the row iterator functionality from ColumnarBatch > --

[jira] [Created] (SPARK-27675) do not use MutableColumnarRow in ColumnarBatch

2019-05-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27675: --- Summary: do not use MutableColumnarRow in ColumnarBatch Key: SPARK-27675 URL: https://issues.apache.org/jira/browse/SPARK-27675 Project: Spark Issue Type: Impr

[jira] [Resolved] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

2019-05-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27625. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24564 [https://gith

[jira] [Assigned] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

2019-05-10 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27625: --- Assignee: Marco Gaido > ScalaReflection.serializerFor fails for annotated types > -

[jira] [Updated] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

2019-05-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26412: -- Summary: Allow Pandas UDF to take an iterator of pd.DataFrames (was: Allow Pandas UDF to take

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2019-05-10 Thread Graton M Gathright (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837346#comment-16837346 ] Graton M Gathright commented on SPARK-14948: Looking forward to a fix for th

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

2019-05-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837353#comment-16837353 ] Xiangrui Meng commented on SPARK-26412: --- [~WeichenXu123] I updated the description

[jira] [Updated] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

2019-05-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26412: -- Description: Pandas UDF is the ideal connection between PySpark and DL model inference worklo

[jira] [Updated] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

2019-05-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26412: -- Description: Pandas UDF is the ideal connection between PySpark and DL model inference worklo

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837403#comment-16837403 ] Stavros Kontopoulos commented on SPARK-25075: - Yes will take some time one o

[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837403#comment-16837403 ] Stavros Kontopoulos edited comment on SPARK-25075 at 5/10/19 3:53 PM:

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stefan Zeiger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837417#comment-16837417 ] Stefan Zeiger commented on SPARK-25075: --- Returning a Seq is not a problem but if y

[jira] [Closed] (SPARK-24980) add support for pandas/arrow etc for python2.7 and pypy builds

2019-05-10 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp closed SPARK-24980. --- > add support for pandas/arrow etc for python2.7 and pypy builds > -

[jira] [Resolved] (SPARK-24980) add support for pandas/arrow etc for python2.7 and pypy builds

2019-05-10 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-24980. - Resolution: Fixed > add support for pandas/arrow etc for python2.7 and pypy builds > ---

[jira] [Commented] (SPARK-21367) R older version of Roxygen2 on Jenkins

2019-05-10 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837440#comment-16837440 ] shane knapp commented on SPARK-21367: - after deciding to clear out my JIRA queue thi

[jira] [Resolved] (SPARK-21367) R older version of Roxygen2 on Jenkins

2019-05-10 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp resolved SPARK-21367. - Resolution: Fixed > R older version of Roxygen2 on Jenkins > ---

[jira] [Assigned] (SPARK-26632) Separate Thread Configurations of Driver and Executor

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26632: -- Assignee: jiafu zhang > Separate Thread Configurations of Driver and Executor > -

[jira] [Resolved] (SPARK-26632) Separate Thread Configurations of Driver and Executor

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26632. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23560 [https:

[jira] [Assigned] (SPARK-27347) Fix supervised driver retry logic when agent crashes/restarts

2019-05-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27347: - Assignee: Dongjoon Hyun > Fix supervised driver retry logic when agent crashes/restarts

[jira] [Resolved] (SPARK-27347) Fix supervised driver retry logic when agent crashes/restarts

2019-05-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27347. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.4 2.3

[jira] [Assigned] (SPARK-27347) Fix supervised driver retry logic when agent crashes/restarts

2019-05-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27347: - Assignee: (was: Dongjoon Hyun) > Fix supervised driver retry logic when agent crash

[jira] [Assigned] (SPARK-27347) Fix supervised driver retry logic when agent crashes/restarts

2019-05-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27347: - Assignee: Sam Tran > Fix supervised driver retry logic when agent crashes/restarts > --

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837516#comment-16837516 ] Sean Owen commented on SPARK-25075: --- OK, so we can possibly defer this by waiting unti

[jira] [Issue Comment Deleted] (SPARK-27660) Allow PySpark toLocalIterator to pre-fetch data

2019-05-10 Thread Holden Karau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-27660: - Comment: was deleted (was: Which is the this an issue duplicate off?) > Allow PySpark toLocalIt

[jira] [Commented] (SPARK-27660) Allow PySpark toLocalIterator to pre-fetch data

2019-05-10 Thread Holden Karau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837529#comment-16837529 ] Holden Karau commented on SPARK-27660: -- Which is the this an issue duplicate off?

[jira] [Commented] (SPARK-27660) Allow PySpark toLocalIterator to pre-fetch data

2019-05-10 Thread Holden Karau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837530#comment-16837530 ] Holden Karau commented on SPARK-27660: -- Related to https://issues.apache.org/jira/b

[jira] [Created] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27676: -- Summary: InMemoryFileIndex should hard-fail on missing files instead of logging and continuing Key: SPARK-27676 URL: https://issues.apache.org/jira/browse/SPARK-27676 Pro

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions ar

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions ar

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions ar

[jira] [Commented] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837552#comment-16837552 ] Michael Armbrust commented on SPARK-27676: -- I tend to agree that all cases wher

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837579#comment-16837579 ] Stavros Kontopoulos commented on SPARK-25075: - Scala team added some more in

[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837579#comment-16837579 ] Stavros Kontopoulos edited comment on SPARK-25075 at 5/10/19 8:24 PM:

[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13

2019-05-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837579#comment-16837579 ] Stavros Kontopoulos edited comment on SPARK-25075 at 5/10/19 8:24 PM:

[jira] [Created] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

2019-05-10 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-27677: Summary: Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation Key: SPARK-27677 URL: https://issues.apache.org/jira/browse/SPARK-27677

[jira] [Commented] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

2019-05-10 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837629#comment-16837629 ] Imran Rashid commented on SPARK-27677: -- I'm breaking this out of SPARK-25888. [~at

[jira] [Commented] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

2019-05-10 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837632#comment-16837632 ] Imran Rashid commented on SPARK-25888: -- There is a lot of stuff listed here, so I t

[jira] [Created] (SPARK-27678) Support Knox user impersonation in UI

2019-05-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-27678: -- Summary: Support Knox user impersonation in UI Key: SPARK-27678 URL: https://issues.apache.org/jira/browse/SPARK-27678 Project: Spark Issue Type: New Fea

[jira] [Assigned] (SPARK-27678) Support Knox user impersonation in UI

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27678: Assignee: (was: Apache Spark) > Support Knox user impersonation in UI > -

[jira] [Assigned] (SPARK-27678) Support Knox user impersonation in UI

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27678: Assignee: Apache Spark > Support Knox user impersonation in UI >

[jira] [Created] (SPARK-27679) Improve queries with LIKE expression

2019-05-10 Thread Achuth Narayan Rajagopal (JIRA)
Achuth Narayan Rajagopal created SPARK-27679: Summary: Improve queries with LIKE expression Key: SPARK-27679 URL: https://issues.apache.org/jira/browse/SPARK-27679 Project: Spark

[jira] [Assigned] (SPARK-27679) Improve queries with LIKE expression

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27679: Assignee: Apache Spark > Improve queries with LIKE expression > -

[jira] [Assigned] (SPARK-27679) Improve queries with LIKE expression

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27679: Assignee: (was: Apache Spark) > Improve queries with LIKE expression > --

[jira] [Created] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27681: - Summary: Use scala.collection.Seq explicitly instead of scala.Seq alias Key: SPARK-27681 URL: https://issues.apache.org/jira/browse/SPARK-27681 Project: Spark Iss

[jira] [Created] (SPARK-27682) Avoid use of Scala collection classes that are removed in 2.13

2019-05-10 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27682: - Summary: Avoid use of Scala collection classes that are removed in 2.13 Key: SPARK-27682 URL: https://issues.apache.org/jira/browse/SPARK-27682 Project: Spark Iss

[jira] [Created] (SPARK-27680) Remove usage of Traversable

2019-05-10 Thread Sean Owen (JIRA)
Sean Owen created SPARK-27680: - Summary: Remove usage of Traversable Key: SPARK-27680 URL: https://issues.apache.org/jira/browse/SPARK-27680 Project: Spark Issue Type: Sub-task Componen

[jira] [Assigned] (SPARK-27680) Remove usage of Traversable

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27680: Assignee: Sean Owen (was: Apache Spark) > Remove usage of Traversable >

[jira] [Assigned] (SPARK-27680) Remove usage of Traversable

2019-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27680: Assignee: Apache Spark (was: Sean Owen) > Remove usage of Traversable >

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837749#comment-16837749 ] Marcelo Vanzin commented on SPARK-27681: OOC, what happens if we do nothing? As

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837752#comment-16837752 ] Marcelo Vanzin commented on SPARK-27681: Also the change you're proposing likely

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837751#comment-16837751 ] Sean Owen commented on SPARK-27681: --- To my understanding, we don't need to do this unt

[jira] [Commented] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837753#comment-16837753 ] Marcelo Vanzin commented on SPARK-27681: bq. we don't need to do this until we s

[jira] [Comment Edited] (SPARK-27681) Use scala.collection.Seq explicitly instead of scala.Seq alias

2019-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837752#comment-16837752 ] Marcelo Vanzin edited comment on SPARK-27681 at 5/11/19 4:28 AM: -

[jira] [Commented] (SPARK-23191) Workers registration failes in case of network drop

2019-05-10 Thread Neeraj Gupta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837756#comment-16837756 ] Neeraj Gupta commented on SPARK-23191: -- Hi [~Ngone51], Yes in my case multiple dri