[jira] [Assigned] (SPARK-23447) Cleanup codegen template for Literal

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23447: Assignee: Apache Spark > Cleanup codegen template for Literal > --

[jira] [Commented] (SPARK-23446) Explicitly check supported types in toPandas

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366686#comment-16366686 ] Apache Spark commented on SPARK-23446: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-23446) Explicitly check supported types in toPandas

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23446: Assignee: (was: Apache Spark) > Explicitly check supported types in toPandas > ---

[jira] [Assigned] (SPARK-23446) Explicitly check supported types in toPandas

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23446: Assignee: Apache Spark > Explicitly check supported types in toPandas > --

[jira] [Updated] (SPARK-23446) Explicitly check supported types in toPandas

2018-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23446: - Summary: Explicitly check supported types in toPandas (was: Explicitly specify supported types i

[jira] [Created] (SPARK-23447) Cleanup codegen template for Literal

2018-02-15 Thread Kris Mok (JIRA)
Kris Mok created SPARK-23447: Summary: Cleanup codegen template for Literal Key: SPARK-23447 URL: https://issues.apache.org/jira/browse/SPARK-23447 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-23446) Explicitly specify supported types in toPandas

2018-02-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-23446: Summary: Explicitly specify supported types in toPandas Key: SPARK-23446 URL: https://issues.apache.org/jira/browse/SPARK-23446 Project: Spark Issue Type: Su

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-15 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366547#comment-16366547 ] Maxim Gekk commented on SPARK-23410: [~sameerag] It is not blocker anymore. I unset t

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-15 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-23410: --- Priority: Major (was: Blocker) > Unable to read jsons in charset different from UTF-8 >

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-15 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366537#comment-16366537 ] Sameer Agarwal commented on SPARK-23410: [~maxgekk] [~smilegator] any ETA on this

[jira] [Assigned] (SPARK-23445) ColumnStat refactoring

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23445: Assignee: (was: Apache Spark) > ColumnStat refactoring > -- > >

[jira] [Commented] (SPARK-23445) ColumnStat refactoring

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366530#comment-16366530 ] Apache Spark commented on SPARK-23445: -- User 'juliuszsompolski' has created a pull r

[jira] [Assigned] (SPARK-23445) ColumnStat refactoring

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23445: Assignee: Apache Spark > ColumnStat refactoring > -- > >

[jira] [Created] (SPARK-23445) ColumnStat refactoring

2018-02-15 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-23445: - Summary: ColumnStat refactoring Key: SPARK-23445 URL: https://issues.apache.org/jira/browse/SPARK-23445 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-02-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366421#comment-16366421 ] Shixiong Zhu commented on SPARK-23433: -- cc [~irashid] > java.lang.IllegalStateExcep

[jira] [Commented] (SPARK-23433) java.lang.IllegalStateException: more than one active taskSet for stage

2018-02-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366417#comment-16366417 ] Shixiong Zhu commented on SPARK-23433: -- {code} 18/02/11 13:22:20 INFO TaskSetManager

[jira] [Commented] (SPARK-23368) Avoid unnecessary Exchange or Sort after projection

2018-02-15 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366374#comment-16366374 ] Maryann Xue commented on SPARK-23368: - [~cloud_fan], [~smilegator], Could you please

[jira] [Created] (SPARK-23444) would like to be able to cancel jobs cleanly

2018-02-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23444: --- Summary: would like to be able to cancel jobs cleanly Key: SPARK-23444 URL: https://issues.apache.org/jira/browse/SPARK-23444 Project: Spark Issue Type: Wish

[jira] [Commented] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-02-15 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366277#comment-16366277 ] Bago Amirbekian commented on SPARK-23265: - What's the status of this? Will this b

[jira] [Resolved] (SPARK-22913) Hive Partition Pruning, Fractional and Timestamp types

2018-02-15 Thread Ameen Tayyebi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ameen Tayyebi resolved SPARK-22913. --- Resolution: Won't Fix Resolving in favor of native Glue integration. These advanced predicate

[jira] [Created] (SPARK-23443) Spark with Glue as external catalog

2018-02-15 Thread Ameen Tayyebi (JIRA)
Ameen Tayyebi created SPARK-23443: - Summary: Spark with Glue as external catalog Key: SPARK-23443 URL: https://issues.apache.org/jira/browse/SPARK-23443 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366252#comment-16366252 ] Apache Spark commented on SPARK-23413: -- User 'squito' has created a pull request for

[jira] [Updated] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23413: - Affects Version/s: (was: 2.4.0) > Sorting tasks by Host / Executor ID on the Stage page does

[jira] [Commented] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366199#comment-16366199 ] Imran Rashid commented on SPARK-23413: -- This was fixed by https://github.com/apache/

[jira] [Resolved] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23413. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20623 [https://git

[jira] [Assigned] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23413: Assignee: Attila Zsolt Piros > Sorting tasks by Host / Executor ID on the Stage page does

[jira] [Updated] (SPARK-23173) from_json can produce nulls for fields which are marked as non-nullable

2018-02-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-23173: - Labels: release-notes (was: ) > from_json can produce nulls for fields which are marked

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366162#comment-16366162 ] Igor Berman edited comment on SPARK-23423 at 2/15/18 7:30 PM: -

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366162#comment-16366162 ] Igor Berman commented on SPARK-23423: - [~skonto] I'll at Sunday probably > Applicati

[jira] [Resolved] (SPARK-23377) Bucketizer with multiple columns persistence bug

2018-02-15 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-23377. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 Resolved in ma

[jira] [Resolved] (SPARK-23430) Cannot sort "Executor ID" or "Host" columns in the task table

2018-02-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23430. -- Resolution: Duplicate > Cannot sort "Executor ID" or "Host" columns in the task table > ---

[jira] [Updated] (SPARK-23442) Reading from partitioned and bucketed table uses only bucketSpec.numBuckets partitions in all cases

2018-02-15 Thread Pranav Rao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Rao updated SPARK-23442: --- Description: Through the DataFrameWriter[T] interface I have created a external HIVE table with 5000

[jira] [Updated] (SPARK-23442) Reading from partitioned and bucketed table uses only bucketSpec.numBuckets partitions in all cases

2018-02-15 Thread Pranav Rao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Rao updated SPARK-23442: --- Description: Through the DataFrameWriter[T] interface I have created a external HIVE table with 5000

[jira] [Updated] (SPARK-23442) Reading from partitioned and bucketed table uses only bucketSpec.numBuckets partitions in all cases

2018-02-15 Thread Pranav Rao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Rao updated SPARK-23442: --- Environment: (was: spark.sql("SET spark.default.parallelism=1000") {{spark.sql("set spa

[jira] [Created] (SPARK-23442) Reading from partitioned and bucketed table uses only bucketSpec.numBuckets partitions in all cases

2018-02-15 Thread Pranav Rao (JIRA)
Pranav Rao created SPARK-23442: -- Summary: Reading from partitioned and bucketed table uses only bucketSpec.numBuckets partitions in all cases Key: SPARK-23442 URL: https://issues.apache.org/jira/browse/SPARK-23442

[jira] [Assigned] (SPARK-23377) Bucketizer with multiple columns persistence bug

2018-02-15 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-23377: - Assignee: Liang-Chi Hsieh > Bucketizer with multiple columns persistence bug > -

[jira] [Commented] (SPARK-14540) Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner

2018-02-15 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365991#comment-16365991 ] Piotr Kołaczkowski commented on SPARK-14540: Any progress on this? Are you pl

[jira] [Assigned] (SPARK-23441) Remove interrupts from ContinuousExecution

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23441: Assignee: (was: Apache Spark) > Remove interrupts from ContinuousExecution > -

[jira] [Commented] (SPARK-23441) Remove interrupts from ContinuousExecution

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365984#comment-16365984 ] Apache Spark commented on SPARK-23441: -- User 'jose-torres' has created a pull reques

[jira] [Assigned] (SPARK-23441) Remove interrupts from ContinuousExecution

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23441: Assignee: Apache Spark > Remove interrupts from ContinuousExecution >

[jira] [Created] (SPARK-23441) Remove interrupts from ContinuousExecution

2018-02-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23441: --- Summary: Remove interrupts from ContinuousExecution Key: SPARK-23441 URL: https://issues.apache.org/jira/browse/SPARK-23441 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23440) Clean up StreamExecution interrupts

2018-02-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23440: --- Summary: Clean up StreamExecution interrupts Key: SPARK-23440 URL: https://issues.apache.org/jira/browse/SPARK-23440 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-23436) Incorrect Date column Inference in partition discovery

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23436: Assignee: Apache Spark > Incorrect Date column Inference in partition discovery >

[jira] [Assigned] (SPARK-23436) Incorrect Date column Inference in partition discovery

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23436: Assignee: (was: Apache Spark) > Incorrect Date column Inference in partition discovery

[jira] [Commented] (SPARK-23436) Incorrect Date column Inference in partition discovery

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365959#comment-16365959 ] Apache Spark commented on SPARK-23436: -- User 'mgaido91' has created a pull request f

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365931#comment-16365931 ] Kazuaki Ishizaki commented on SPARK-23415: -- I realized that an issue in this tes

[jira] [Comment Edited] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365931#comment-16365931 ] Kazuaki Ishizaki edited comment on SPARK-23415 at 2/15/18 5:02 PM:

[jira] [Commented] (SPARK-23437) [ML] Distributed Gaussian Process Regression for MLlib

2018-02-15 Thread Simon Dirmeier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365929#comment-16365929 ] Simon Dirmeier commented on SPARK-23437: Great suggestion. If there is a way to c

[jira] [Assigned] (SPARK-23438) DStreams could lose blocks with WAL enabled when driver crashes

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23438: Assignee: (was: Apache Spark) > DStreams could lose blocks with WAL enabled when drive

[jira] [Commented] (SPARK-23438) DStreams could lose blocks with WAL enabled when driver crashes

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365928#comment-16365928 ] Apache Spark commented on SPARK-23438: -- User 'gaborgsomogyi' has created a pull requ

[jira] [Assigned] (SPARK-23438) DStreams could lose blocks with WAL enabled when driver crashes

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23438: Assignee: Apache Spark > DStreams could lose blocks with WAL enabled when driver crashes >

[jira] [Assigned] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23390: Assignee: Wenchen Fan (was: Apache Spark) > Flaky Test Suite: FileBasedDataSourceSuite in

[jira] [Commented] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365927#comment-16365927 ] Apache Spark commented on SPARK-23390: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Assigned] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23390: Assignee: Apache Spark (was: Wenchen Fan) > Flaky Test Suite: FileBasedDataSourceSuite in

[jira] [Updated] (SPARK-23340) Upgrade Apache ORC to 1.4.3

2018-02-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23340: Priority: Major (was: Blocker) > Upgrade Apache ORC to 1.4.3 > --- > >

[jira] [Resolved] (SPARK-23340) Upgrade Apache ORC to 1.4.3

2018-02-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23340. - Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0 > Upgrade Apache ORC to 1.

[jira] [Resolved] (SPARK-23426) Use `hive` ORC impl and disable PPD for Spark 2.3.0

2018-02-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23426. - Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0 > Use `hive` ORC impl and

[jira] [Commented] (SPARK-17025) Cannot persist PySpark ML Pipeline model that includes custom Transformer

2018-02-15 Thread Lucas Partridge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365904#comment-16365904 ] Lucas Partridge commented on SPARK-17025: - What's the up-to-date status for this

[jira] [Updated] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

2018-02-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Description: We're seeing multiple failures in {{FileBasedDataSourceSuite}} in {{spark-branch-

[jira] [Reopened] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

2018-02-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-23390: --- I'm reopening this issue due to Parquet leakage is detected. - https://amplab.cs.berkeley.edu/j

[jira] [Created] (SPARK-23439) Ambiguous reference when selecting column inside StructType with same name that outer colum

2018-02-15 Thread Alejandro Trujillo Caballero (JIRA)
Alejandro Trujillo Caballero created SPARK-23439: Summary: Ambiguous reference when selecting column inside StructType with same name that outer colum Key: SPARK-23439 URL: https://issues.apache.or

[jira] [Commented] (SPARK-23438) DStreams could lose blocks with WAL enabled when driver crashes

2018-02-15 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365861#comment-16365861 ] Gabor Somogyi commented on SPARK-23438: --- I'm working on that. > DStreams could los

[jira] [Created] (SPARK-23438) DStreams could lose blocks with WAL enabled when driver crashes

2018-02-15 Thread Gabor Somogyi (JIRA)
Gabor Somogyi created SPARK-23438: - Summary: DStreams could lose blocks with WAL enabled when driver crashes Key: SPARK-23438 URL: https://issues.apache.org/jira/browse/SPARK-23438 Project: Spark

[jira] [Updated] (SPARK-23437) [ML] Distributed Gaussian Process Regression for MLlib

2018-02-15 Thread Valeriy Avanesov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valeriy Avanesov updated SPARK-23437: - Summary: [ML] Distributed Gaussian Process Regression for MLlib (was: Distributed Gaussi

[jira] [Created] (SPARK-23437) Distributed Gaussian Process Regression for MLlib

2018-02-15 Thread Valeriy Avanesov (JIRA)
Valeriy Avanesov created SPARK-23437: Summary: Distributed Gaussian Process Regression for MLlib Key: SPARK-23437 URL: https://issues.apache.org/jira/browse/SPARK-23437 Project: Spark Iss

[jira] [Commented] (SPARK-23436) Incorrect Date column Inference in partition discovery

2018-02-15 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365780#comment-16365780 ] Marco Gaido commented on SPARK-23436: - Thanks for reporting this. This affects also c

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365468#comment-16365468 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/15/18 3:37 PM: -

[jira] [Created] (SPARK-23436) Incorrect Date column Inference in partition discovery

2018-02-15 Thread Apoorva Sareen (JIRA)
Apoorva Sareen created SPARK-23436: -- Summary: Incorrect Date column Inference in partition discovery Key: SPARK-23436 URL: https://issues.apache.org/jira/browse/SPARK-23436 Project: Spark Is

[jira] [Commented] (SPARK-21302) history server WebUI show HTTP ERROR 500

2018-02-15 Thread bharath kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365691#comment-16365691 ] bharath kumar commented on SPARK-21302: --- >From what i have noticed that Jobs are pr

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365666#comment-16365666 ] Hyukjin Kwon commented on SPARK-23410: -- It's reverted in https://github.com/apache/s

[jira] [Resolved] (SPARK-23405) The task will hang up when a small table left semi join a big table

2018-02-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23405. --- Resolution: Invalid > The task will hang up when a small table left semi join a big table > -

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365549#comment-16365549 ] Sean Owen commented on SPARK-23402: --- This is all just the master branch of github.com/a

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-15 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365495#comment-16365495 ] Steve Loughran commented on SPARK-23308: I'm going to recommend this is closed as

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Labels: Mesos dynamic_allocation (was: ) > Application declines any offers when killed+active exec

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365468#comment-16365468 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/15/18 12:31 PM:

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365468#comment-16365468 ] Stavros Kontopoulos commented on SPARK-23423: - The task updates delivery is a

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365438#comment-16365438 ] Igor Berman commented on SPARK-23423: - [~skonto] do you think it's could be connected

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365436#comment-16365436 ] Igor Berman commented on SPARK-23423: - [~skonto], yes this is correct. Besides TASK_R

[jira] [Assigned] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23422: -- Assignee: Gabor Somogyi > YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASS

[jira] [Resolved] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23422. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pu

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-15 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365295#comment-16365295 ] Stavros Kontopoulos commented on SPARK-23423: - [~igor.berman] From the log I

[jira] [Created] (SPARK-23435) R tests should support latest testthat

2018-02-15 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-23435: Summary: R tests should support latest testthat Key: SPARK-23435 URL: https://issues.apache.org/jira/browse/SPARK-23435 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-23359) Adds an alias 'names' of 'fieldNames' in Scala's StructType

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23359. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20545 [https://githu

[jira] [Comment Edited] (SPARK-22817) Use fixed testthat version for SparkR tests in AppVeyor

2018-02-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365263#comment-16365263 ] Felix Cheung edited comment on SPARK-22817 at 2/15/18 9:13 AM:

[jira] [Assigned] (SPARK-23359) Adds an alias 'names' of 'fieldNames' in Scala's StructType

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23359: --- Assignee: Hyukjin Kwon > Adds an alias 'names' of 'fieldNames' in Scala's StructType > -

[jira] [Resolved] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23366. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20555 [https://githu

[jira] [Assigned] (SPARK-23366) Improve hot reading path in ReadAheadInputStream

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23366: --- Assignee: Juliusz Sompolski > Improve hot reading path in ReadAheadInputStream > ---

[jira] [Commented] (SPARK-22817) Use fixed testthat version for SparkR tests in AppVeyor

2018-02-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365263#comment-16365263 ] Felix Cheung commented on SPARK-22817: -- I should have caught this - we need to fix t

[jira] [Assigned] (SPARK-23329) Update the function descriptions with the arguments and returned values of the trigonometric functions

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23329: Assignee: Apache Spark > Update the function descriptions with the arguments and returned

[jira] [Assigned] (SPARK-23329) Update the function descriptions with the arguments and returned values of the trigonometric functions

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23329: Assignee: (was: Apache Spark) > Update the function descriptions with the arguments an

[jira] [Commented] (SPARK-23329) Update the function descriptions with the arguments and returned values of the trigonometric functions

2018-02-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365261#comment-16365261 ] Apache Spark commented on SPARK-23329: -- User 'misutoth' has created a pull request f

[jira] [Resolved] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23416. - Resolution: Fixed Fix Version/s: 2.3.1 Issue resolved by pull request 20605 [https://githu

[jira] [Resolved] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

2018-02-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23419. - Resolution: Fixed Fix Version/s: 2.3.1 Issue resolved by pull request 20605 [https://githu

[jira] [Updated] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23413: --- Priority: Blocker (was: Major) > Sorting tasks by Host / Executor ID on the Stage page does