[jira] [Assigned] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24120: Assignee: (was: Apache Spark) > Show `Jobs` page when `jobId` is missing > ---

[jira] [Commented] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460721#comment-16460721 ] Apache Spark commented on SPARK-24120: -- User 'jongyoul' has created a pull request f

[jira] [Assigned] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24120: Assignee: Apache Spark > Show `Jobs` page when `jobId` is missing > --

[jira] [Commented] (SPARK-23775) Flaky test: DataFrameRangeSuite

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460802#comment-16460802 ] Apache Spark commented on SPARK-23775: -- User 'gaborgsomogyi' has created a pull requ

[jira] [Created] (SPARK-24145) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-02 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-24145: -- Summary: spark.ml parity for sequential pattern mining - PrefixSpan: Python API Key: SPARK-24145 URL: https://issues.apache.org/jira/browse/SPARK-24145 Project: Spark

[jira] [Created] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-02 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-24146: -- Summary: spark.ml parity for sequential pattern mining - PrefixSpan: Python API Key: SPARK-24146 URL: https://issues.apache.org/jira/browse/SPARK-24146 Project: Spark

[jira] [Commented] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-02 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460804#comment-16460804 ] Weichen Xu commented on SPARK-24146: I will create PR soon. :) > spark.ml parity for

[jira] [Updated] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-02 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-24146: --- Issue Type: Sub-task (was: New Feature) Parent: SPARK-14501 > spark.ml parity for sequential

[jira] [Created] (SPARK-24147) .count() reports wrong size of dataframe when filtering dataframe on

2018-05-02 Thread Rich Smith (JIRA)
Rich Smith created SPARK-24147: -- Summary: .count() reports wrong size of dataframe when filtering dataframe on Key: SPARK-24147 URL: https://issues.apache.org/jira/browse/SPARK-24147 Project: Spark

[jira] [Updated] (SPARK-24147) .count() reports wrong size of dataframe when filtering dataframe on corrupt record field

2018-05-02 Thread Rich Smith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rich Smith updated SPARK-24147: --- Summary: .count() reports wrong size of dataframe when filtering dataframe on corrupt record field (

[jira] [Updated] (SPARK-24147) .count() reports wrong size of dataframe when filtering dataframe on corrupt record field

2018-05-02 Thread Rich Smith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rich Smith updated SPARK-24147: --- Description: Spark reports the wrong size of dataframe using .count() after filtering on a corruptFi

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

2018-05-02 Thread Joachim Hereth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460870#comment-16460870 ] Joachim Hereth commented on SPARK-24067: It would be great if this fix could go i

[jira] [Commented] (SPARK-23180) RFormulaModel should have labels member

2018-05-02 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460967#comment-16460967 ] Teng Peng commented on SPARK-23180: --- Can you give me an example for 1. the current work

[jira] [Commented] (SPARK-22943) OneHotEncoder supports manual specification of categorySizes

2018-05-02 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461006#comment-16461006 ] Teng Peng commented on SPARK-22943: --- This issue looks quiet interesting, but can you be

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461024#comment-16461024 ] Matt Cheah commented on SPARK-24135: I think we should not count these towards job fa

[jira] [Created] (SPARK-24148) Adding Ability to Specify SQL Type of Empty Arrays

2018-05-02 Thread Marek Novotny (JIRA)
Marek Novotny created SPARK-24148: - Summary: Adding Ability to Specify SQL Type of Empty Arrays Key: SPARK-24148 URL: https://issues.apache.org/jira/browse/SPARK-24148 Project: Spark Issue Ty

[jira] [Commented] (SPARK-24147) .count() reports wrong size of dataframe when filtering dataframe on corrupt record field

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461062#comment-16461062 ] Hyukjin Kwon commented on SPARK-24147: -- I think it's a duplicate of SPARK-21610. >

[jira] [Assigned] (SPARK-24148) Adding Ability to Specify SQL Type of Empty Arrays

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24148: Assignee: Apache Spark > Adding Ability to Specify SQL Type of Empty Arrays >

[jira] [Assigned] (SPARK-24148) Adding Ability to Specify SQL Type of Empty Arrays

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24148: Assignee: (was: Apache Spark) > Adding Ability to Specify SQL Type of Empty Arrays > -

[jira] [Commented] (SPARK-24148) Adding Ability to Specify SQL Type of Empty Arrays

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461064#comment-16461064 ] Apache Spark commented on SPARK-24148: -- User 'mn-mikke' has created a pull request f

[jira] [Created] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2018-05-02 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-24149: --- Summary: Automatic namespaces discovery in HDFS federation Key: SPARK-24149 URL: https://issues.apache.org/jira/browse/SPARK-24149 Project: Spark Issue Type: I

[jira] [Assigned] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24149: Assignee: (was: Apache Spark) > Automatic namespaces discovery in HDFS federation > --

[jira] [Assigned] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24149: Assignee: Apache Spark > Automatic namespaces discovery in HDFS federation > -

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461078#comment-16461078 ] Apache Spark commented on SPARK-24149: -- User 'mgaido91' has created a pull request f

[jira] [Resolved] (SPARK-24107) ChunkedByteBuffer.writeFully method has not reset the limit value

2018-05-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24107. - Resolution: Fixed Assignee: wangjinhai Fix Version/s: 2.3.1 > ChunkedByteBuffer.w

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461154#comment-16461154 ] Erik Erlandson commented on SPARK-24135: IIRC the dynamic allocation heuristic wa

[jira] [Created] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
William Montaz created SPARK-24150: -- Summary: Race condition in FsHistoryProvider Key: SPARK-24150 URL: https://issues.apache.org/jira/browse/SPARK-24150 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition between the method checkLogs and cleanLogs. cleanL

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461188#comment-16461188 ] Matt Cheah commented on SPARK-24135: > Restarting seems like it would eventually be l

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461188#comment-16461188 ] Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM: {

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460047#comment-16460047 ] Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM: {

[jira] [Created] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread James Thompson (JIRA)
James Thompson created SPARK-24151: -- Summary: CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled Key: SPARK-24151 URL: https://issues.apache.org/jira/browse/SPARK-2

[jira] [Commented] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461237#comment-16461237 ] Apache Spark commented on SPARK-24151: -- User 'jamesthomp' has created a pull request

[jira] [Assigned] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24151: Assignee: Apache Spark > CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column na

[jira] [Assigned] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24151: Assignee: (was: Apache Spark) > CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved a

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

2018-05-02 Thread Sam Garrett (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461250#comment-16461250 ] Sam Garrett commented on SPARK-22918: - +1 same issue > sbt test (spark - local) fail

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of replayExecu

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of replayExecu

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of replayExecu

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Priority: Major (was: Minor) > Race condition in FsHistoryProvider > ---

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of replayExecu

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of replayExecu

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461324#comment-16461324 ] Erik Erlandson commented on SPARK-24135: > In the case of the executor failing to

[jira] [Created] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-24152: - Summary: Flaky Test: SparkR Key: SPARK-24152 URL: https://issues.apache.org/jira/browse/SPARK-24152 Project: Spark Issue Type: Bug Components: Sp

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder fails with the following SparkR error with unknown reason. The followi

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461328#comment-16461328 ] Dongjoon Hyun commented on SPARK-24152: --- cc [~shivaram], [~felixcheung], [~yanbolia

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder fails with the following SparkR error with unknown reason. The followi

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder and master branch test fails with the following SparkR error with unkn

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder and master branch test fails with the following SparkR error with unkn

[jira] [Created] (SPARK-24153) Flaky Test: DirectKafkaStreamSuite

2018-05-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-24153: - Summary: Flaky Test: DirectKafkaStreamSuite Key: SPARK-24153 URL: https://issues.apache.org/jira/browse/SPARK-24153 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23489) Flaky Test: HiveExternalCatalogVersionsSuite

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23489: -- Description: I saw this error in an unrelated PR. It seems to me a bad configuration in the Je

[jira] [Resolved] (SPARK-24013) ApproximatePercentile grinds to a halt on sorted input.

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24013. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > ApproximatePercentile grin

[jira] [Updated] (SPARK-23971) Should not leak Spark sessions across test suites

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23971: Component/s: Tests > Should not leak Spark sessions across test suites > --

[jira] [Updated] (SPARK-23971) Should not leak Spark sessions across test suites

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23971: Fix Version/s: 2.3.1 > Should not leak Spark sessions across test suites >

[jira] [Commented] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2018-05-02 Thread Evan McClain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461502#comment-16461502 ] Evan McClain commented on SPARK-4502: - The workaround I've been using is to explicitly

[jira] [Created] (SPARK-24154) AccumulatorV2 loses type information during serialization

2018-05-02 Thread Sergey Zhemzhitsky (JIRA)
Sergey Zhemzhitsky created SPARK-24154: -- Summary: AccumulatorV2 loses type information during serialization Key: SPARK-24154 URL: https://issues.apache.org/jira/browse/SPARK-24154 Project: Spark

[jira] [Updated] (SPARK-24154) AccumulatorV2 loses type information during serialization

2018-05-02 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-24154: --- Description: AccumulatorV2 loses type information during serialization. It happens [

[jira] [Resolved] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24133. - Resolution: Fixed Assignee: Ala Luszczak Fix Version/s: 2.4.0 > Reading Parquet files con

[jira] [Assigned] (SPARK-24097) Instruments improvements - RandomForest and GradientBoostedTree

2018-05-02 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-24097: - Assignee: Weichen Xu > Instruments improvements - RandomForest and GradientBoost

[jira] [Updated] (SPARK-24097) Instruments improvements - RandomForest and GradientBoostedTree

2018-05-02 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-24097: -- Shepherd: Joseph K. Bradley > Instruments improvements - RandomForest and GradientBoost

[jira] [Resolved] (SPARK-24123) Fix a flaky test `DateTimeUtilsSuite.monthsBetween`

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24123. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > Fix a flaky test `DateTime

[jira] [Resolved] (SPARK-23923) High-order function: cardinality(x) → bigint

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23923. - Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.4.0 > High-order function:

[jira] [Resolved] (SPARK-18791) Stream-Stream Joins

2018-05-02 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-18791. --- Resolution: Done Fix Version/s: 2.3.0 > Stream-Stream Joins > --- > >

[jira] [Created] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Lu Wang (JIRA)
Lu Wang created SPARK-24155: --- Summary: Instrument improvement for clustering Key: SPARK-24155 URL: https://issues.apache.org/jira/browse/SPARK-24155 Project: Spark Issue Type: Sub-task Co

[jira] [Assigned] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24155: Assignee: (was: Apache Spark) > Instrument improvement for clustering > --

[jira] [Assigned] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24155: Assignee: Apache Spark > Instrument improvement for clustering > -

[jira] [Commented] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461640#comment-16461640 ] Apache Spark commented on SPARK-24155: -- User 'ludatabricks' has created a pull reque

[jira] [Created] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24156: - Summary: Enable no-data micro batches for more eager streaming state clean up Key: SPARK-24156 URL: https://issues.apache.org/jira/browse/SPARK-24156 Project: Spar

[jira] [Created] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24157: - Summary: Enable no-data micro batches for streaming aggregation and deduplication Key: SPARK-24157 URL: https://issues.apache.org/jira/browse/SPARK-24157 Project: S

[jira] [Assigned] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-02 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24158: - Assignee: Tathagata Das > Enable no-data micro batches for streaming joins > ---

[jira] [Created] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24158: - Summary: Enable no-data micro batches for streaming joins Key: SPARK-24158 URL: https://issues.apache.org/jira/browse/SPARK-24158 Project: Spark Issue Type

[jira] [Created] (SPARK-24159) Enable no-data micro batches for streaming mapGroupswithState

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24159: - Summary: Enable no-data micro batches for streaming mapGroupswithState Key: SPARK-24159 URL: https://issues.apache.org/jira/browse/SPARK-24159 Project: Spark

[jira] [Updated] (SPARK-24132) Instrumentation improvement for classification

2018-05-02 Thread Lu Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lu Wang updated SPARK-24132: Summary: Instrumentation improvement for classification (was: Instruments improvement for classification)

[jira] [Updated] (SPARK-24155) Instrumentation improvement for clustering

2018-05-02 Thread Lu Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lu Wang updated SPARK-24155: Summary: Instrumentation improvement for clustering (was: Instrument improvement for clustering) > Instru

[jira] [Created] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-24160: -- Summary: ShuffleBlockFetcherIterator should fail if it receives zero-size blocks Key: SPARK-24160 URL: https://issues.apache.org/jira/browse/SPARK-24160 Project: Spark

[jira] [Assigned] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24160: Assignee: Apache Spark (was: Josh Rosen) > ShuffleBlockFetcherIterator should fail if it

[jira] [Commented] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461675#comment-16461675 ] Apache Spark commented on SPARK-24160: -- User 'JoshRosen' has created a pull request

[jira] [Assigned] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24160: Assignee: Josh Rosen (was: Apache Spark) > ShuffleBlockFetcherIterator should fail if it

[jira] [Assigned] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24157: Assignee: Tathagata Das (was: Apache Spark) > Enable no-data micro batches for streaming

[jira] [Commented] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461680#comment-16461680 ] Apache Spark commented on SPARK-24157: -- User 'tdas' has created a pull request for t

[jira] [Assigned] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24157: Assignee: Apache Spark (was: Tathagata Das) > Enable no-data micro batches for streaming

[jira] [Commented] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461713#comment-16461713 ] Jungtaek Lim commented on SPARK-24161: -- I have a working patch. Will raise a PR soon

[jira] [Created] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-24161: Summary: Enable debug package feature on structured streaming Key: SPARK-24161 URL: https://issues.apache.org/jira/browse/SPARK-24161 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-24111) Add TPCDS v2.7 (latest) queries in TPCDSQueryBenchmark

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24111. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.4.0 > Add TPCDS v2.7 (lates

[jira] [Created] (SPARK-24162) Support aliased literal values for Pivot "IN" clause

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24162: --- Summary: Support aliased literal values for Pivot "IN" clause Key: SPARK-24162 URL: https://issues.apache.org/jira/browse/SPARK-24162 Project: Spark Issue Type

[jira] [Created] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24163: --- Summary: Support "ANY" or sub-query for Pivot "IN" clause Key: SPARK-24163 URL: https://issues.apache.org/jira/browse/SPARK-24163 Project: Spark Issue Type: Im

[jira] [Created] (SPARK-24164) Support column list as the pivot column in Pivot

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24164: --- Summary: Support column list as the pivot column in Pivot Key: SPARK-24164 URL: https://issues.apache.org/jira/browse/SPARK-24164 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461752#comment-16461752 ] Apache Spark commented on SPARK-23429: -- User 'edwinalu' has created a pull request f

[jira] [Created] (SPARK-24165) UDF within when().otherwise() raises NullPointerException

2018-05-02 Thread Jingxuan Wang (JIRA)
Jingxuan Wang created SPARK-24165: - Summary: UDF within when().otherwise() raises NullPointerException Key: SPARK-24165 URL: https://issues.apache.org/jira/browse/SPARK-24165 Project: Spark I

[jira] [Assigned] (SPARK-22812) Failing cran-check on master

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22812: Assignee: Liang-Chi Hsieh > Failing cran-check on master > -

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461798#comment-16461798 ] Hyukjin Kwon commented on SPARK-24152: -- cc [~viirya] too > Flaky Test: SparkR > ---

[jira] [Assigned] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24110: --- Assignee: Saisai Shao > Avoid calling UGI loginUserFromKeytab in ThriftServer >

[jira] [Resolved] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24110. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21178 [https://githu

[jira] [Commented] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461858#comment-16461858 ] Apache Spark commented on SPARK-24161: -- User 'HeartSaVioR' has created a pull reques

[jira] [Assigned] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24161: Assignee: (was: Apache Spark) > Enable debug package feature on structured streaming >

[jira] [Assigned] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24161: Assignee: Apache Spark > Enable debug package feature on structured streaming > --

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461867#comment-16461867 ] Liang-Chi Hsieh commented on SPARK-24152: - Thanks [~hyukjin.kwon] for pinging me.

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461920#comment-16461920 ] Shivaram Venkataraman commented on SPARK-24152: --- Unfortunately I dont have

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461929#comment-16461929 ] Hyukjin Kwon commented on SPARK-24152: -- >From Liang-Chi's comment and given previous

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461929#comment-16461929 ] Hyukjin Kwon edited comment on SPARK-24152 at 5/3/18 4:39 AM: -

  1   2   >