[jira] [Updated] (SPARK-25708) HAVING without GROUP BY means global aggregate

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25708: Description: According to the SQL standard, when a query contains `HAVING`, it indicates an aggre

[jira] [Commented] (SPARK-26182) Cost increases when optimizing scalaUDF

2018-11-29 Thread Jiayi Liao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704347#comment-16704347 ] Jiayi Liao commented on SPARK-26182: [~fanweiwen] We're doing some optimizations on

[jira] [Commented] (SPARK-26206) Spark structured streaming with kafka integration fails in update mode

2018-11-29 Thread indraneel r (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704277#comment-16704277 ] indraneel r commented on SPARK-26206: - Will check on spark-shell but not sure if it

[jira] [Created] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
Chen Lin created SPARK-26228: Summary: OOM issue encountered when computing Gramian matrix Key: SPARK-26228 URL: https://issues.apache.org/jira/browse/SPARK-26228 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed on matri

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: {quote}/**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed o

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  * @note This cannot be computed on matri

[jira] [Updated] (SPARK-26228) OOM issue encountered when computing Gramian matrix

2018-11-29 Thread Chen Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Lin updated SPARK-26228: - Description: /**  * Computes the Gramian matrix `A^T A`.  *  *@note This cannot be computed on matrice

[jira] [Resolved] (SPARK-26188) Spark 2.4.0 Partitioning behavior breaks backwards compatibility

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26188. - Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by pull re

[jira] [Assigned] (SPARK-26188) Spark 2.4.0 Partitioning behavior breaks backwards compatibility

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26188: --- Assignee: Gengliang Wang > Spark 2.4.0 Partitioning behavior breaks backwards compatibility

[jira] [Commented] (SPARK-26227) from_[csv|json] should accept schema_of_[csv|json] in R API

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704230#comment-16704230 ] Apache Spark commented on SPARK-26227: -- User 'HyukjinKwon' has created a pull reque

[jira] [Resolved] (SPARK-26060) Track SparkConf entries and make SET command reject such entries.

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26060. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23031 [https://gith

[jira] [Assigned] (SPARK-26060) Track SparkConf entries and make SET command reject such entries.

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26060: --- Assignee: Takuya Ueshin > Track SparkConf entries and make SET command reject such entries.

[jira] [Commented] (SPARK-26227) from_[csv|json] should accept schema_of_[csv|json] in R API

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704231#comment-16704231 ] Apache Spark commented on SPARK-26227: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-26227) from_[csv|json] should accept schema_of_[csv|json] in R API

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26227: Assignee: Apache Spark > from_[csv|json] should accept schema_of_[csv|json] in R API > --

[jira] [Assigned] (SPARK-26227) from_[csv|json] should accept schema_of_[csv|json] in R API

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26227: Assignee: (was: Apache Spark) > from_[csv|json] should accept schema_of_[csv|json] in

[jira] [Created] (SPARK-26227) from_[csv|json] should accept schema_of_[csv|json] in R API

2018-11-29 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26227: Summary: from_[csv|json] should accept schema_of_[csv|json] in R API Key: SPARK-26227 URL: https://issues.apache.org/jira/browse/SPARK-26227 Project: Spark

[jira] [Resolved] (SPARK-25446) Add schema_of_json() and schema_of_csv() to R

2018-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25446. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22939 [https://gi

[jira] [Resolved] (SPARK-25501) Kafka delegation token support

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25501. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22598 [https:

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25501: -- Assignee: Gabor Somogyi > Kafka delegation token support > --

[jira] [Commented] (SPARK-26200) Column values are incorrectly transposed when a field in a PySpark Row requires serialization

2018-11-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704092#comment-16704092 ] Bryan Cutler commented on SPARK-26200: -- I think this is a duplicate of https://iss

[jira] [Comment Edited] (SPARK-26200) Column values are incorrectly transposed when a field in a PySpark Row requires serialization

2018-11-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704092#comment-16704092 ] Bryan Cutler edited comment on SPARK-26200 at 11/30/18 12:56 AM: -

[jira] [Assigned] (SPARK-25977) Parsing decimals from CSV using locale

2018-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25977: Assignee: Maxim Gekk > Parsing decimals from CSV using locale > -

[jira] [Resolved] (SPARK-25977) Parsing decimals from CSV using locale

2018-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25977. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22979 [https://gi

[jira] [Commented] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704010#comment-16704010 ] Apache Spark commented on SPARK-26226: -- User 'rxin' has created a pull request for

[jira] [Assigned] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26226: Assignee: Apache Spark (was: Reynold Xin) > Update query tracker to report timeline for

[jira] [Commented] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704011#comment-16704011 ] Apache Spark commented on SPARK-26226: -- User 'rxin' has created a pull request for

[jira] [Assigned] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26226: Assignee: Reynold Xin (was: Apache Spark) > Update query tracker to report timeline for

[jira] [Assigned] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reassigned SPARK-26226: --- Assignee: Reynold Xin > Update query tracker to report timeline for phases, rather than dur

[jira] [Created] (SPARK-26226) Update query tracker to report timeline for phases, rather than duration

2018-11-29 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-26226: --- Summary: Update query tracker to report timeline for phases, rather than duration Key: SPARK-26226 URL: https://issues.apache.org/jira/browse/SPARK-26226 Project: Spark

[jira] [Updated] (SPARK-26221) Improve Spark SQL instrumentation and metrics

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26221: Description: This is an umbrella ticket for various small improvements for better metrics and ins

[jira] [Updated] (SPARK-26221) Improve Spark SQL instrumentation and metrics

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26221: Description: This is an umbrella ticket for various small improvements for better metrics and ins

[jira] [Updated] (SPARK-26129) Instrumentation for query planning time

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26129: Issue Type: Sub-task (was: New Feature) Parent: SPARK-26221 > Instrumentation for query p

[jira] [Updated] (SPARK-26221) Improve Spark SQL instrumentation and metrics

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26221: Description: This is an umbrella ticket for various small improvements for better metrics and ins

[jira] [Updated] (SPARK-26223) Scan: track metastore operation time

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26223: Description: The Scan node should report how much time it spent in metastore operations. Similar

[jira] [Created] (SPARK-26225) Scan: track decoding time for row-based data sources

2018-11-29 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-26225: --- Summary: Scan: track decoding time for row-based data sources Key: SPARK-26225 URL: https://issues.apache.org/jira/browse/SPARK-26225 Project: Spark Issue Type

[jira] [Commented] (SPARK-26209) Allow for dataframe bucketization without Hive

2018-11-29 Thread Sam hendley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703824#comment-16703824 ] Sam hendley commented on SPARK-26209: - Seems like dataframe needs a side-channel for

[jira] [Created] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

2018-11-29 Thread Dorjee Tsering (JIRA)
Dorjee Tsering created SPARK-26224: -- Summary: Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe. Key: SPARK-26224 URL: https://issues.apache.org/jira/browse/

[jira] [Assigned] (SPARK-25905) BlockManager should expose getRemoteManagedBuffer to avoid creating bytebuffers

2018-11-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-25905: Assignee: Wing Yew Poon > BlockManager should expose getRemoteManagedBuffer to avoid crea

[jira] [Resolved] (SPARK-25905) BlockManager should expose getRemoteManagedBuffer to avoid creating bytebuffers

2018-11-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-25905. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23058 [https://gi

[jira] [Created] (SPARK-26223) Scan: track metastore operation time

2018-11-29 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-26223: --- Summary: Scan: track metastore operation time Key: SPARK-26223 URL: https://issues.apache.org/jira/browse/SPARK-26223 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-26222) Scan: track file listing time

2018-11-29 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-26222: --- Summary: Scan: track file listing time Key: SPARK-26222 URL: https://issues.apache.org/jira/browse/SPARK-26222 Project: Spark Issue Type: Sub-task Co

[jira] [Updated] (SPARK-26221) Improve Spark SQL instrumentation and metrics

2018-11-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-26221: Description: This is an umbrella ticket for various small improvements for better metrics and inst

[jira] [Created] (SPARK-26221) Improve Spark SQL instrumentation and metrics

2018-11-29 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-26221: --- Summary: Improve Spark SQL instrumentation and metrics Key: SPARK-26221 URL: https://issues.apache.org/jira/browse/SPARK-26221 Project: Spark Issue Type: Umbre

[jira] [Assigned] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26219: Assignee: (was: Apache Spark) > Executor summary is not getting updated for failure j

[jira] [Commented] (SPARK-26100) [History server ]Jobs table and Aggregate metrics table are showing lesser number of tasks

2018-11-29 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703693#comment-16703693 ] shahid commented on SPARK-26100: Sorry, I have wrongly linked the JIRA > [History serve

[jira] [Updated] (SPARK-26166) CrossValidator.fit() bug,training and validation dataset may overlap

2018-11-29 Thread Xinyong Tian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyong Tian updated SPARK-26166: - Description: In the code pyspark.ml.tuning.CrossValidator.fit(), after adding random column df

[jira] [Updated] (SPARK-26166) CrossValidator.fit() bug,training and validation dataset may overlap

2018-11-29 Thread Xinyong Tian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyong Tian updated SPARK-26166: - Description: In the code pyspark.ml.tuning.CrossValidator.fit(), after adding random column df

[jira] [Commented] (SPARK-26177) Automated formatting for Scala code

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703700#comment-16703700 ] Apache Spark commented on SPARK-26177: -- User 'koeninger' has created a pull request

[jira] [Commented] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703694#comment-16703694 ] Apache Spark commented on SPARK-26219: -- User 'shahidki31' has created a pull reques

[jira] [Commented] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703696#comment-16703696 ] Apache Spark commented on SPARK-26219: -- User 'shahidki31' has created a pull reques

[jira] [Assigned] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26219: Assignee: Apache Spark > Executor summary is not getting updated for failure jobs in hist

[jira] [Assigned] (SPARK-26158) Enhance the accuracy of covariance in RowMatrix for DenseVector

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26158: - Assignee: Liang Li > Enhance the accuracy of covariance in RowMatrix for DenseVector >

[jira] [Resolved] (SPARK-26158) Enhance the accuracy of covariance in RowMatrix for DenseVector

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26158. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23126 [https://github.c

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Thincrs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703667#comment-16703667 ] Thincrs commented on SPARK-26183: - testing thincrs > ConcurrentModificationException wh

[jira] [Updated] (SPARK-26220) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2018-11-29 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26220: -- Description: There was a kafka-clients version update lately and I've seen a test failure lik

[jira] [Commented] (SPARK-26220) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2018-11-29 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703644#comment-16703644 ] Gabor Somogyi commented on SPARK-26220: --- [~dongjoon] Don't know whether it's relat

[jira] [Updated] (SPARK-26220) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2018-11-29 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26220: -- Issue Type: Bug (was: New Feature) > Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStr

[jira] [Created] (SPARK-26220) Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite

2018-11-29 Thread Gabor Somogyi (JIRA)
Gabor Somogyi created SPARK-26220: - Summary: Flaky Test: org.apache.spark.sql.kafka010.KafkaSourceStressSuite Key: SPARK-26220 URL: https://issues.apache.org/jira/browse/SPARK-26220 Project: Spark

[jira] [Commented] (SPARK-26100) [History server ]Jobs table and Aggregate metrics table are showing lesser number of tasks

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703631#comment-16703631 ] Apache Spark commented on SPARK-26100: -- User 'shahidki31' has created a pull reques

[jira] [Updated] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-26219: --- Attachment: Screenshot from 2018-11-29 22-13-44.png > Executor summary is not getting updated for failure jo

[jira] [Updated] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-26219: --- Description: Test step to reproduce: {code:java} bin/spark-shell --master yarn --conf spark.executor.instan

[jira] [Updated] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-26219: --- Attachment: Screenshot from 2018-11-29 22-13-34.png > Executor summary is not getting updated for failure jo

[jira] [Created] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread shahid (JIRA)
shahid created SPARK-26219: -- Summary: Executor summary is not getting updated for failure jobs in history server UI Key: SPARK-26219 URL: https://issues.apache.org/jira/browse/SPARK-26219 Project: Spark

[jira] [Commented] (SPARK-26219) Executor summary is not getting updated for failure jobs in history server UI

2018-11-29 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703619#comment-16703619 ] shahid commented on SPARK-26219: I will raise a PR > Executor summary is not getting up

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: testing thincrs) > ConcurrentModificationException when using Spark collectionAccu

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: .../) > ConcurrentModificationException when using Spark collectionAccumulator > -

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: ..///) > ConcurrentModificationException when using Spark collectionAccumulator >

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: .) > ConcurrentModificationException when using Spark collectionAccumulator >

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: .) > ConcurrentModificationException when using Spark collectionAccumulator >

[jira] [Issue Comment Deleted] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jairo updated SPARK-26183: -- Comment: was deleted (was: .) > ConcurrentModificationException when using Spark collectionAccumulator >

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703596#comment-16703596 ] Jairo commented on SPARK-26183: --- testing thincrs > ConcurrentModificationException when u

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703593#comment-16703593 ] Jairo commented on SPARK-26183: --- ../// > ConcurrentModificationException when using Spark

[jira] [Assigned] (SPARK-26015) Include a USER directive in project provided Spark Dockerfiles

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26015: -- Assignee: Rob Vesse > Include a USER directive in project provided Spark Dockerfiles

[jira] [Resolved] (SPARK-26015) Include a USER directive in project provided Spark Dockerfiles

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26015. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23017 [https:

[jira] [Resolved] (SPARK-24498) Add JDK compiler for runtime codegen

2018-11-29 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-24498. -- Resolution: Won't Do > Add JDK compiler for runtime codegen >

[jira] [Assigned] (SPARK-26184) Last updated time is not getting updated in the History Server UI

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26184: -- Assignee: shahid > Last updated time is not getting updated in the History Server UI

[jira] [Resolved] (SPARK-26184) Last updated time is not getting updated in the History Server UI

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26184. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by p

[jira] [Resolved] (SPARK-26186) In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26186. Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by p

[jira] [Assigned] (SPARK-26186) In progress applications with last updated time is lesser than the cleaning interval are getting removed during cleaning logs

2018-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26186: -- Assignee: shahid > In progress applications with last updated time is lesser than the

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-11-29 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703556#comment-16703556 ] Kazuaki Ishizaki commented on SPARK-24498: -- I see. Let us close for now. We may

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703534#comment-16703534 ] Jairo commented on SPARK-26183: --- .../ > ConcurrentModificationException when using Spark

[jira] [Updated] (SPARK-26081) Do not write empty files by text datasources

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26081: -- Docs Text: In Spark 3.0, when empty partitions are written to a CSV, JSON or text data source, they no

[jira] [Resolved] (SPARK-26081) Do not write empty files by text datasources

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26081. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23052 [https://github.c

[jira] [Assigned] (SPARK-26081) Do not write empty files by text datasources

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26081: - Assignee: Maxim Gekk > Do not write empty files by text datasources > -

[jira] [Updated] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26180: -- Priority: Minor (was: Major) > Add a withCreateTempDir function to the SparkCore test case >

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2018-11-29 Thread Jairo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703444#comment-16703444 ] Jairo commented on SPARK-26183: --- . > ConcurrentModificationException when using Spark col

[jira] [Commented] (SPARK-26209) Allow for dataframe bucketization without Hive

2018-11-29 Thread Walt Elder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703324#comment-16703324 ] Walt Elder commented on SPARK-26209: I've attempted to get on the dev@ boards about

[jira] [Resolved] (SPARK-26211) Fix InSet for binary, and struct and array with null.

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26211. - Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 3.0.0

[jira] [Resolved] (SPARK-26177) Automated formatting for Scala code

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26177. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23148 [https://github.c

[jira] [Assigned] (SPARK-26177) Automated formatting for Scala code

2018-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26177: - Assignee: Cody Koeninger > Automated formatting for Scala code > --

[jira] [Commented] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703266#comment-16703266 ] Apache Spark commented on SPARK-26218: -- User 'mgaido91' has created a pull request

[jira] [Created] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26218: --- Summary: Throw exception on overflow for integers Key: SPARK-26218 URL: https://issues.apache.org/jira/browse/SPARK-26218 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-26215) define reserved keywords after SQL standard

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-26215: Issue Type: Sub-task (was: Improvement) Parent: SPARK-26217 > define reserved keywords af

[jira] [Commented] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703270#comment-16703270 ] Apache Spark commented on SPARK-26218: -- User 'mgaido91' has created a pull request

[jira] [Assigned] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26218: Assignee: (was: Apache Spark) > Throw exception on overflow for integers > --

[jira] [Assigned] (SPARK-26218) Throw exception on overflow for integers

2018-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26218: Assignee: Apache Spark > Throw exception on overflow for integers > -

[jira] [Updated] (SPARK-23179) Support option to throw exception if overflow occurs

2018-11-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23179: Issue Type: Sub-task (was: Improvement) Parent: SPARK-26217 > Support option to throw exc

[jira] [Created] (SPARK-26217) Compliance to SQL standard (SQL:2011)

2018-11-29 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-26217: --- Summary: Compliance to SQL standard (SQL:2011) Key: SPARK-26217 URL: https://issues.apache.org/jira/browse/SPARK-26217 Project: Spark Issue Type: Umbrella

[jira] [Resolved] (SPARK-26163) Parsing decimals from JSON using locale

2018-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26163. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23132 [https://gith

  1   2   >