[jira] [Assigned] (SPARK-23622) Flaky Test: HiveClientSuites

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23622: Assignee: Apache Spark > Flaky Test: HiveClientSuites > > >

[jira] [Commented] (SPARK-23622) Flaky Test: HiveClientSuites

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628317#comment-16628317 ] Apache Spark commented on SPARK-23622: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-23622) Flaky Test: HiveClientSuites

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23622: Assignee: (was: Apache Spark) > Flaky Test: HiveClientSuites > --

[jira] [Assigned] (SPARK-25541) CaseInsensitiveMap should be serializable after '-' or 'filterKeys'

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25541: Assignee: (was: Apache Spark) > CaseInsensitiveMap should be serializable after '-' o

[jira] [Commented] (SPARK-25541) CaseInsensitiveMap should be serializable after '-' or 'filterKeys'

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628313#comment-16628313 ] Apache Spark commented on SPARK-25541: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-25541) CaseInsensitiveMap should be serializable after '-' or 'filterKeys'

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25541: Assignee: Apache Spark > CaseInsensitiveMap should be serializable after '-' or 'filterKe

[jira] [Commented] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628311#comment-16628311 ] Apache Spark commented on SPARK-25540: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25540: Assignee: (was: Apache Spark) > Make HiveContext in PySpark behave as the same as Sca

[jira] [Assigned] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25540: Assignee: Apache Spark > Make HiveContext in PySpark behave as the same as Scala. > -

[jira] [Commented] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628309#comment-16628309 ] Apache Spark commented on SPARK-25540: -- User 'ueshin' has created a pull request fo

[jira] [Created] (SPARK-25542) Flaky test: OpenHashMapSuite

2018-09-25 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25542: - Summary: Flaky test: OpenHashMapSuite Key: SPARK-25542 URL: https://issues.apache.org/jira/browse/SPARK-25542 Project: Spark Issue Type: Bug Comp

[jira] [Created] (SPARK-25541) CaseInsensitiveMap should be serializable after '-' or 'filterKeys'

2018-09-25 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-25541: -- Summary: CaseInsensitiveMap should be serializable after '-' or 'filterKeys' Key: SPARK-25541 URL: https://issues.apache.org/jira/browse/SPARK-25541 Project: Spar

[jira] [Updated] (SPARK-25425) Extra options must overwrite sessions options

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25425: -- Fix Version/s: 2.3.3 > Extra options must overwrite sessions options > ---

[jira] [Commented] (SPARK-25539) Update lz4-java to get speed improvement

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628300#comment-16628300 ] Apache Spark commented on SPARK-25539: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-25539) Update lz4-java to get speed improvement

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25539: Assignee: Apache Spark > Update lz4-java to get speed improvement > -

[jira] [Assigned] (SPARK-25539) Update lz4-java to get speed improvement

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25539: Assignee: (was: Apache Spark) > Update lz4-java to get speed improvement > --

[jira] [Created] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

2018-09-25 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-25540: - Summary: Make HiveContext in PySpark behave as the same as Scala. Key: SPARK-25540 URL: https://issues.apache.org/jira/browse/SPARK-25540 Project: Spark Is

[jira] [Created] (SPARK-25539) Update lz4-java to get speed improvement

2018-09-25 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-25539: --- Summary: Update lz4-java to get speed improvement Key: SPARK-25539 URL: https://issues.apache.org/jira/browse/SPARK-25539 Project: Spark Issue Type: Improvemen

[jira] [Resolved] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25534. --- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22548 [https://

[jira] [Created] (SPARK-25538) incorrect row counts after distinct()

2018-09-25 Thread Steven Rand (JIRA)
Steven Rand created SPARK-25538: --- Summary: incorrect row counts after distinct() Key: SPARK-25538 URL: https://issues.apache.org/jira/browse/SPARK-25538 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25537) spark.pyspark.driver.python when set in code doesnt work

2018-09-25 Thread Venkat Sambath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkat Sambath updated SPARK-25537: --- Description: spark.pyspark.driver.python, spark.pyspark.python when set in code doesnt get p

[jira] [Commented] (SPARK-25536) executorSource.METRIC read wrong record in Executor.scala Line444

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628267#comment-16628267 ] shahid commented on SPARK-25536: I will raise a pr > executorSource.METRIC read wrong r

[jira] [Created] (SPARK-25537) spark.pyspark.driver.python when set in code doesnt work

2018-09-25 Thread Venkat Sambath (JIRA)
Venkat Sambath created SPARK-25537: -- Summary: spark.pyspark.driver.python when set in code doesnt work Key: SPARK-25537 URL: https://issues.apache.org/jira/browse/SPARK-25537 Project: Spark

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628231#comment-16628231 ] Wenchen Fan commented on SPARK-25378: - [~mengxr] what do you think? This is not a re

[jira] [Created] (SPARK-25536) executorSource.METRIC read wrong record in Executor.scala Line444

2018-09-25 Thread ZhuoerXu (JIRA)
ZhuoerXu created SPARK-25536: Summary: executorSource.METRIC read wrong record in Executor.scala Line444 Key: SPARK-25536 URL: https://issues.apache.org/jira/browse/SPARK-25536 Project: Spark Is

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628225#comment-16628225 ] Liang-Chi Hsieh commented on SPARK-25378: - Don't we have any decision on this ye

[jira] [Updated] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-25 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai updated SPARK-25523: -- Description: public static void test2() throws Exception{ String ckUrlPrefix="jdbc:clickhouse://"; S

[jira] [Updated] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-25 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huanghuai updated SPARK-25523: -- Description: public static void test2() throws Exception{ String ckUrlPrefix="jdbc:clickhouse://"; S

[jira] [Commented] (SPARK-23839) consider bucket join in cost-based JoinReorder rule

2018-09-25 Thread Xiaoju Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628184#comment-16628184 ] Xiaoju Wu commented on SPARK-23839: --- [~smilegator] Is there any plan on the cost-based

[jira] [Updated] (SPARK-25517) Spark DataFrame option inferSchema="true", dataFormat=MM/dd/yyyy, fails to detect date type from the csv file while reading

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25517: -- Environment: (was: Spark 2.3.0) > Spark DataFrame option inferSchema="true", dataFormat=MM

[jira] [Resolved] (SPARK-25517) Spark DataFrame option inferSchema="true", dataFormat=MM/dd/yyyy, fails to detect date type from the csv file while reading

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25517. --- Resolution: Duplicate According to the comments on the PR, I'll close this as `Duplicate` fo

[jira] [Closed] (SPARK-25518) Spark kafka delegation token supported

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25518. - > Spark kafka delegation token supported > --- > >

[jira] [Closed] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25526. - > SparkSQL over REST API not JDBC > --- > > Key: SPARK-2

[jira] [Updated] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25526: -- Component/s: (was: Spark Submit) (was: Spark Core)

[jira] [Resolved] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25526. --- Resolution: Won't Fix > SparkSQL over REST API not JDBC > --- >

[jira] [Commented] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628165#comment-16628165 ] Dongjoon Hyun commented on SPARK-25526: --- In addition to that, recently LIVY-494 st

[jira] [Commented] (SPARK-25526) SparkSQL over REST API not JDBC

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628161#comment-16628161 ] Dongjoon Hyun commented on SPARK-25526: --- Could you try Apache Livy? Apache Livy su

[jira] [Closed] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25511. - > Map with "null" key not working in spark 2.3 > > >

[jira] [Updated] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25511: -- Component/s: (was: Optimizer) SQL > Map with "null" key not working in sp

[jira] [Resolved] (SPARK-25511) Map with "null" key not working in spark 2.3

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25511. --- Resolution: Invalid Hi, [~ravi_b_shankar]. Historically, it's designed and documented like

[jira] [Comment Edited] (SPARK-23985) predicate push down doesn't work with simple compound partition spec

2018-09-25 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625876#comment-16625876 ] Yuming Wang edited comment on SPARK-23985 at 9/26/18 1:54 AM:

[jira] [Resolved] (SPARK-25514) Generating pretty JSON by to_json

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25514. -- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22534 [https://gi

[jira] [Assigned] (SPARK-25514) Generating pretty JSON by to_json

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25514: Assignee: Maxim Gekk > Generating pretty JSON by to_json > --

[jira] [Resolved] (SPARK-21291) R bucketBy partitionBy API

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21291. -- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22537 [https://gi

[jira] [Assigned] (SPARK-21291) R bucketBy partitionBy API

2018-09-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-21291: Assignee: Huaxin Gao > R bucketBy partitionBy API > -- > >

[jira] [Issue Comment Deleted] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Comment: was deleted (was: User 'dongjoon-hyun' has created a pull request for this issue: htt

[jira] [Updated] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Summary: Make `SQLHelper` trait (was: Make `SupportWithSQLConf` trait) > Make `SQLHelper` tra

[jira] [Updated] (SPARK-25534) Make `SQLHelper` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 7 `withTempPath` and 6 `withSQLConf` functions. This PR aim

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628120#comment-16628120 ] Apache Spark commented on SPARK-25501: -- User 'merlintang' has created a pull reques

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628119#comment-16628119 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] if you like, you can propose a

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25501: Assignee: (was: Apache Spark) > Kafka delegation token support >

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628117#comment-16628117 ] Apache Spark commented on SPARK-25501: -- User 'merlintang' has created a pull reques

[jira] [Assigned] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25501: Assignee: Apache Spark > Kafka delegation token support > --

[jira] [Resolved] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25422. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22546 [https://gith

[jira] [Assigned] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25422: --- Assignee: Imran Rashid > flaky test: org.apache.spark.DistributedSuite.caching on disk, rep

[jira] [Commented] (SPARK-23839) consider bucket join in cost-based JoinReorder rule

2018-09-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628080#comment-16628080 ] Xiao Li commented on SPARK-23839: - To implement CBO in planner, we need a major change i

[jira] [Commented] (SPARK-25535) Work around bad error checking in commons-crypto

2018-09-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628078#comment-16628078 ] Marcelo Vanzin commented on SPARK-25535: I filed CRYPTO-141 for the commons-cryp

[jira] [Created] (SPARK-25535) Work around bad error checking in commons-crypto

2018-09-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-25535: -- Summary: Work around bad error checking in commons-crypto Key: SPARK-25535 URL: https://issues.apache.org/jira/browse/SPARK-25535 Project: Spark Issue Ty

[jira] [Updated] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Description: Test steps:  1) bin/spark-shell {code:java} sc.parallelize(1 to 5, 5).collect() sc.parallelize

[jira] [Updated] (SPARK-25102) Write Spark version information to Parquet file footers

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25102: -- Component/s: (was: Spark Core) SQL > Write Spark version information to P

[jira] [Updated] (SPARK-25102) Write Spark version information to Parquet file footers

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25102: -- Priority: Minor (was: Major) > Write Spark version information to Parquet file footers >

[jira] [Commented] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627979#comment-16627979 ] Apache Spark commented on SPARK-25533: -- User 'shahidki31' has created a pull reques

[jira] [Assigned] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25533: Assignee: (was: Apache Spark) > Inconsistent message for Completed Jobs in the JobUI

[jira] [Assigned] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25533: Assignee: Apache Spark > Inconsistent message for Completed Jobs in the JobUI, when ther

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627974#comment-16627974 ] Apache Spark commented on SPARK-25534: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25534: Assignee: Dongjoon Hyun (was: Apache Spark) > Make `SupportWithSQLConf` trait >

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25534: Assignee: Apache Spark (was: Dongjoon Hyun) > Make `SupportWithSQLConf` trait >

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627973#comment-16627973 ] Apache Spark commented on SPARK-25534: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove duplica

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove duplica

[jira] [Updated] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25534: -- Description: Currently, Spark has 6 `withSQLConf` function. This issue aims to remove duplica

[jira] [Commented] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627961#comment-16627961 ] Dongjoon Hyun commented on SPARK-25534: --- I'll make a PR soon. > Make `SupportWith

[jira] [Assigned] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25534: - Assignee: Dongjoon Hyun > Make `SupportWithSQLConf` trait > ---

[jira] [Created] (SPARK-25534) Make `SupportWithSQLConf` trait

2018-09-25 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25534: - Summary: Make `SupportWithSQLConf` trait Key: SPARK-25534 URL: https://issues.apache.org/jira/browse/SPARK-25534 Project: Spark Issue Type: Bug C

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-25 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627934#comment-16627934 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] any updates on your PR, I can p

[jira] [Updated] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Summary: Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spar

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: (was: Screenshot from 2018-09-26 00-42-00.png) > Inconsistent message of Completed Jobs in t

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Description: Test steps:  1) bin/spark-shell {code:java} sc.parallelize(1 to 5, 5).collect() sc.parallelize

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-46-35.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-42-00.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Commented] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627823#comment-16627823 ] shahid commented on SPARK-25533: I am working on it. > Inconsistent message of Complete

[jira] [Updated] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-25533: --- Attachment: Screenshot from 2018-09-26 00-42-00.png > Inconsistent message of Completed Jobs in the JobUI,

[jira] [Created] (SPARK-25533) Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-25 Thread shahid (JIRA)
shahid created SPARK-25533: -- Summary: Inconsistent message of Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2 Key: SPARK-25533 URL: https://issues.apache.org/jira/browse/SPARK-25533

[jira] [Resolved] (SPARK-25495) FetchedData.reset doesn't reset _nextOffsetInFetchedData and _offsetAfterPoll

2018-09-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25495. -- Resolution: Fixed Fix Version/s: 2.4.0 > FetchedData.reset doesn't reset _nextOffsetInF

[jira] [Assigned] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-17636: --- Assignee: DB Tsai > Parquet filter push down doesn't handle struct fields > ---

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-25 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-17636: Fix Version/s: 2.5.0 > Parquet filter push down doesn't handle struct fields > ---

[jira] [Resolved] (SPARK-25486) Refactor SortBenchmark to use main method

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25486. --- Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22495 [https://

[jira] [Assigned] (SPARK-25486) Refactor SortBenchmark to use main method

2018-09-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25486: - Assignee: yucai > Refactor SortBenchmark to use main method > -

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627650#comment-16627650 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627649#comment-16627649 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25528: Assignee: Wenchen Fan (was: Apache Spark) > data source V2 read side API refactoring > -

[jira] [Commented] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627648#comment-16627648 ] Apache Spark commented on SPARK-25528: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-25528) data source V2 read side API refactoring

2018-09-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25528: Assignee: Apache Spark (was: Wenchen Fan) > data source V2 read side API refactoring > -

[jira] [Resolved] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2018-09-25 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-23521. --- Resolution: Fixed Marking this as "FIxed" because the vote passed. > SPIP: Standardize SQL logical

[jira] [Commented] (SPARK-25527) Job stuck waiting for last stage to start

2018-09-25 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627593#comment-16627593 ] Li Yuanjian commented on SPARK-25527: - {quote} There are no Tasks waiting for comple

[jira] [Created] (SPARK-25532) A stable and efficient row representation

2018-09-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25532: --- Summary: A stable and efficient row representation Key: SPARK-25532 URL: https://issues.apache.org/jira/browse/SPARK-25532 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-25531) new write APIs for data source v2

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627572#comment-16627572 ] Wenchen Fan commented on SPARK-25531: - Hi [~rdblue], I've created this umbrella JIRA

[jira] [Updated] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23521: Issue Type: Sub-task (was: Improvement) Parent: SPARK-25531 > SPIP: Standardize SQL logic

[jira] [Updated] (SPARK-24923) DataSourceV2: Add CTAS and RTAS logical operations

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24923: Parent Issue: SPARK-25531 (was: SPARK-22386) > DataSourceV2: Add CTAS and RTAS logical operations

[jira] [Updated] (SPARK-24253) DataSourceV2: Add DeleteSupport for delete and overwrite operations

2018-09-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24253: Parent Issue: SPARK-25531 (was: SPARK-22386) > DataSourceV2: Add DeleteSupport for delete and ove

  1   2   >