[jira] [Updated] (SPARK-25277) YARN applicationMaster metrics should not register static and JVM metrics

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25277: -- Fix Version/s: 2.4.5 > YARN applicationMaster metrics should not register static and JVM metri

[jira] [Updated] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29091: Affects Version/s: 2.4.4 > spark-shell don't support added jar's class as Serde class > -

[jira] [Commented] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930295#comment-16930295 ] Yuming Wang commented on SPARK-29091: - {noformat} [root@spark-3267648 spark-2.4.4-bi

[jira] [Updated] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29091: Parent: (was: SPARK-28684) Issue Type: Bug (was: Sub-task) > spark-shell don't suppor

[jira] [Created] (SPARK-29093) Remove automatically generated param setters in _shared_params_code_gen.py

2019-09-16 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29093: Summary: Remove automatically generated param setters in _shared_params_code_gen.py Key: SPARK-29093 URL: https://issues.apache.org/jira/browse/SPARK-29093 Project: S

[jira] [Commented] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930296#comment-16930296 ] Yuming Wang commented on SPARK-29091: - Spark 2.3.4: {noformat} [root@spark-3267648 s

[jira] [Updated] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29091: Affects Version/s: 2.3.4 > spark-shell don't support added jar's class as Serde class > -

[jira] [Comment Edited] (SPARK-29091) spark-shell don't support added jar's class as Serde class

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930295#comment-16930295 ] Yuming Wang edited comment on SPARK-29091 at 9/16/19 7:49 AM:

[jira] [Commented] (SPARK-29035) unpersist() ignoring cache/persist()

2019-09-16 Thread Jose Silva (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930297#comment-16930297 ] Jose Silva commented on SPARK-29035: [~hyukjin.kwon] What do you mean with "full re

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930298#comment-16930298 ] Adrian Wang commented on SPARK-13446: - [~elgalu][~toopt4][~headcra6][~jpbordi][~F775

[jira] [Updated] (SPARK-29089) DataFrameReader bottleneck in DataSource#checkAndGlobPathIfNecessary when reading large amount of S3 files

2019-09-16 Thread Arwin S Tio (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arwin S Tio updated SPARK-29089: Component/s: (was: Spark Core) SQL > DataFrameReader bottleneck in DataSource

[jira] [Updated] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Qiang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Wang updated SPARK-28927: --- Description: The stack trace is below: {quote}19/08/28 07:00:40 WARN Executor task launch worker for

[jira] [Updated] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Qiang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Wang updated SPARK-28927: --- Description: The stack trace is below: {quote}19/08/28 07:00:40 WARN Executor task launch worker for

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Qiang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930314#comment-16930314 ] Qiang Wang commented on SPARK-28927:  As the code above,  I cannot see any non-deter

[jira] [Created] (SPARK-29094) Add extractInstances method

2019-09-16 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29094: Summary: Add extractInstances method Key: SPARK-29094 URL: https://issues.apache.org/jira/browse/SPARK-29094 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-29094) Add extractInstances method

2019-09-16 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-29094. -- Resolution: Duplicate > Add extractInstances method > --- > >

[jira] [Created] (SPARK-29095) add extractInstances

2019-09-16 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29095: Summary: add extractInstances Key: SPARK-29095 URL: https://issues.apache.org/jira/browse/SPARK-29095 Project: Spark Issue Type: Improvement Compon

[jira] [Commented] (SPARK-29086) Use added jar's class as Serde class, SparkGetColumnsOperation return empty columns

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930449#comment-16930449 ] Yuming Wang commented on SPARK-29086: - Could we {{add jar}} before listing columns:

[jira] [Created] (SPARK-29096) The exact math method should be called only when there is a corresponding function in Math

2019-09-16 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-29096: -- Summary: The exact math method should be called only when there is a corresponding function in Math Key: SPARK-29096 URL: https://issues.apache.org/jira/browse/SPARK-29096

[jira] [Commented] (SPARK-29055) Memory leak in Spark Driver

2019-09-16 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930467#comment-16930467 ] Maxim Gekk commented on SPARK-29055: [~Geopap] Can you provide the csv files? > Mem

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread Vadim Panov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930468#comment-16930468 ] Vadim Panov commented on SPARK-13446: - [~adrian-wang] thanks! I've tried doing that,

[jira] [Commented] (SPARK-29086) Use added jar's class as Serde class, SparkGetColumnsOperation return empty columns

2019-09-16 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930470#comment-16930470 ] Yuming Wang commented on SPARK-29086: - I think this isn't a JDK issue. I can reprodu

[jira] [Commented] (SPARK-29086) Use added jar's class as Serde class, SparkGetColumnsOperation return empty columns

2019-09-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930510#comment-16930510 ] angerszhu commented on SPARK-29086: --- [~yumwang] Yes, can be reproduced in JDK8 hadoop2

[jira] [Commented] (SPARK-29086) Use added jar's class as Serde class, SparkGetColumnsOperation return empty columns

2019-09-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930514#comment-16930514 ] angerszhu commented on SPARK-29086: --- Add jar before listing column is OK.  But if you

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Description: I used Spark 2.1.1 and I upgraded into the latest version 2.4.4. I observed from Spa

[jira] [Updated] (SPARK-29055) Memory leak in Spark Driver

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29055: Attachment: test_csvs.zip > Memory leak in Spark Driver > --- > >

[jira] [Commented] (SPARK-29055) Memory leak in Spark Driver

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930517#comment-16930517 ] George Papa commented on SPARK-29055: - [~maxgekk] I have uploaded a zip with few tes

[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Attachment: driver_memory.png > Spark driver memory exceeded the storage memory >

[jira] [Created] (SPARK-29097) Spark driver memory exceeded the storage memory

2019-09-16 Thread George Papa (Jira)
George Papa created SPARK-29097: --- Summary: Spark driver memory exceeded the storage memory Key: SPARK-29097 URL: https://issues.apache.org/jira/browse/SPARK-29097 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Attachment: (was: driver_memory.png) > Spark driver memory exceeded the storage memory > -

[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory

2019-09-16 Thread George Papa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Description: In Spark UI, the driver used storage memory (2.1GB) exceeds the total available memo

[jira] [Commented] (SPARK-26205) Optimize InSet expression for bytes, shorts, ints, dates

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930527#comment-16930527 ] Wenchen Fan commented on SPARK-26205: - You can reproduce it if the value is not lite

[jira] [Assigned] (SPARK-29061) Prints bytecode statistics in debugCodegen

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29061: --- Assignee: Takeshi Yamamuro > Prints bytecode statistics in debugCodegen > -

[jira] [Resolved] (SPARK-29061) Prints bytecode statistics in debugCodegen

2019-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29061. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25766 [https://gith

[jira] [Assigned] (SPARK-29072) Properly track shuffle write time with refactor

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29072: Assignee: Matt Cheah > Properly track shuffle write time with refactor >

[jira] [Resolved] (SPARK-29072) Properly track shuffle write time with refactor

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29072. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25780 [https://gi

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-16 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930600#comment-16930600 ] Thomas Graves commented on SPARK-27495: --- The SPIP vote passed -  https://mail-arch

[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

2019-09-16 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930616#comment-16930616 ] Imran Rashid commented on SPARK-28917: -- I finally got some more info about this cas

[jira] [Updated] (SPARK-29070) Make SparkLauncher log full spark-submit command line

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Evans updated SPARK-29070: --- Summary: Make SparkLauncher log full spark-submit command line (was: Allow SparkLauncher to return

[jira] [Updated] (SPARK-29070) Make SparkLauncher log full spark-submit command line

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Evans updated SPARK-29070: --- Description: {{org.apache.spark.launcher.SparkLauncher}} wraps a {{ProcessBuilder}}, and builds up

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930658#comment-16930658 ] Liang-Chi Hsieh commented on SPARK-28927: - Because you are using 2.2.1, spark.sq

[jira] [Comment Edited] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930658#comment-16930658 ] Liang-Chi Hsieh edited comment on SPARK-28927 at 9/16/19 3:35 PM:

[jira] [Comment Edited] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930658#comment-16930658 ] Liang-Chi Hsieh edited comment on SPARK-28927 at 9/16/19 3:36 PM:

[jira] [Created] (SPARK-29098) Test both ANSI mode and Spark mode

2019-09-16 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-29098: -- Summary: Test both ANSI mode and Spark mode Key: SPARK-29098 URL: https://issues.apache.org/jira/browse/SPARK-29098 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930685#comment-16930685 ] JP Bordenave commented on SPARK-13446: -- i remove hive-exec from spark/jars and  hdf

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:41 PM: ---

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:43 PM: ---

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:43 PM: ---

[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2019-09-16 Thread JP Bordenave (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930685#comment-16930685 ] JP Bordenave edited comment on SPARK-13446 at 9/16/19 4:42 PM: ---

[jira] [Created] (SPARK-29099) org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set

2019-09-16 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-29099: Summary: org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set Key: SPARK-29099 URL: https://issues.apache.org/jira/browse/SPARK-29099 Project:

[jira] [Resolved] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26929. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 23952 [https:

[jira] [Assigned] (SPARK-26929) table owner should use user instead of principal when use kerberos

2019-09-16 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26929: -- Assignee: hong dongdong > table owner should use user instead of principal when use k

[jira] [Resolved] (SPARK-26524) If the application directory fails to be created on the SPARK_WORKER_DIR on some woker nodes (for example, bad disk or disk has no capacity), the application executor w

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26524. --- Resolution: Won't Fix > If the application directory fails to be created on the SPARK_WORKER_DIR on

[jira] [Resolved] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19184. --- Resolution: Won't Fix > Improve numerical stability for method tallSkinnyQR. > -

[jira] [Resolved] (SPARK-24671) DataFrame length using a dunder/magic method in PySpark

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24671. --- Resolution: Won't Fix > DataFrame length using a dunder/magic method in PySpark > --

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930763#comment-16930763 ] Harichandan Pulagam commented on SPARK-27648: - Here's a code example that re

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930763#comment-16930763 ] Harichandan Pulagam edited comment on SPARK-27648 at 9/16/19 6:38 PM:

[jira] [Comment Edited] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-09-16 Thread Harichandan Pulagam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930763#comment-16930763 ] Harichandan Pulagam edited comment on SPARK-27648 at 9/16/19 6:40 PM:

[jira] [Resolved] (SPARK-23694) The staging directory should under hive.exec.stagingdir if we set hive.exec.stagingdir but not under the table directory

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23694. --- Resolution: Won't Fix > The staging directory should under hive.exec.stagingdir if we set > hive.ex

[jira] [Created] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
Liang-Chi Hsieh created SPARK-29100: --- Summary: Codegen with switch in InSet expression causes compilation error Key: SPARK-29100 URL: https://issues.apache.org/jira/browse/SPARK-29100 Project: Spark

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-29100: Description: SPARK-26205 adds an optimization to InSet that generates Java switch conditio

[jira] [Resolved] (SPARK-23714) Add metrics for cached KafkaConsumer

2019-09-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-23714. --- Resolution: Duplicate Apache commons pool added which provides jmx metrics. > Add metrics f

[jira] [Closed] (SPARK-23714) Add metrics for cached KafkaConsumer

2019-09-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-23714. - > Add metrics for cached KafkaConsumer > > > Ke

[jira] [Assigned] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reassigned SPARK-29100: --- Assignee: Liang-Chi Hsieh > Codegen with switch in InSet expression causes compilat

[jira] [Commented] (SPARK-26205) Optimize InSet expression for bytes, shorts, ints, dates

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930787#comment-16930787 ] Liang-Chi Hsieh commented on SPARK-26205: - [~cloud_fan]. I see now. Created SPAR

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930796#comment-16930796 ] Karthikeyan Ravi commented on SPARK-25721: -- Hi Team, any updates on this, we ar

[jira] [Updated] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthikeyan Ravi updated SPARK-25721: - Attachment: Screen Shot 2019-09-16 at 12.27.25 PM.png > maxRate configuration not being

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930801#comment-16930801 ] Karthikeyan Ravi commented on SPARK-25721: -- Attaching screenshot  !Screen Shot

[jira] [Commented] (SPARK-25721) maxRate configuration not being used in Kinesis receiver

2019-09-16 Thread Karthikeyan Ravi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930803#comment-16930803 ] Karthikeyan Ravi commented on SPARK-25721: -- This problem very similar to https:

[jira] [Created] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-16 Thread Stuart White (Jira)
Stuart White created SPARK-29101: Summary: CSV datasource returns incorrect .count() from file with malformed records Key: SPARK-29101 URL: https://issues.apache.org/jira/browse/SPARK-29101 Project: S

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

2019-09-16 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930827#comment-16930827 ] Liang-Chi Hsieh commented on SPARK-28927: - Regarding to AUC unstable issue, the

[jira] [Created] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-16 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-29102: Summary: Read gzipped file into multiple partitions without full gzip expansion on a single-node Key: SPARK-29102 URL: https://issues.apache.org/jira/browse/SPARK-29102

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930835#comment-16930835 ] Nicholas Chammas commented on SPARK-29102: -- cc [~cloud_fan] and [~hyukjin.kwon]

[jira] [Resolved] (SPARK-24806) Brush up generated code so that JDK Java compilers can handle it

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24806. --- Resolution: Won't Fix > Brush up generated code so that JDK Java compilers can handle it > -

[jira] [Resolved] (SPARK-22111) OnlineLDAOptimizer should filter out empty documents beforehand

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22111. --- Resolution: Won't Fix > OnlineLDAOptimizer should filter out empty documents beforehand > -

[jira] [Resolved] (SPARK-10408) Autoencoder

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10408. --- Resolution: Won't Fix > Autoencoder > --- > > Key: SPARK-10408 >

[jira] [Resolved] (SPARK-22381) Add StringParam that supports valid options

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22381. --- Resolution: Won't Fix > Add StringParam that supports valid options > --

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Priority: Minor (was: Major) > Codegen with switch in InSet expression causes compilation err

[jira] [Commented] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930871#comment-16930871 ] Dongjoon Hyun commented on SPARK-29100: --- Since this is a prevention of the potenti

[jira] [Issue Comment Deleted] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Comment: was deleted (was: Since this is a prevention of the potential bug situation, I lower

[jira] [Updated] (SPARK-29100) Codegen with switch in InSet expression causes compilation error

2019-09-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29100: -- Priority: Major (was: Minor) > Codegen with switch in InSet expression causes compilation err

[jira] [Commented] (SPARK-25153) Improve error messages for columns with dots/periods

2019-09-16 Thread Jeff Evans (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930896#comment-16930896 ] Jeff Evans commented on SPARK-25153: Opened a pull request for this (see link added

[jira] [Created] (SPARK-29103) CheckAnalysis for data source V2 ALTER TABLE ignores case sensitivity

2019-09-16 Thread Jose Torres (Jira)
Jose Torres created SPARK-29103: --- Summary: CheckAnalysis for data source V2 ALTER TABLE ignores case sensitivity Key: SPARK-29103 URL: https://issues.apache.org/jira/browse/SPARK-29103 Project: Spark

[jira] [Resolved] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-09-16 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26565. --- Resolution: Not A Problem > modify dev/create-release/release-build.sh to let jenkins build packages

[jira] [Created] (SPARK-29104) Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished

2019-09-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29104: - Summary: Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished Key: SPARK-29104 URL: https://issues.apache.org/jira/browse/SPARK-29104

[jira] [Created] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-16 Thread Marcelo Vanzin (Jira)
Marcelo Vanzin created SPARK-29105: -- Summary: SHS may delete driver log file of in progress application Key: SPARK-29105 URL: https://issues.apache.org/jira/browse/SPARK-29105 Project: Spark

[jira] [Resolved] (SPARK-26781) Additional exchange gets added

2019-09-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26781. -- Resolution: Duplicate > Additional exchange gets added > --- > >

[jira] [Resolved] (SPARK-25216) Provide better error message when a column contains dot and needs backticks quote

2019-09-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25216. -- Resolution: Duplicate > Provide better error message when a column contains dot and needs back

[jira] [Created] (SPARK-29106) Add jenkins arm test for spark

2019-09-16 Thread huangtianhua (Jira)
huangtianhua created SPARK-29106: Summary: Add jenkins arm test for spark Key: SPARK-29106 URL: https://issues.apache.org/jira/browse/SPARK-29106 Project: Spark Issue Type: Test Com

[jira] [Created] (SPARK-29107) Add window.sql - Part 1

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29107: Summary: Add window.sql - Part 1 Key: SPARK-29107 URL: https://issues.apache.org/jira/browse/SPARK-29107 Project: Spark Issue Type: Sub-task Compon

[jira] [Created] (SPARK-29108) Add window.sql - Part 2

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29108: Summary: Add window.sql - Part 2 Key: SPARK-29108 URL: https://issues.apache.org/jira/browse/SPARK-29108 Project: Spark Issue Type: Sub-task Compon

[jira] [Created] (SPARK-29109) Add window.sql - Part 3

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29109: Summary: Add window.sql - Part 3 Key: SPARK-29109 URL: https://issues.apache.org/jira/browse/SPARK-29109 Project: Spark Issue Type: Sub-task Compon

[jira] [Created] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29110: Summary: Add window.sql - Part 4 Key: SPARK-29110 URL: https://issues.apache.org/jira/browse/SPARK-29110 Project: Spark Issue Type: Sub-task Compon

[jira] [Updated] (SPARK-29110) Add window.sql - Part 4

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29110: - Description: In this ticket, we plan to add the regression test cases of  [https://github.com/pos

[jira] [Updated] (SPARK-29108) Add window.sql - Part 2

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29108: - Description: In this ticket, we plan to add the regression test cases of  [https://github.com/pos

[jira] [Updated] (SPARK-29109) Add window.sql - Part 3

2019-09-16 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29109: - Description: In this ticket, we plan to add the regression test cases of  [https://github.com/pos

[jira] [Commented] (SPARK-29047) use spark-submit Not a file: hdfs://

2019-09-16 Thread bailin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931013#comment-16931013 ] bailin commented on SPARK-29047: i solved, it's code problem.   Generate and read parque

[jira] [Resolved] (SPARK-29008) Define an individual method for each common subexpression in HashAggregateExec

2019-09-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29008. -- Fix Version/s: 3.0.0 Assignee: Takeshi Yamamuro Resolution: Fixed Reso

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in `Expr

[jira] [Updated] (SPARK-29049) Rename DataSourceStrategy#normalizeFilters to DataSourceStrategy#normalizeAttrNames

2019-09-16 Thread Xianyin Xin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated SPARK-29049: Description: DataSourceStrategy#normalizeFilters can also be used to normalize attributes in \{{Ex

  1   2   >