[jira] [Commented] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

2019-02-04 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760539#comment-16760539 ] shahid commented on SPARK-26760: Can you try with time taking task? Also, see the job page and stage

[jira] [Commented] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

2019-02-04 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760534#comment-16760534 ] ABHISHEK KUMAR GUPTA commented on SPARK-26760: -- spark.ui.liveUpdate.period after setting

[jira] [Comment Edited] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

2019-02-04 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760521#comment-16760521 ] shahid edited comment on SPARK-26760 at 2/5/19 7:12 AM: After analyzing,

[jira] [Commented] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

2019-02-04 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760521#comment-16760521 ] shahid commented on SPARK-26760: After analyzing, following are the findings 1) Tasks related

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760444#comment-16760444 ] Takeshi Yamamuro commented on SPARK-24657: -- Can you put a simple example query for the issue?

[jira] [Resolved] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-02-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26758. --- Resolution: Fixed Fix Version/s: 2.3.4 2.4.1 3.0.0

[jira] [Assigned] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-02-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26758: - Assignee: sandeep katta > Idle Executors are not getting killed after >

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

2019-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: (was: Marcelo Vanzin) > SystemClock uses non-monotonic time in its

[jira] [Assigned] (SPARK-10614) SystemClock uses non-monotonic time in its wait logic

2019-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-10614: -- Assignee: Marcelo Vanzin > SystemClock uses non-monotonic time in its wait logic >

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo edited comment on SPARK-24657 at 2/5/19 12:25 AM: -- Nice find. This

[jira] [Created] (SPARK-26825) Spark Structure Streaming job failing when submitted in cluster mode

2019-02-04 Thread Andre Araujo (JIRA)
Andre Araujo created SPARK-26825: Summary: Spark Structure Streaming job failing when submitted in cluster mode Key: SPARK-26825 URL: https://issues.apache.org/jira/browse/SPARK-26825 Project: Spark

[jira] [Comment Edited] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo edited comment on SPARK-24657 at 2/5/19 12:05 AM: -- Nice find. This

[jira] [Commented] (SPARK-24657) SortMergeJoin may cause SparkOutOfMemory in execution memory because of not cleanup resource when finished the merge join

2019-02-04 Thread Tao Luo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760328#comment-16760328 ] Tao Luo commented on SPARK-24657: - Nice find. This looks like 

[jira] [Assigned] (SPARK-26801) Spark unable to read valid avro types

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26801: Assignee: (was: Apache Spark) > Spark unable to read valid avro types >

[jira] [Assigned] (SPARK-26801) Spark unable to read valid avro types

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26801: Assignee: Apache Spark > Spark unable to read valid avro types >

[jira] [Commented] (SPARK-26801) Spark unable to read valid avro types

2019-02-04 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760178#comment-16760178 ] Dhruve Ashar commented on SPARK-26801: -- I have given a short summary of the issue in the PR

[jira] [Commented] (SPARK-20049) Writing data to Parquet with partitions takes very long after the job finishes

2019-02-04 Thread Juan Ramos Fuentes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760123#comment-16760123 ] Juan Ramos Fuentes commented on SPARK-20049: Was there ever a fix for this issue? I'm

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Affects Version/s: 2.0.0 2.1.0 2.2.0

[jira] [Assigned] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26824: Assignee: Shixiong Zhu (was: Apache Spark) > Streaming queries may store checkpoint

[jira] [Assigned] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26824: Assignee: Apache Spark (was: Shixiong Zhu) > Streaming queries may store checkpoint

[jira] [Commented] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760071#comment-16760071 ] Shixiong Zhu commented on SPARK-26824: -- This will need a release note. After the fix, the paths to

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Description: When a user specifies a checkpoint location containing special chars that need to

[jira] [Created] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-26824: Summary: Streaming queries may store checkpoint data in a wrong directory Key: SPARK-26824 URL: https://issues.apache.org/jira/browse/SPARK-26824 Project: Spark

[jira] [Updated] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26824: - Labels: release-notes (was: ) > Streaming queries may store checkpoint data in a wrong

[jira] [Commented] (SPARK-26821) filters not working with char datatype when querying against hive table

2019-02-04 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760063#comment-16760063 ] Sujith commented on SPARK-26821: bit tricky to handle this scenario eventhough. > filters not working

[jira] [Commented] (SPARK-26821) filters not working with char datatype when querying against hive table

2019-02-04 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760061#comment-16760061 ] Sujith commented on SPARK-26821: yes sean, but same i tested with MYSQL its giving me a result. not sure

[jira] [Commented] (SPARK-26821) filters not working with char datatype when querying against hive table

2019-02-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760058#comment-16760058 ] Sean Owen commented on SPARK-26821: --- I don't know a lot about this, but assuming the padding behavior

[jira] [Comment Edited] (SPARK-26821) filters not working with char datatype when querying against hive table

2019-02-04 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759382#comment-16759382 ] Sujith edited comment on SPARK-26821 at 2/4/19 5:38 PM: cc [~dongjoon]

[jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760048#comment-16760048 ] Raj commented on SPARK-26804: -   Also you can see how my query Col3 in where clause fails.  

[jira] [Updated] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-26804: Attachment: image-2019-02-04-12-28-21-117.png > Spark sql carries newline char from last csv column when imported

[jira] [Comment Edited] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760036#comment-16760036 ] Raj edited comment on SPARK-26804 at 2/4/19 5:22 PM: - Hi Hyukjin,     I have

[jira] [Updated] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-26804: Attachment: TestFile.csv > Spark sql carries newline char from last csv column when imported >

[jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760036#comment-16760036 ] Raj commented on SPARK-26804: - Hi Hyukjin,     I have attached the sample file to reproduce the same at

[jira] [Updated] (SPARK-26804) Spark sql carries newline char from last csv column when imported

2019-02-04 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-26804: Attachment: image-2019-02-04-12-09-19-210.png > Spark sql carries newline char from last csv column when imported

[jira] [Created] (SPARK-26823) SBT Build Warnings

2019-02-04 Thread Ahshan (JIRA)
Ahshan created SPARK-26823: -- Summary: SBT Build Warnings Key: SPARK-26823 URL: https://issues.apache.org/jira/browse/SPARK-26823 Project: Spark Issue Type: Bug Components: Build

[jira] [Assigned] (SPARK-26389) temp checkpoint folder at executor should be deleted on graceful shutdown

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26389: Assignee: Apache Spark > temp checkpoint folder at executor should be deleted on

[jira] [Assigned] (SPARK-26389) temp checkpoint folder at executor should be deleted on graceful shutdown

2019-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26389: Assignee: (was: Apache Spark) > temp checkpoint folder at executor should be deleted

[jira] [Resolved] (SPARK-26813) Consolidate java version across language compilers and build tools

2019-02-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26813. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23724

[jira] [Assigned] (SPARK-26813) Consolidate java version across language compilers and build tools

2019-02-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26813: - Assignee: Chenxiao Mao > Consolidate java version across language compilers and build tools >

[jira] [Comment Edited] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : wo

2019-02-04 Thread M. Le Bihan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759692#comment-16759692 ] M. Le Bihan edited comment on SPARK-26819 at 2/4/19 12:40 PM: -- I use

[jira] [Updated] (SPARK-26572) Join on distinct column with monotonically_increasing_id produces wrong output

2019-02-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26572: - Labels: correctness (was: ) > Join on distinct column with monotonically_increasing_id

[jira] [Commented] (SPARK-26572) Join on distinct column with monotonically_increasing_id produces wrong output

2019-02-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759788#comment-16759788 ] Takeshi Yamamuro commented on SPARK-26572: -- I checked I could reproduce this below and I set

[jira] [Updated] (SPARK-26572) Join on distinct column with monotonically_increasing_id produces wrong output

2019-02-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-26572: - Affects Version/s: 2.3.2 > Join on distinct column with monotonically_increasing_id

[jira] [Commented] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : working

2019-02-04 Thread M. Le Bihan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759692#comment-16759692 ] M. Le Bihan commented on SPARK-26819: - If I have these dependencies in my POM, my operation will

[jira] [Comment Edited] (SPARK-26819) ArrayIndexOutOfBoundsException while loading a CSV to a Dataset with dependencies spark-core_2.12 and spark-sql_2.12 (with spark-core_2.11 and spark-sql_2.11 : wo

2019-02-04 Thread M. Le Bihan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759692#comment-16759692 ] M. Le Bihan edited comment on SPARK-26819 at 2/4/19 8:43 AM: - If I have