[jira] [Commented] (SPARK-29142) Pyspark clustering models support column setters/getters/predict

2019-09-18 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933081#comment-16933081 ] Huaxin Gao commented on SPARK-29142: I will work on this. Thanks! > Pyspark clustering models

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933077#comment-16933077 ] Dongjoon Hyun commented on SPARK-29106: --- Ur, [~huangtianhua]. You should update the `Description`.

[jira] [Resolved] (SPARK-28989) Add `spark.sql.ansi.enabled`

2019-09-18 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28989. - Fix Version/s: 3.0.0 Resolution: Fixed > Add `spark.sql.ansi.enabled` >

[jira] [Updated] (SPARK-29168) Fix the appearance issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Summary: Fix the appearance issue on timeline view (was: Fix the appearnce issue on

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: SPARK-29168.url > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: SPARK-29168.url > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: (was: SPARK-29168.url) > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: (was: SPARK-29168.url) > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Description: In WebUI, executor bar's color changes blue to green with no meaning when

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: (was: html_before_c.png) > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: after_click.png > Fix the appearnce issue on timeline view >

[jira] [Updated] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Komiyama updated SPARK-29168: Attachment: html_before_c.png > Fix the appearnce issue on timeline view >

[jira] [Created] (SPARK-29168) Fix the appearnce issue on timeline view

2019-09-18 Thread Tomoko Komiyama (Jira)
Tomoko Komiyama created SPARK-29168: --- Summary: Fix the appearnce issue on timeline view Key: SPARK-29168 URL: https://issues.apache.org/jira/browse/SPARK-29168 Project: Spark Issue Type:

[jira] [Created] (SPARK-29167) Metrics of Analyzer/Optimizer use Scientific counting is not human readable

2019-09-18 Thread angerszhu (Jira)
angerszhu created SPARK-29167: - Summary: Metrics of Analyzer/Optimizer use Scientific counting is not human readable Key: SPARK-29167 URL: https://issues.apache.org/jira/browse/SPARK-29167 Project: Spark

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933014#comment-16933014 ] huangtianhua commented on SPARK-29106: -- The other important thing is about the leveldbjni 

[jira] [Comment Edited] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933001#comment-16933001 ] huangtianhua edited comment on SPARK-29106 at 9/19/19 2:34 AM: ---

[jira] [Comment Edited] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933001#comment-16933001 ] huangtianhua edited comment on SPARK-29106 at 9/19/19 2:31 AM: ---

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933001#comment-16933001 ] huangtianhua commented on SPARK-29106: -- [~dongjoon], thanks :) Till now we made two arm test

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932997#comment-16932997 ] Nicholas Chammas commented on SPARK-29102: -- {quote}It duplicately decompresses and each map

[jira] [Resolved] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-29102. -- Resolution: Won't Fix > Read gzipped file into multiple partitions without full gzip

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2019-09-18 Thread avinash v kodikal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932981#comment-16932981 ] avinash v kodikal commented on SPARK-27891: --- [~vanzin] - DId you get a chance to look at the

[jira] [Comment Edited] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932968#comment-16932968 ] Hyukjin Kwon edited comment on SPARK-29102 at 9/19/19 1:27 AM: --- Yea, that

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932970#comment-16932970 ] Hyukjin Kwon commented on SPARK-29102: -- So .. the workaround _might_ be

[jira] [Updated] (SPARK-29166) Add a parameter to limit the number of dynamic partitions for data source table

2019-09-18 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29166: --- Description: Dynamic partition in Hive table has some restrictions to limit the max number of

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932968#comment-16932968 ] Hyukjin Kwon commented on SPARK-29102: -- Yea, that _might_ work. It's been too long since I

[jira] [Created] (SPARK-29166) Add a parameter to limit the number of dynamic partitions for data source table

2019-09-18 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29166: -- Summary: Add a parameter to limit the number of dynamic partitions for data source table Key: SPARK-29166 URL: https://issues.apache.org/jira/browse/SPARK-29166 Project:

[jira] [Updated] (SPARK-28683) Upgrade Scala to 2.12.10

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28683: -- Fix Version/s: 2.4.5 > Upgrade Scala to 2.12.10 > > >

[jira] [Commented] (SPARK-28683) Upgrade Scala to 2.12.10

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932962#comment-16932962 ] Dongjoon Hyun commented on SPARK-28683: --- This is backported to `branch-2.4` for Apache Spark 2.4.5

[jira] [Updated] (SPARK-28683) Upgrade Scala to 2.12.10

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28683: -- Affects Version/s: 2.4.5 > Upgrade Scala to 2.12.10 > > >

[jira] [Resolved] (SPARK-29141) Use SqlBasedBenchmark in SQL benchmarks

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29141. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25828

[jira] [Created] (SPARK-29165) Set log level of log generated code as ERROR in case of compile error on generated code in UT

2019-09-18 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29165: Summary: Set log level of log generated code as ERROR in case of compile error on generated code in UT Key: SPARK-29165 URL: https://issues.apache.org/jira/browse/SPARK-29165

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932953#comment-16932953 ] Nicholas Chammas commented on SPARK-29102: -- Ah, thanks for the reference! So if I'm just trying

[jira] [Updated] (SPARK-29157) DataSourceV2: Add DataFrameWriterV2 to Python API

2019-09-18 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terry Kim updated SPARK-29157: -- Description: *strong text*After SPARK-28612 is committed, we need to add the corresponding PySpark

[jira] [Updated] (SPARK-29157) DataSourceV2: Add DataFrameWriterV2 to Python API

2019-09-18 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terry Kim updated SPARK-29157: -- Description: After SPARK-28612 is committed, we need to add the corresponding PySpark API. (was:

[jira] [Updated] (SPARK-29162) Simplify NOT(isnull(x)) and NOT(isnotnull(x))

2019-09-18 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-29162: --- Description: I propose the following expression rewrite optimizations: {code} NOT isnull(x) ->

[jira] [Created] (SPARK-29164) Rewrite coalesce(boolean, booleanLit) as boolean expression

2019-09-18 Thread Josh Rosen (Jira)
Josh Rosen created SPARK-29164: -- Summary: Rewrite coalesce(boolean, booleanLit) as boolean expression Key: SPARK-29164 URL: https://issues.apache.org/jira/browse/SPARK-29164 Project: Spark

[jira] [Created] (SPARK-29163) Provide a mixin to simplify HadoopConf access patterns in DataSource V2

2019-09-18 Thread holdenk (Jira)
holdenk created SPARK-29163: --- Summary: Provide a mixin to simplify HadoopConf access patterns in DataSource V2 Key: SPARK-29163 URL: https://issues.apache.org/jira/browse/SPARK-29163 Project: Spark

[jira] [Created] (SPARK-29162) Simplify NOT(isnull(x)) and NOT(isnotnull(x))

2019-09-18 Thread Josh Rosen (Jira)
Josh Rosen created SPARK-29162: -- Summary: Simplify NOT(isnull(x)) and NOT(isnotnull(x)) Key: SPARK-29162 URL: https://issues.apache.org/jira/browse/SPARK-29162 Project: Spark Issue Type:

[jira] [Created] (SPARK-29161) Unify default wait time for LiveListenerBus.waitUntilEmpty

2019-09-18 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29161: Summary: Unify default wait time for LiveListenerBus.waitUntilEmpty Key: SPARK-29161 URL: https://issues.apache.org/jira/browse/SPARK-29161 Project: Spark

[jira] [Commented] (SPARK-29078) Spark shell fails if read permission is not granted to hive warehouse directory

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932923#comment-16932923 ] Hyukjin Kwon commented on SPARK-29078: -- ping [~misutoth] > Spark shell fails if read permission is

[jira] [Commented] (SPARK-29099) org.apache.spark.sql.catalyst.catalog.CatalogTable.lastAccessTime is not set

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932922#comment-16932922 ] Hyukjin Kwon commented on SPARK-29099: -- Seems like we use "UNKNOWN" in that case: 1.

[jira] [Commented] (SPARK-29102) Read gzipped file into multiple partitions without full gzip expansion on a single-node

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932921#comment-16932921 ] Hyukjin Kwon commented on SPARK-29102: -- Hm, I took a look for this one few years ago and was pretty

[jira] [Updated] (SPARK-29145) Spark SQL cannot handle "NOT IN" condition when using "JOIN"

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29145: - Description: sample sql:  {code} spark.range(10).createOrReplaceTempView("A")

[jira] [Commented] (SPARK-29147) Spark doesn't use shuffleHashJoin as expected

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932915#comment-16932915 ] Hyukjin Kwon commented on SPARK-29147: -- [~ayudovin], let's interact with the mailing list first

[jira] [Commented] (SPARK-29146) 'DataFrame' object has no attribute 'copy'

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932917#comment-16932917 ] Hyukjin Kwon commented on SPARK-29146: -- Can you show the reproducer please? > 'DataFrame' object

[jira] [Resolved] (SPARK-29147) Spark doesn't use shuffleHashJoin as expected

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29147. -- Resolution: Invalid > Spark doesn't use shuffleHashJoin as expected >

[jira] [Updated] (SPARK-29147) Spark doesn't use shuffleHashJoin as expected

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29147: - Priority: Major (was: Critical) > Spark doesn't use shuffleHashJoin as expected >

[jira] [Commented] (SPARK-29156) Hive has appending data as part of cdc, In write mode we should be able to write only changes captured to teradata or datasource.

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932913#comment-16932913 ] Hyukjin Kwon commented on SPARK-29156: -- Can you clarify it and show the reproducer please? I cannot

[jira] [Updated] (SPARK-29156) Hive has appending data as part of cdc, In write mode we should be able to write only changes captured to teradata or datasource.

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29156: - Target Version/s: (was: 2.4.4) > Hive has appending data as part of cdc, In write mode we

[jira] [Commented] (SPARK-29160) Event log file is written without specific charset which should be ideally UTF-8

2019-09-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932909#comment-16932909 ] Jungtaek Lim commented on SPARK-29160: -- I'll raise a patch today. It might need to have config to

[jira] [Commented] (SPARK-29160) Event log file is written without specific charset which should be ideally UTF-8

2019-09-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932901#comment-16932901 ] Jungtaek Lim commented on SPARK-29160: -- While I just added 3.0.0 as Affected Version, all versions

[jira] [Updated] (SPARK-29160) Event log file is written without specific charset which should be ideally UTF-8

2019-09-18 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-29160: - Description: This issue is from observation by [~vanzin] :

[jira] [Created] (SPARK-29160) Event log file is written without specific charset which should be ideally UTF-8

2019-09-18 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29160: Summary: Event log file is written without specific charset which should be ideally UTF-8 Key: SPARK-29160 URL: https://issues.apache.org/jira/browse/SPARK-29160

[jira] [Updated] (SPARK-29159) Increase ReservedCodeCacheSize to 1G

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29159: -- Summary: Increase ReservedCodeCacheSize to 1G (was: Increase CodeCacheSize to 1G) >

[jira] [Commented] (SPARK-29042) Sampling-based RDD with unordered input should be INDETERMINATE

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932890#comment-16932890 ] Hyukjin Kwon commented on SPARK-29042: -- Usually I only set "Fix Version/s" as that's what the merge

[jira] [Created] (SPARK-29159) Increase CodeCacheSize to 1G

2019-09-18 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29159: - Summary: Increase CodeCacheSize to 1G Key: SPARK-29159 URL: https://issues.apache.org/jira/browse/SPARK-29159 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-29158) Expose SerializableConfiguration for DSv2

2019-09-18 Thread holdenk (Jira)
holdenk created SPARK-29158: --- Summary: Expose SerializableConfiguration for DSv2 Key: SPARK-29158 URL: https://issues.apache.org/jira/browse/SPARK-29158 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-29157) DataSourceV2: Add DataFrameWriterV2 to Python API

2019-09-18 Thread Ryan Blue (Jira)
Ryan Blue created SPARK-29157: - Summary: DataSourceV2: Add DataFrameWriterV2 to Python API Key: SPARK-29157 URL: https://issues.apache.org/jira/browse/SPARK-29157 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-09-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932833#comment-16932833 ] koert kuipers commented on SPARK-27665: --- oh wait i didnt realize there is a setting

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-09-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932823#comment-16932823 ] koert kuipers commented on SPARK-27665: --- i am a little nervous that this got merged into master

[jira] [Resolved] (SPARK-28683) Upgrade Scala to 2.12.10

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28683. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25404

[jira] [Assigned] (SPARK-29082) Spark driver cannot start with only delegation tokens

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-29082: -- Assignee: Marcelo Vanzin > Spark driver cannot start with only delegation tokens >

[jira] [Resolved] (SPARK-29082) Spark driver cannot start with only delegation tokens

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29082. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25805

[jira] [Assigned] (SPARK-28683) Upgrade Scala to 2.12.10

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28683: - Assignee: Yuming Wang > Upgrade Scala to 2.12.10 > > >

[jira] [Commented] (SPARK-29042) Sampling-based RDD with unordered input should be INDETERMINATE

2019-09-18 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932784#comment-16932784 ] Liang-Chi Hsieh commented on SPARK-29042: - [~hyukjin.kwon] Am I setting the fix versions and

[jira] [Updated] (SPARK-29042) Sampling-based RDD with unordered input should be INDETERMINATE

2019-09-18 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-29042: Fix Version/s: 2.4.5 > Sampling-based RDD with unordered input should be INDETERMINATE >

[jira] [Created] (SPARK-29156) Hive has appending data as part of cdc, In write mode we should be able to write only changes captured to teradata or datasource.

2019-09-18 Thread raju (Jira)
raju created SPARK-29156: Summary: Hive has appending data as part of cdc, In write mode we should be able to write only changes captured to teradata or datasource. Key: SPARK-29156 URL:

[jira] [Resolved] (SPARK-22796) Add multiple column support to PySpark QuantileDiscretizer

2019-09-18 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-22796. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25812

[jira] [Assigned] (SPARK-22796) Add multiple column support to PySpark QuantileDiscretizer

2019-09-18 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reassigned SPARK-22796: --- Assignee: Huaxin Gao > Add multiple column support to PySpark QuantileDiscretizer

[jira] [Commented] (SPARK-22390) Aggregate push down

2019-09-18 Thread holdenk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932739#comment-16932739 ] holdenk commented on SPARK-22390: - Love to follow where this is going, especially if it gets broken into

[jira] [Created] (SPARK-29155) Support special date/timestamp values in the PostgreSQL dialect only

2019-09-18 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29155: -- Summary: Support special date/timestamp values in the PostgreSQL dialect only Key: SPARK-29155 URL: https://issues.apache.org/jira/browse/SPARK-29155 Project: Spark

[jira] [Assigned] (SPARK-28091) Extend Spark metrics system with user-defined metrics using executor plugins

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28091: -- Assignee: Luca Canali > Extend Spark metrics system with user-defined metrics using

[jira] [Resolved] (SPARK-28091) Extend Spark metrics system with user-defined metrics using executor plugins

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28091. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 24901

[jira] [Reopened] (SPARK-29058) Reading csv file with DROPMALFORMED showing incorrect record count

2019-09-18 Thread Suchintak Patnaik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suchintak Patnaik reopened SPARK-29058: --- Though the workaround of caching the dataframe first and then using count() works

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2019-09-18 Thread Jacob Niebloom (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932666#comment-16932666 ] Jacob Niebloom commented on SPARK-25075: I am a possible new contributor the Spark. Is there a

[jira] [Assigned] (SPARK-29141) Use SqlBasedBenchmark in SQL benchmarks

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29141: - Assignee: Maxim Gekk > Use SqlBasedBenchmark in SQL benchmarks >

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932646#comment-16932646 ] Dongjoon Hyun commented on SPARK-29106: --- I changed the affected version to 3.0.0 because this is a

[jira] [Updated] (SPARK-29141) Use SqlBasedBenchmark in SQL benchmarks

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29141: -- Priority: Minor (was: Trivial) > Use SqlBasedBenchmark in SQL benchmarks >

[jira] [Updated] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29106: -- Affects Version/s: (was: 2.4.4) 3.0.0 > Add jenkins arm test for

[jira] [Updated] (SPARK-29106) Add jenkins arm test for spark

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29106: -- Priority: Minor (was: Major) > Add jenkins arm test for spark >

[jira] [Assigned] (SPARK-28208) When upgrading to ORC 1.5.6, the reader needs to be closed.

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28208: - Assignee: Owen O'Malley > When upgrading to ORC 1.5.6, the reader needs to be closed.

[jira] [Resolved] (SPARK-28208) When upgrading to ORC 1.5.6, the reader needs to be closed.

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28208. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25006

[jira] [Resolved] (SPARK-29030) Simplify lookupV2Relation

2019-09-18 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29030. - Fix Version/s: 3.0.0 Assignee: John Zhuge Resolution: Done Resolved by

[jira] [Updated] (SPARK-16452) basic INFORMATION_SCHEMA support

2019-09-18 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-16452: Target Version/s: (was: 3.0.0) > basic INFORMATION_SCHEMA support > >

[jira] [Resolved] (SPARK-26022) PySpark Comparison with Pandas

2019-09-18 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26022. - Target Version/s: (was: 3.0.0) Resolution: Later > PySpark Comparison with Pandas >

[jira] [Commented] (SPARK-26022) PySpark Comparison with Pandas

2019-09-18 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932610#comment-16932610 ] Xiao Li commented on SPARK-26022: - [https://github.com/databricks/koalas] is to close the gap.  >

[jira] [Assigned] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-29105: -- Assignee: Marcelo Vanzin > SHS may delete driver log file of in progress application

[jira] [Resolved] (SPARK-29105) SHS may delete driver log file of in progress application

2019-09-18 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-29105. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25819

[jira] [Updated] (SPARK-29104) Fix Flaky Test - PipedRDDSuite. stdin_writer_thread_should_be_exited_when_task_is_finished

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29104: -- Fix Version/s: 2.4.5 > Fix Flaky Test - PipedRDDSuite. >

[jira] [Updated] (SPARK-26713) PipedRDD may holds stdin writer and stdout read threads even if the task is finished

2019-09-18 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26713: -- Fix Version/s: 2.4.5 > PipedRDD may holds stdin writer and stdout read threads even if the

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-18 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Epic Name: Stage Level Scheduling > SPIP: Support Stage level resource configuration and

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-18 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Labels: SPIP (was: ) > SPIP: Support Stage level resource configuration and scheduling >

[jira] [Created] (SPARK-29154) Update Spark scheduler for stage level scheduling

2019-09-18 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-29154: - Summary: Update Spark scheduler for stage level scheduling Key: SPARK-29154 URL: https://issues.apache.org/jira/browse/SPARK-29154 Project: Spark Issue

[jira] [Updated] (SPARK-29150) Update RDD API for Stage level scheduling

2019-09-18 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29150: -- Description: See the SPIP and design doc attached to SPARK-27495.  Note this is meant to be

[jira] [Created] (SPARK-29153) ResourceProfile conflict resolution stage level scheduling

2019-09-18 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-29153: - Summary: ResourceProfile conflict resolution stage level scheduling Key: SPARK-29153 URL: https://issues.apache.org/jira/browse/SPARK-29153 Project: Spark

[jira] [Resolved] (SPARK-29118) Avoid redundant computation in GMM.transform && GLR.transform

2019-09-18 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-29118. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25815

[jira] [Assigned] (SPARK-29118) Avoid redundant computation in GMM.transform && GLR.transform

2019-09-18 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-29118: - Assignee: zhengruifeng > Avoid redundant computation in GMM.transform && GLR.transform >

[jira] [Assigned] (SPARK-19926) Make pyspark exception more readable

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-19926: Assignee: Xianjin YE (was: Genmao Yu) > Make pyspark exception more readable >

[jira] [Updated] (SPARK-19926) Make pyspark exception more readable

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19926: - Labels: (was: bulk-closed) > Make pyspark exception more readable >

[jira] [Resolved] (SPARK-29101) CSV datasource returns incorrect .count() from file with malformed records

2019-09-18 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29101. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25820

  1   2   >