[jira] [Resolved] (SPARK-21743) top-most limit should not cause memory leak

[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21743. - Resolution: Fixed Fix Version/s: 2.3.0 > top-most limit should not cause memory leak >

[jira] [Updated] (SPARK-21759) PullupCorrelatedPredicates should not produce unresolved plan

[ https://issues.apache.org/jira/browse/SPARK-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21759: Summary: PullupCorrelatedPredicates should not produce unresolved plan (was:

[jira] [Created] (SPARK-21759) PullupCorrelatedPredicates can produce unresolved plan

Liang-Chi Hsieh created SPARK-21759: --- Summary: PullupCorrelatedPredicates can produce unresolved plan Key: SPARK-21759 URL: https://issues.apache.org/jira/browse/SPARK-21759 Project: Spark

[jira] [Commented] (SPARK-16473) BisectingKMeans Algorithm failing with java.util.NoSuchElementException: key not found

[ https://issues.apache.org/jira/browse/SPARK-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129818#comment-16129818 ] zhengruifeng commented on SPARK-16473: -- If the {{sparseDataset}} in {{BisectingKMeansSuite}} is

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129815#comment-16129815 ] cen yuhai commented on SPARK-16188: --- [~xianlongZhang] You can use distribute by rand() or some other

[jira] [Created] (SPARK-21758) `SHOW TBLPROPERTIES` can not get properties start with spark.sql.*

StanZhai created SPARK-21758: Summary: `SHOW TBLPROPERTIES` can not get properties start with spark.sql.* Key: SPARK-21758 URL: https://issues.apache.org/jira/browse/SPARK-21758 Project: Spark

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129765#comment-16129765 ] Hyukjin Kwon commented on SPARK-21753: -- Hi [~tgraves] and [~holdenk], I had some time to reproduce

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129742#comment-16129742 ] xianlongZhang commented on SPARK-16188: --- But when we use Spark sql, we can not call the 'coalesce'

[jira] [Assigned] (SPARK-3151) DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit

[ https://issues.apache.org/jira/browse/SPARK-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-3151: -- Assignee: Eyal Farago > DiskStore attempts to map any size BlockId without checking

[jira] [Resolved] (SPARK-3151) DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit

[ https://issues.apache.org/jira/browse/SPARK-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-3151. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18855

[jira] [Updated] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

[ https://issues.apache.org/jira/browse/SPARK-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ash updated SPARK-21757: --- Description: At the following two places if the {{e.reason}} value contains a single quote

[jira] [Created] (SPARK-21757) Jobs page fails to load when executor removed event's reason contains single quote

Andrew Ash created SPARK-21757: -- Summary: Jobs page fails to load when executor removed event's reason contains single quote Key: SPARK-21757 URL: https://issues.apache.org/jira/browse/SPARK-21757

[jira] [Created] (SPARK-21756) Add JSON option to allow unquoted control characters

Bryan Chen created SPARK-21756: -- Summary: Add JSON option to allow unquoted control characters Key: SPARK-21756 URL: https://issues.apache.org/jira/browse/SPARK-21756 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129589#comment-16129589 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 11:18 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129589#comment-16129589 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] I tried the second example with

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129472#comment-16129472 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 10:29 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129512#comment-16129512 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 10:27 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129512#comment-16129512 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 10:15 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129512#comment-16129512 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 10:11 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129512#comment-16129512 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] Actually it didnt help I tried

[jira] [Updated] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakub Nowacki updated SPARK-21752: -- Description: If I put a config key {{spark.jars.packages}} using {{SparkSession}} builder as

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 10:01 PM:

[jira] [Created] (SPARK-21755) Spark 2.1.1 UI page not displaying any dynamic updates on job progress after showing progress for initial few minutes of job run.

Ankur created SPARK-21755: - Summary: Spark 2.1.1 UI page not displaying any dynamic updates on job progress after showing progress for initial few minutes of job run. Key: SPARK-21755 URL:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129502#comment-16129502 ] Jakub Nowacki commented on SPARK-21752: --- Not sure if [SPARK-11520] would help in this case. I'll

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129472#comment-16129472 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:59 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:58 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:57 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:54 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:52 PM:

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129490#comment-16129490 ] poplav commented on SPARK-21720: Are there any updates on this? > Filter predicate with many conditions

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:50 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129488#comment-16129488 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] I didnt say parenthesis is the

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129475#comment-16129475 ] Jakub Nowacki edited comment on SPARK-21752 at 8/16/17 9:46 PM: I'm aware

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129475#comment-16129475 ] Jakub Nowacki commented on SPARK-21752: --- OK so you don't need session creation with pyspark command

[jira] [Updated] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakub Nowacki updated SPARK-21752: -- Description: If I put a config key {{spark.jars.packages}} using {{SparkSession}} builder as

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129472#comment-16129472 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:41 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129472#comment-16129472 ] Stavros Kontopoulos commented on SPARK-21752: - @Jakub Nowacki I figured out the import, no I

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129466#comment-16129466 ] Jakub Nowacki commented on SPARK-21752: --- Not really, maybe you're missing {{import pyspark}} at the

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:37 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:35 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:33 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:33 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:32 PM:

[jira] [Updated] (SPARK-21754) No Exception/Warn When Join Columns are Differing Types

[ https://issues.apache.org/jira/browse/SPARK-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Lee updated SPARK-21754: --- Description: No Exception/Warn When Join Columns are Differing Types, which can lead to problematic join

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129454#comment-16129454 ] Stavros Kontopoulos commented on SPARK-21752: - I couldnt reproduce this one:

[jira] [Created] (SPARK-21754) No Exception/Warn When Join Columns are Differing Types

Ed Lee created SPARK-21754: -- Summary: No Exception/Warn When Join Columns are Differing Types Key: SPARK-21754 URL: https://issues.apache.org/jira/browse/SPARK-21754 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129432#comment-16129432 ] Jakub Nowacki edited comment on SPARK-21752 at 8/16/17 9:15 PM: [~skonto]

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129432#comment-16129432 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] Not sure which one you couldn't reproduce. Using

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129406#comment-16129406 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:04 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129406#comment-16129406 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/16/17 9:04 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129406#comment-16129406 ] Stavros Kontopoulos commented on SPARK-21752: - "unclear why passing SparkConf via

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129380#comment-16129380 ] Imran Rashid commented on SPARK-20589: -- Its pretty much the same thing whether you're trying to

[jira] [Updated] (SPARK-21713) Replace LogicalPlan.isStreaming with OutputMode

[ https://issues.apache.org/jira/browse/SPARK-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21713: - Component/s: (was: Spark Core) Structured Streaming SQL >

[jira] [Commented] (SPARK-20952) ParquetFileFormat should forward TaskContext to its forkjoinpool

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129327#comment-16129327 ] Robert Kruszewski commented on SPARK-20952: --- I don't care about what actual filesystem impls

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129326#comment-16129326 ] Thomas Graves commented on SPARK-21753: --- I should also state that many other things work fine with

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129322#comment-16129322 ] Jakub Nowacki commented on SPARK-21752: --- Well, it seems so, but it is, at least logically, unclear

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129320#comment-16129320 ] Thomas Graves commented on SPARK-21753: --- [~holden.ka...@gmail.com] would be curious if you have run

[jira] [Created] (SPARK-21753) running pi example with pypy on spark fails to serialize

Thomas Graves created SPARK-21753: - Summary: running pi example with pypy on spark fails to serialize Key: SPARK-21753 URL: https://issues.apache.org/jira/browse/SPARK-21753 Project: Spark

[jira] [Updated] (SPARK-21739) timestamp partition would fail in v2.2.0

[ https://issues.apache.org/jira/browse/SPARK-21739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21739: Priority: Critical (was: Major) > timestamp partition would fail in v2.2.0 >

[jira] [Updated] (SPARK-21739) timestamp partition would fail in v2.2.0

[ https://issues.apache.org/jira/browse/SPARK-21739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21739: Component/s: (was: Spark Core) SQL > timestamp partition would fail in v2.2.0 >

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129296#comment-16129296 ] Sean Owen commented on SPARK-21752: --- At that point, your app has already launched. You can't change the

[jira] [Created] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

Jakub Nowacki created SPARK-21752: - Summary: Config spark.jars.packages is ignored in SparkSession config Key: SPARK-21752 URL: https://issues.apache.org/jira/browse/SPARK-21752 Project: Spark

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129289#comment-16129289 ] Amit Kumar commented on SPARK-20589: As you said, adding job boundary via code will be much easier

[jira] [Assigned] (SPARK-21680) ML/MLLIB Vector compressed optimization

[ https://issues.apache.org/jira/browse/SPARK-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21680: - Assignee: Peng Meng Priority: Minor (was: Major) > ML/MLLIB Vector compressed optimization

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129198#comment-16129198 ] Imran Rashid commented on SPARK-20589: -- the proposed solution will only let you control concurrency

[jira] [Resolved] (SPARK-21680) ML/MLLIB Vector compressed optimization

[ https://issues.apache.org/jira/browse/SPARK-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21680. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18899

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129149#comment-16129149 ] Miles Crawford edited comment on SPARK-18838 at 8/16/17 5:57 PM: -

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129149#comment-16129149 ] Miles Crawford commented on SPARK-18838: [~imranr] and [~vanzin] Here is your log:

[jira] [Created] (SPARK-21751) CodeGeneraor.splitExpressions counts code size more precisely

Kazuaki Ishizaki created SPARK-21751: Summary: CodeGeneraor.splitExpressions counts code size more precisely Key: SPARK-21751 URL: https://issues.apache.org/jira/browse/SPARK-21751 Project: Spark

[jira] [Commented] (SPARK-21750) Use arrow 0.6.0

[ https://issues.apache.org/jira/browse/SPARK-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129133#comment-16129133 ] Kazuaki Ishizaki commented on SPARK-21750: -- Waiting for it on mvnrepository > Use arrow 0.6.0 >

[jira] [Created] (SPARK-21750) Use arrow 0.6.0

Kazuaki Ishizaki created SPARK-21750: Summary: Use arrow 0.6.0 Key: SPARK-21750 URL: https://issues.apache.org/jira/browse/SPARK-21750 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-21738) Thriftserver doesn't cancel jobs when session is closed

[ https://issues.apache.org/jira/browse/SPARK-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21738. - Resolution: Fixed Fix Version/s: 2.3.0 > Thriftserver doesn't cancel jobs when session is closed

[jira] [Assigned] (SPARK-21738) Thriftserver doesn't cancel jobs when session is closed

[ https://issues.apache.org/jira/browse/SPARK-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21738: --- Assignee: Marco Gaido > Thriftserver doesn't cancel jobs when session is closed >

[jira] [Resolved] (SPARK-21603) The wholestage codegen will be much slower then wholestage codegen is closed when the function is too long

[ https://issues.apache.org/jira/browse/SPARK-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21603. - Resolution: Fixed Assignee: eaton Fix Version/s: 2.3.0 > The wholestage codegen will be

[jira] [Commented] (SPARK-21684) df.write double escaping all the already escaped characters except the first one

[ https://issues.apache.org/jira/browse/SPARK-21684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128960#comment-16128960 ] Vinod KC commented on SPARK-21684: -- [~taransaini43] Can try with option("quoteAll", "true") ? Eg :

[jira] [Commented] (SPARK-20952) ParquetFileFormat should forward TaskContext to its forkjoinpool

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128891#comment-16128891 ] Steve Loughran commented on SPARK-20952: out of curiosity, what "filesystem games" are you

[jira] [Assigned] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21656: - Assignee: Jong Yoon Lee > spark dynamic allocation should not idle timeout executors

[jira] [Resolved] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21656. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > spark dynamic

[jira] [Created] (SPARK-21749) Add comments for MessageEncoder to explain the wire format

Xu Zhang created SPARK-21749: Summary: Add comments for MessageEncoder to explain the wire format Key: SPARK-21749 URL: https://issues.apache.org/jira/browse/SPARK-21749 Project: Spark Issue

[jira] [Updated] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21737: -- Target Version/s: (was: 2.1.1) Fix Version/s: (was: 2.1.1) > Create communication channel

[jira] [Comment Edited] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128725#comment-16128725 ] Yan Facai (颜发才) edited comment on SPARK-21748 at 8/16/17 12:38 PM: ---

[jira] [Commented] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128725#comment-16128725 ] Yan Facai (颜发才) commented on SPARK-21748: - [~yanboliang] Thanks, yanbo. As discussed on

[jira] [Created] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

Yan Facai (颜发才) created SPARK-21748: --- Summary: Migrate the implementation of HashingTF from MLlib to ML Key: SPARK-21748 URL: https://issues.apache.org/jira/browse/SPARK-21748 Project: Spark

[jira] [Resolved] (SPARK-21747) Java encoders - switch fields on collectAsList

[ https://issues.apache.org/jira/browse/SPARK-21747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21747. --- Resolution: Duplicate There's no need to copy it > Java encoders - switch fields on collectAsList >

[jira] [Created] (SPARK-21747) Java encoders - switch fields on collectAsList

Paul Praet created SPARK-21747: -- Summary: Java encoders - switch fields on collectAsList Key: SPARK-21747 URL: https://issues.apache.org/jira/browse/SPARK-21747 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128623#comment-16128623 ] zhengruifeng commented on SPARK-21742: -- [~srowen] I create {{random}} and {{rdd}} twice in REPL with

[jira] [Updated] (SPARK-21746) nondeterministic expressions incorrectly for filter predicates

[ https://issues.apache.org/jira/browse/SPARK-21746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caoxuewen updated SPARK-21746: -- Summary: nondeterministic expressions incorrectly for filter predicates (was: nondeterministic

[jira] [Created] (SPARK-21746) nondeterministic expressions correctly for filter predicates

caoxuewen created SPARK-21746: - Summary: nondeterministic expressions correctly for filter predicates Key: SPARK-21746 URL: https://issues.apache.org/jira/browse/SPARK-21746 Project: Spark

[jira] [Commented] (SPARK-19019) PySpark does not work with Python 3.6.0

[ https://issues.apache.org/jira/browse/SPARK-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128593#comment-16128593 ] Mathias M. Andersen commented on SPARK-19019: - Yea. This was just a pythonpath mishab on our

[jira] [Commented] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128580#comment-16128580 ] Sean Owen commented on SPARK-21742: --- Fixing the seed still doesn't mean that the two cases get the same

[jira] [Commented] (SPARK-13758) Error message is misleading when RDD refer to null spark context

[ https://issues.apache.org/jira/browse/SPARK-13758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128570#comment-16128570 ] Yun Tang commented on SPARK-13758: -- For developers who visit this page in the future, there is one case

[jira] [Updated] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-21742: - Description: I found that {{BisectingKMeans}} will generate different models if the input is

[jira] [Commented] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128535#comment-16128535 ] zhengruifeng commented on SPARK-21742: -- [~mlnick] The seed is already fixed. It looks like if we use

[jira] [Updated] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-21742: - Description: I found that {{BisectingKMeans}} will generate different models if the input is

[jira] [Updated] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-21742: - Description: I found that {{BisectingKMeans}} will generate different models if the input is

[jira] [Updated] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-21742: - Description: I found that {{BisectingKMeans}} will generate different models if the input is

[jira] [Commented] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128512#comment-16128512 ] Nick Pentreath commented on SPARK-21742: Isn't the solution to set a fixed seed for the randomly

[jira] [Commented] (SPARK-21742) BisectingKMeans generate different models with/without caching

[ https://issues.apache.org/jira/browse/SPARK-21742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128509#comment-16128509 ] zhengruifeng commented on SPARK-21742: -- [~srowen] you are right. When I create the same dataset in a

[jira] [Resolved] (SPARK-13969) Extend input format that feature hashing can handle

[ https://issues.apache.org/jira/browse/SPARK-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-13969. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18513

[jira] [Assigned] (SPARK-13969) Extend input format that feature hashing can handle

[ https://issues.apache.org/jira/browse/SPARK-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-13969: -- Assignee: Nick Pentreath > Extend input format that feature hashing can handle >

  1   2   >