[jira] [Commented] (SPARK-16683) Group by does not work after multiple joins of the same dataframe

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095790#comment-16095790 ] Apache Spark commented on SPARK-16683: -- User 'aray' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16683) Group by does not work after multiple joins of the same dataframe

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16683: Assignee: (was: Apache Spark) > Group by does not work after multiple joins of the

[jira] [Assigned] (SPARK-16683) Group by does not work after multiple joins of the same dataframe

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16683: Assignee: Apache Spark > Group by does not work after multiple joins of the same

[jira] [Assigned] (SPARK-21497) Pull non-deterministic joining keys from Join operator

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21497: Assignee: Apache Spark > Pull non-deterministic joining keys from Join operator >

[jira] [Assigned] (SPARK-21497) Pull non-deterministic joining keys from Join operator

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21497: Assignee: (was: Apache Spark) > Pull non-deterministic joining keys from Join

[jira] [Commented] (SPARK-21497) Pull non-deterministic joining keys from Join operator

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095756#comment-16095756 ] Apache Spark commented on SPARK-21497: -- User 'viirya' has created a pull request for this issue:

[jira] [Updated] (SPARK-21497) Pull non-deterministic joining keys from Join operator

2017-07-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-21497: Description: Currently SparkSQL doesn't support non-deterministic joining conditions in

[jira] [Created] (SPARK-21497) Pull non-deterministic joining keys from Join operator

2017-07-20 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-21497: --- Summary: Pull non-deterministic joining keys from Join operator Key: SPARK-21497 URL: https://issues.apache.org/jira/browse/SPARK-21497 Project: Spark

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095744#comment-16095744 ] Shixiong Zhu commented on SPARK-21425: -- [~rdub] I just realized we never document local-cluster

[jira] [Updated] (SPARK-21495) DIGEST-MD5: Out of order sequencing of messages from server

2017-07-20 Thread Xin Yu Pan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Yu Pan updated SPARK-21495: --- Description: We hit an issue when enabling authentication and Sasl encryption, see bold font in

[jira] [Updated] (SPARK-21495) DIGEST-MD5: Out of order sequencing of messages from server

2017-07-20 Thread Xin Yu Pan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Yu Pan updated SPARK-21495: --- Description: We hit an issue when enabling authentication and Sasl encryption, see bold font in

[jira] [Updated] (SPARK-21495) DIGEST-MD5: Out of order sequencing of messages from server

2017-07-20 Thread Xin Yu Pan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Yu Pan updated SPARK-21495: --- Description: We hit an issue when enabling authentication and Sasl encryption, see bold font in

[jira] [Created] (SPARK-21496) Support codegen for TakeOrderedAndProjectExec

2017-07-20 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-21496: Summary: Support codegen for TakeOrderedAndProjectExec Key: SPARK-21496 URL: https://issues.apache.org/jira/browse/SPARK-21496 Project: Spark Issue Type:

[jira] [Created] (SPARK-21495) DIGEST-MD5: Out of order sequencing of messages from server

2017-07-20 Thread Xin Yu Pan (JIRA)
Xin Yu Pan created SPARK-21495: -- Summary: DIGEST-MD5: Out of order sequencing of messages from server Key: SPARK-21495 URL: https://issues.apache.org/jira/browse/SPARK-21495 Project: Spark

[jira] [Comment Edited] (SPARK-21486) Fail when using aliased column of a aliased table from a subquery

2017-07-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095689#comment-16095689 ] Liang-Chi Hsieh edited comment on SPARK-21486 at 7/21/17 2:34 AM: -- Since

[jira] [Commented] (SPARK-21486) Fail when using aliased column of a aliased table from a subquery

2017-07-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095689#comment-16095689 ] Liang-Chi Hsieh commented on SPARK-21486: - Since 2.2.0, it is not allowed to use the qualifier

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095678#comment-16095678 ] Ryan Williams commented on SPARK-21425: --- [~zsxwing] yea, it's static accumulators, and seems to

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090017#comment-16090017 ] Ryan Williams edited comment on SPARK-21425 at 7/21/17 2:12 AM: Yea,

[jira] [Updated] (SPARK-20960) make ColumnVector public

2017-07-20 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Jun updated SPARK-20960: - Description: ColumnVector is an internal interface in Spark SQL, which is only used for vectorized

[jira] [Updated] (SPARK-20960) make ColumnVector public

2017-07-20 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Song Jun updated SPARK-20960: - Description: _emphasized text_ColumnVector is an internal interface in Spark SQL, which is only used

[jira] [Commented] (SPARK-21485) API Documentation for Spark SQL functions

2017-07-20 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095600#comment-16095600 ] Reynold Xin commented on SPARK-21485: - Pretty cool. Would be great to just generate the function list

[jira] [Comment Edited] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095587#comment-16095587 ] Iurii Antykhovych edited comment on SPARK-21491 at 7/21/17 12:30 AM: -

[jira] [Updated] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

2017-07-20 Thread Udit Mehrotra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra updated SPARK-21494: -- Attachment: logs.zip > Spark 2.2.0 AES encryption not working with External shuffle >

[jira] [Updated] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

2017-07-20 Thread Udit Mehrotra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra updated SPARK-21494: -- Description: Spark’s new AES based authentication mechanism does not seem to work when

[jira] [Created] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

2017-07-20 Thread Udit Mehrotra (JIRA)
Udit Mehrotra created SPARK-21494: - Summary: Spark 2.2.0 AES encryption not working with External shuffle Key: SPARK-21494 URL: https://issues.apache.org/jira/browse/SPARK-21494 Project: Spark

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095587#comment-16095587 ] Iurii Antykhovych commented on SPARK-21491: --- I searched for all such places in the whole GraphX

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095546#comment-16095546 ] Thomas Graves commented on SPARK-21460: --- I didn't think that was the case, but took a look at the

[jira] [Commented] (SPARK-21493) Add more metrics to External Shuffle Service

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095466#comment-16095466 ] Sean Owen commented on SPARK-21493: --- How is this different from SPARK-21334 that you opened? > Add

[jira] [Created] (SPARK-21493) Add more metrics to External Shuffle Service

2017-07-20 Thread Raajay Viswanathan (JIRA)
Raajay Viswanathan created SPARK-21493: -- Summary: Add more metrics to External Shuffle Service Key: SPARK-21493 URL: https://issues.apache.org/jira/browse/SPARK-21493 Project: Spark

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095447#comment-16095447 ] Saisai Shao commented on SPARK-21460: - I think this is basically a ListenerBus issue, not a dynamic

[jira] [Assigned] (SPARK-21490) SparkLauncher may fail to redirect streams

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21490: Assignee: Apache Spark > SparkLauncher may fail to redirect streams >

[jira] [Commented] (SPARK-21490) SparkLauncher may fail to redirect streams

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095445#comment-16095445 ] Apache Spark commented on SPARK-21490: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21490) SparkLauncher may fail to redirect streams

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21490: Assignee: (was: Apache Spark) > SparkLauncher may fail to redirect streams >

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095432#comment-16095432 ] Apache Spark commented on SPARK-12717: -- User 'BryanCutler' has created a pull request for this

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

2017-07-20 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095425#comment-16095425 ] Zhan Zhang commented on SPARK-21492: root cause: In the SortMergeJoin, inner/leftOuter/rightOuter,

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095418#comment-16095418 ] Apache Spark commented on SPARK-21492: -- User 'zhzhan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21492) Memory leak in SortMergeJoin

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21492: Assignee: (was: Apache Spark) > Memory leak in SortMergeJoin >

[jira] [Assigned] (SPARK-21492) Memory leak in SortMergeJoin

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21492: Assignee: Apache Spark > Memory leak in SortMergeJoin > > >

[jira] [Created] (SPARK-21492) Memory leak in SortMergeJoin

2017-07-20 Thread Zhan Zhang (JIRA)
Zhan Zhang created SPARK-21492: -- Summary: Memory leak in SortMergeJoin Key: SPARK-21492 URL: https://issues.apache.org/jira/browse/SPARK-21492 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095413#comment-16095413 ] Sean Owen commented on SPARK-21491: --- I see it now, yeah, and it's used in one place in Spark: {code}

[jira] [Comment Edited] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095398#comment-16095398 ] Iurii Antykhovych edited comment on SPARK-21491 at 7/20/17 9:22 PM:

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095398#comment-16095398 ] Iurii Antykhovych commented on SPARK-21491: --- This is relevant to all scala versions starting

[jira] [Commented] (SPARK-21489) Update release docs to point out Python 2.6 support is removed.

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095386#comment-16095386 ] Sean Owen commented on SPARK-21489: --- I think that was just changed:

[jira] [Assigned] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21491: Assignee: (was: Apache Spark) > Performance enhancement: eliminate creation of

[jira] [Assigned] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21491: Assignee: Apache Spark > Performance enhancement: eliminate creation of intermediate

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095378#comment-16095378 ] Apache Spark commented on SPARK-21491: -- User 'SereneAnt' has created a pull request for this issue:

[jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095379#comment-16095379 ] Sean Owen commented on SPARK-21491: --- I can't even find this in Scala. The article talks about 2.8. Is

[jira] [Updated] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Antykhovych updated SPARK-21491: -- Priority: Trivial (was: Minor) > Performance enhancement: eliminate creation of

[jira] [Created] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections

2017-07-20 Thread Iurii Antykhovych (JIRA)
Iurii Antykhovych created SPARK-21491: - Summary: Performance enhancement: eliminate creation of intermediate collections Key: SPARK-21491 URL: https://issues.apache.org/jira/browse/SPARK-21491

[jira] [Created] (SPARK-21490) SparkLauncher may fail to redirect streams

2017-07-20 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-21490: -- Summary: SparkLauncher may fail to redirect streams Key: SPARK-21490 URL: https://issues.apache.org/jira/browse/SPARK-21490 Project: Spark Issue Type:

[jira] [Created] (SPARK-21489) Update release docs to point out Python 2.6 support is removed.

2017-07-20 Thread holdenk (JIRA)
holdenk created SPARK-21489: --- Summary: Update release docs to point out Python 2.6 support is removed. Key: SPARK-21489 URL: https://issues.apache.org/jira/browse/SPARK-21489 Project: Spark Issue

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095323#comment-16095323 ] Shixiong Zhu edited comment on SPARK-21425 at 7/20/17 8:45 PM: --- [~srowen]

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095323#comment-16095323 ] Shixiong Zhu edited comment on SPARK-21425 at 7/20/17 8:38 PM: --- [~srowen]

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095323#comment-16095323 ] Shixiong Zhu commented on SPARK-21425: -- [~srowen] The issue is static accumulators. Right? They

[jira] [Assigned] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21417: Assignee: (was: Apache Spark) > Detect transitive join conditions via expressions >

[jira] [Assigned] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21417: Assignee: Apache Spark > Detect transitive join conditions via expressions >

[jira] [Commented] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095317#comment-16095317 ] Apache Spark commented on SPARK-21417: -- User 'aokolnychyi' has created a pull request for this

[jira] [Commented] (SPARK-7146) Should ML sharedParams be a public API?

2017-07-20 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095299#comment-16095299 ] holdenk commented on SPARK-7146: So it seems like there is a (more recent) agreement that exposing this as

[jira] [Comment Edited] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-07-20 Thread Roberto Mirizzi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095275#comment-16095275 ] Roberto Mirizzi edited comment on SPARK-21478 at 7/20/17 8:06 PM: -- Sorry

[jira] [Commented] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-07-20 Thread Roberto Mirizzi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095275#comment-16095275 ] Roberto Mirizzi commented on SPARK-21478: - Sorry about that. I totally misunderstood you. :-) >

[jira] [Commented] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095208#comment-16095208 ] Sean Owen commented on SPARK-21478: --- I said I can reproduce it, I agree with you. > Unpersist a DF

[jira] [Commented] (SPARK-19908) Direct buffer memory OOM should not cause stage retries.

2017-07-20 Thread Kaushal Prajapati (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095203#comment-16095203 ] Kaushal Prajapati commented on SPARK-19908: --- [~bojanbabic] This error comes when your

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Raajay Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095204#comment-16095204 ] Raajay Viswanathan commented on SPARK-21334: I think SPARK-18364 aims to implement metrics in

[jira] [Commented] (SPARK-21243) Limit the number of maps in a single shuffle fetch

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095199#comment-16095199 ] Apache Spark commented on SPARK-21243: -- User 'dhruve' has created a pull request for this issue:

[jira] [Commented] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-07-20 Thread Roberto Mirizzi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095187#comment-16095187 ] Roberto Mirizzi commented on SPARK-21478: - That's weird you are not able to reproduce it. Did

[jira] [Updated] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

2017-07-20 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-21488: -- Description: It would be great to make saveAsTable() return dataframe of created

[jira] [Updated] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

2017-07-20 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-21488: -- Summary: Make saveAsTable() and createOrReplaceTempView() return dataframe of created

[jira] [Created] (SPARK-21488) Make saveAsTable() return dataframe of created table

2017-07-20 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-21488: - Summary: Make saveAsTable() return dataframe of created table Key: SPARK-21488 URL: https://issues.apache.org/jira/browse/SPARK-21488 Project: Spark

[jira] [Resolved] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21463. -- Resolution: Fixed Fix Version/s: 2.3.0 > Output of StructuredStreaming tables don't

[jira] [Updated] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-07-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21478: - Component/s: (was: Spark Core) SQL > Unpersist a DF also unpersists related

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-20 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095131#comment-16095131 ] Ruslan Dautkhanov commented on SPARK-21460: --- [~Dhruve Ashar], I can email logs to you. Although

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095095#comment-16095095 ] Robert Kruszewski commented on SPARK-21334: --- I think this is a dupe of

[jira] [Assigned] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21334: Assignee: Apache Spark > Fix metrics for external shuffle service >

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095063#comment-16095063 ] Apache Spark commented on SPARK-21334: -- User 'raajay' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21334: Assignee: (was: Apache Spark) > Fix metrics for external shuffle service >

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Raajay Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095056#comment-16095056 ] Raajay Viswanathan commented on SPARK-21334: [~jerryshao] I am using external shuffle service

[jira] [Updated] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Raajay Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raajay Viswanathan updated SPARK-21334: --- Description: SPARK-16405 introduced metrics for external shuffle service. However,

[jira] [Updated] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Raajay Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raajay Viswanathan updated SPARK-21334: --- Description: SPARK-16405 introduced metrics for external shuffle service. However,

[jira] [Updated] (SPARK-21334) Fix metrics for external shuffle service

2017-07-20 Thread Raajay Viswanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raajay Viswanathan updated SPARK-21334: --- Description: SPARK-16405 introduced metrics for external shuffle service. However,

[jira] [Resolved] (SPARK-21142) spark-streaming-kafka-0-10 has too fat dependency on kafka

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21142. --- Resolution: Fixed > spark-streaming-kafka-0-10 has too fat dependency on kafka >

[jira] [Assigned] (SPARK-21142) spark-streaming-kafka-0-10 has too fat dependency on kafka

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21142: - Assignee: Tim Van Wassenhove Fix Version/s: 2.3.0 Issue Type: Improvement (was:

[jira] [Resolved] (SPARK-19531) History server doesn't refresh jobs for long-life apps like thriftserver

2017-07-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19531. Resolution: Fixed Assignee: Oleg Danilov Fix Version/s: 2.3.0 > History

[jira] [Resolved] (SPARK-20394) Replication factor value Not changing properly

2017-07-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20394. Resolution: Workaround Great. I doubt we'll fix this in 1.6 at this point, and I believe

[jira] [Comment Edited] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Claus Stadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094844#comment-16094844 ] Claus Stadler edited comment on SPARK-21417 at 7/20/17 3:41 PM: Hi Anton,

[jira] [Comment Edited] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Claus Stadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094844#comment-16094844 ] Claus Stadler edited comment on SPARK-21417 at 7/20/17 3:40 PM: Hi Anton,

[jira] [Commented] (SPARK-21417) Detect transitive join conditions via expressions

2017-07-20 Thread Claus Stadler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094844#comment-16094844 ] Claus Stadler commented on SPARK-21417: --- Hi Anton, I have a rough idea how the issue could be

[jira] [Commented] (SPARK-15544) Bouncing Zookeeper node causes Active spark master to exit

2017-07-20 Thread David Kats (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094797#comment-16094797 ] David Kats commented on SPARK-15544: Confirming the same issue with Spark 2.1.0 and 2.2.0, ubuntu

[jira] [Commented] (SPARK-21487) WebUI-Executors Page results in "Request is a replay (34) attack"

2017-07-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094794#comment-16094794 ] Sean Owen commented on SPARK-21487: --- I think this is a YARN issue/question. > WebUI-Executors Page

[jira] [Updated] (SPARK-21487) WebUI-Executors Page results in "Request is a replay (34) attack"

2017-07-20 Thread lishuming (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lishuming updated SPARK-21487: -- Description: We upgraded Spark version from 2.0.2 to 2.1.1 recently, WebUI `Executors Page` becomed

[jira] [Created] (SPARK-21487) WebUI-Executors Page results in "Request is a replay (34) attack"

2017-07-20 Thread lishuming (JIRA)
lishuming created SPARK-21487: - Summary: WebUI-Executors Page results in "Request is a replay (34) attack" Key: SPARK-21487 URL: https://issues.apache.org/jira/browse/SPARK-21487 Project: Spark

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2017-07-20 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094689#comment-16094689 ] Dhruve Ashar commented on SPARK-21460: -- [~Tagar] Can you attach the driver logs so that it helps in

[jira] [Commented] (SPARK-20394) Replication factor value Not changing properly

2017-07-20 Thread Kannan Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094686#comment-16094686 ] Kannan Subramanian commented on SPARK-20394: Yes. I have edited the hdfs-site.xml for

[jira] [Assigned] (SPARK-21472) Introduce ArrowColumnVector as a reader for Arrow vectors.

2017-07-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21472: --- Assignee: Takuya Ueshin > Introduce ArrowColumnVector as a reader for Arrow vectors. >

[jira] [Resolved] (SPARK-21472) Introduce ArrowColumnVector as a reader for Arrow vectors.

2017-07-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21472. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18680

[jira] [Commented] (SPARK-10063) Remove DirectParquetOutputCommitter

2017-07-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094610#comment-16094610 ] Apache Spark commented on SPARK-10063: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-21483) Make org.apache.spark.ml.linalg.Vector bean-compliant so it can be used in Encoders.bean(Vector.class)

2017-07-20 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094607#comment-16094607 ] Aseem Bansal edited comment on SPARK-21483 at 7/20/17 12:29 PM: Some

[jira] [Commented] (SPARK-21483) Make org.apache.spark.ml.linalg.Vector bean-compliant so it can be used in Encoders.bean(Vector.class)

2017-07-20 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094607#comment-16094607 ] Aseem Bansal commented on SPARK-21483: -- Some pseudo code to show what I am trying to achieve

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

2017-07-20 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094603#comment-16094603 ] Peng Meng commented on SPARK-21476: --- I am optimizing RF and GBT these days, if no one works on it. I

[jira] [Commented] (SPARK-19842) Informational Referential Integrity Constraints Support in Spark

2017-07-20 Thread Ioana Delaney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094547#comment-16094547 ] Ioana Delaney commented on SPARK-19842: --- Yes, we've been working to productize our code. Our

[jira] [Assigned] (SPARK-21477) Mark LocalTableScanExec's input data transient

2017-07-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21477: --- Assignee: Xiao Li > Mark LocalTableScanExec's input data transient >

  1   2   >