[jira] [Assigned] (SPARK-17736) Update R README for rmarkdown, pandoc

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17736: Assignee: (was: Apache Spark) > Update R README for rmarkdown, pandoc >

[jira] [Commented] (SPARK-17736) Update R README for rmarkdown, pandoc

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535081#comment-15535081 ] Apache Spark commented on SPARK-17736: -- User 'jagadeesanas2' has created a pull request for this

[jira] [Assigned] (SPARK-17736) Update R README for rmarkdown, pandoc

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17736: Assignee: Apache Spark > Update R README for rmarkdown, pandoc >

[jira] [Commented] (SPARK-17717) Add existence checks to user facing catalog

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535062#comment-15535062 ] Apache Spark commented on SPARK-17717: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Updated] (SPARK-17737) cannot import name accumulators error

2016-09-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17737: -- Flags: (was: Important) I suspect it's a problem with how you've got ipython set up to run pyspark.

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535023#comment-15535023 ] Herman van Hovell commented on SPARK-17728: --- I think calling {{explain(true)}} on your plans

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Jacob Eisinger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534923#comment-15534923 ] Jacob Eisinger commented on SPARK-17728: Thanks for the explanation, but I still think this is an

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534910#comment-15534910 ] Xiao Li commented on SPARK-17709: - [~ashrowty]Can you share the exact way how you load the external

[jira] [Commented] (SPARK-17731) Metrics for Structured Streaming

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534905#comment-15534905 ] Apache Spark commented on SPARK-17731: -- User 'tdas' has created a pull request for this issue:

[jira] [Created] (SPARK-17741) Grammar to parse top level and nested data fields separately

2016-09-29 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-17741: --- Summary: Grammar to parse top level and nested data fields separately Key: SPARK-17741 URL: https://issues.apache.org/jira/browse/SPARK-17741 Project: Spark

[jira] [Updated] (SPARK-17697) BinaryLogisticRegressionSummary, GLM Summary should handle non-Double numeric types

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17697: -- Summary: BinaryLogisticRegressionSummary, GLM Summary should handle non-Double numeric

[jira] [Commented] (SPARK-17697) BinaryLogisticRegressionSummary, GLM Summary should handle non-Double numeric types

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534847#comment-15534847 ] Joseph K. Bradley commented on SPARK-17697: --- Leaving open for follow-up to fix all GLM issues

[jira] [Commented] (SPARK-17737) cannot import name accumulators error

2016-09-29 Thread Pruthveej Reddy Kasarla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534839#comment-15534839 ] Pruthveej Reddy Kasarla commented on SPARK-17737: - I am trying to set up spark context. I

[jira] [Closed] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell closed SPARK-17728. - Resolution: Not A Problem > UDFs are run too many times > --- >

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534792#comment-15534792 ] Herman van Hovell commented on SPARK-17728: --- I am going to close this as not a problem, but

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534790#comment-15534790 ] Herman van Hovell commented on SPARK-17728: --- Spark assumes UDF's are pure function; we do not

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534774#comment-15534774 ] Liang-Chi Hsieh commented on SPARK-17556: - Update the design document to add more description for

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Commented] (SPARK-17740) Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534730#comment-15534730 ] Apache Spark commented on SPARK-17740: -- User 'ericl' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17740) Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17740: Assignee: (was: Apache Spark) > Spark tests should mock / interpose HDFS to ensure

[jira] [Assigned] (SPARK-17740) Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17740: Assignee: Apache Spark > Spark tests should mock / interpose HDFS to ensure that streams

[jira] [Created] (SPARK-17740) Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-29 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17740: -- Summary: Spark tests should mock / interpose HDFS to ensure that streams are closed Key: SPARK-17740 URL: https://issues.apache.org/jira/browse/SPARK-17740 Project:

[jira] [Resolved] (SPARK-17717) Add existence checks to user facing catalog

2016-09-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17717. - Resolution: Fixed Fix Version/s: 2.1.0 > Add existence checks to user facing catalog >

[jira] [Comment Edited] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-09-29 Thread Jo Desmet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534603#comment-15534603 ] Jo Desmet edited comment on SPARK-15343 at 9/30/16 12:53 AM: - I think this

[jira] [Comment Edited] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-09-29 Thread Jo Desmet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534603#comment-15534603 ] Jo Desmet edited comment on SPARK-15343 at 9/30/16 12:51 AM: - I think this

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-09-29 Thread Jo Desmet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534607#comment-15534607 ] Jo Desmet commented on SPARK-15343: --- Hadoop Yarn is not 'just' 3rd party. It is an important framework

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-09-29 Thread Jo Desmet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534603#comment-15534603 ] Jo Desmet commented on SPARK-15343: --- I think this issue has not been properly addressed, and should be

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Jacob Eisinger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534583#comment-15534583 ] Jacob Eisinger commented on SPARK-17728: I am a little confused. # Could you explain how a

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Ashish Shrowty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534518#comment-15534518 ] Ashish Shrowty commented on SPARK-17709: Dilip, I tried your code and it works on my end too.

[jira] [Commented] (SPARK-12666) spark-shell --packages cannot load artifacts which are publishLocal'd by SBT

2016-09-29 Thread Alexander Temerev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534475#comment-15534475 ] Alexander Temerev commented on SPARK-12666: --- If you came here for a workaround for 2.0.0 (like

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534434#comment-15534434 ] Dilip Biswal commented on SPARK-17709: -- [~smilegator] Sure. > spark 2.0 join - column resolution

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534429#comment-15534429 ] Xiao Li commented on SPARK-17709: - Can you try it in the latest 2.0? > spark 2.0 join - column

[jira] [Assigned] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17738: Assignee: Apache Spark (was: Davies Liu) > Flaky test:

[jira] [Assigned] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17738: Assignee: Davies Liu (was: Apache Spark) > Flaky test:

[jira] [Commented] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534426#comment-15534426 ] Apache Spark commented on SPARK-17738: -- User 'davies' has created a pull request for this issue:

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534417#comment-15534417 ] Dilip Biswal commented on SPARK-17709: -- [~smilegator] Hi Sean, I tried it on my master branch and

[jira] [Commented] (SPARK-17737) cannot import name accumulators error

2016-09-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534383#comment-15534383 ] Bryan Cutler commented on SPARK-17737: -- What exactly are you trying to do? The recommended way to

[jira] [Commented] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534375#comment-15534375 ] Joseph K. Bradley commented on SPARK-17721: --- OK I did an audit, and this will not have affected

[jira] [Resolved] (SPARK-17412) FsHistoryProviderSuite fails if `root` user runs it

2016-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17412. Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.1.0 >

[jira] [Updated] (SPARK-17412) FsHistoryProviderSuite fails if `root` user runs it

2016-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17412: --- Component/s: Documentation > FsHistoryProviderSuite fails if `root` user runs it >

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-09-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534338#comment-15534338 ] Xiao Li commented on SPARK-17709: - Let me try to reproduce it. Thanks! > spark 2.0 join - column

[jira] [Updated] (SPARK-17739) Collapse adjacent similar Window operators

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-17739: -- Summary: Collapse adjacent similar Window operators (was: Collapse adjacent similar

[jira] [Created] (SPARK-17739) Collapse adjacent similar Window operations.

2016-09-29 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-17739: - Summary: Collapse adjacent similar Window operations. Key: SPARK-17739 URL: https://issues.apache.org/jira/browse/SPARK-17739 Project: Spark Issue

[jira] [Updated] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17721: -- Assignee: Bjarne Fruergaard > Erroneous computation in multiplication of transposed

[jira] [Created] (SPARK-17738) Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract

2016-09-29 Thread Davies Liu (JIRA)
Davies Liu created SPARK-17738: -- Summary: Flaky test: org.apache.spark.sql.execution.columnar.ColumnTypeSuite MAP append/extract Key: SPARK-17738 URL: https://issues.apache.org/jira/browse/SPARK-17738

[jira] [Updated] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17721: -- Fix Version/s: 2.1.0 2.0.2 > Erroneous computation in

[jira] [Resolved] (SPARK-17676) FsHistoryProvider should ignore hidden files

2016-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17676. Resolution: Fixed Fix Version/s: 2.1.0 > FsHistoryProvider should ignore hidden

[jira] [Updated] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17721: -- Target Version/s: 1.5.3, 1.6.3, 2.0.2, 2.1.0 > Erroneous computation in multiplication

[jira] [Updated] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17721: -- Affects Version/s: (was: 1.6.1) (was: 1.4.0)

[jira] [Resolved] (SPARK-17612) Support `DESCRIBE table PARTITION` SQL syntax

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17612. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.1.0

[jira] [Closed] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

2016-09-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-17725. --- Resolution: Later > Spark should not write out parquet files with schema containing non-nullable >

[jira] [Updated] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

2016-09-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17725: Target Version/s: (was: 2.0.1) > Spark should not write out parquet files with schema containing

[jira] [Commented] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

2016-09-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534238#comment-15534238 ] Dongjoon Hyun commented on SPARK-17725: --- After sending a RC4 voting email, I found this issue is

[jira] [Created] (SPARK-17737) cannot import name accumulators error

2016-09-29 Thread Pruthveej Reddy Kasarla (JIRA)
Pruthveej Reddy Kasarla created SPARK-17737: --- Summary: cannot import name accumulators error Key: SPARK-17737 URL: https://issues.apache.org/jira/browse/SPARK-17737 Project: Spark

[jira] [Commented] (SPARK-17549) InMemoryRelation doesn't scale to large tables

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534135#comment-15534135 ] Apache Spark commented on SPARK-17549: -- User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534127#comment-15534127 ] Josh Rosen commented on SPARK-17733: Here's an even simpler test case: {code} sql("""CREATE

[jira] [Created] (SPARK-17736) Update R README for rmarkdown, pandoc

2016-09-29 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-17736: - Summary: Update R README for rmarkdown, pandoc Key: SPARK-17736 URL: https://issues.apache.org/jira/browse/SPARK-17736 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17653. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.1.0

[jira] [Comment Edited] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen edited comment on SPARK-17733 at 9/29/16 9:30 PM: - I managed to

[jira] [Comment Edited] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen edited comment on SPARK-17733 at 9/29/16 9:28 PM: - I managed to

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534082#comment-15534082 ] Josh Rosen commented on SPARK-17733: I managed to shrink to a smaller case which freezes {{explain}}:

[jira] [Commented] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Saif Addin Ellafi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534073#comment-15534073 ] Saif Addin Ellafi commented on SPARK-17735: --- Appreciate. That's exactly what I ended up doing.

[jira] [Commented] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534069#comment-15534069 ] Herman van Hovell commented on SPARK-17735: --- It it better to completely do this on the driver

[jira] [Commented] (SPARK-17728) UDFs are run too many times

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534059#comment-15534059 ] Herman van Hovell commented on SPARK-17728: --- You really should not try to use any external

[jira] [Updated] (SPARK-17718) Update MLib Classification Documentation

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17718: -- Issue Type: Documentation (was: Improvement) > Update MLib Classification

[jira] [Commented] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Saif Addin Ellafi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534046#comment-15534046 ] Saif Addin Ellafi commented on SPARK-17735: --- Hello, thanks I needed to know not only if hive

[jira] [Closed] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell closed SPARK-17735. - Resolution: Not A Problem > Cannot call sqlContext inside udf >

[jira] [Updated] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-17735: -- Description: Hello, I know it is a strange use case but I just wanted to append is

[jira] [Commented] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534040#comment-15534040 ] Herman van Hovell commented on SPARK-17735: --- You really should not use {{sqlContext}} inside a

[jira] [Commented] (SPARK-17721) Erroneous computation in multiplication of transposed SparseMatrix with SparseVector

2016-09-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534041#comment-15534041 ] Joseph K. Bradley commented on SPARK-17721: --- Noting here: We should audit MLlib for uses of

[jira] [Commented] (SPARK-17671) Spark 2.0 history server summary page is slow even set spark.history.ui.maxApplications

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534034#comment-15534034 ] Apache Spark commented on SPARK-17671: -- User 'wgtmac' has created a pull request for this issue:

[jira] [Updated] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Saif Addin Ellafi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saif Addin Ellafi updated SPARK-17735: -- Description: Hello, I know it is a strange use case but I just wanted to append is

[jira] [Created] (SPARK-17735) Cannot call sqlContext inside udf

2016-09-29 Thread Saif Addin Ellafi (JIRA)
Saif Addin Ellafi created SPARK-17735: - Summary: Cannot call sqlContext inside udf Key: SPARK-17735 URL: https://issues.apache.org/jira/browse/SPARK-17735 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533996#comment-15533996 ] Josh Rosen commented on SPARK-17733: Actually, the above log segment wasn't super useful, so let me

[jira] [Commented] (SPARK-17097) Pregel does not keep vertex state properly; fails to terminate

2016-09-29 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533976#comment-15533976 ] ding commented on SPARK-17097: -- Because diff of case class behaves different with regular class. A case

[jira] [Created] (SPARK-17734) inner equi-join shorthand that returns Datasets, like DataFrame already has

2016-09-29 Thread Leif Warner (JIRA)
Leif Warner created SPARK-17734: --- Summary: inner equi-join shorthand that returns Datasets, like DataFrame already has Key: SPARK-17734 URL: https://issues.apache.org/jira/browse/SPARK-17734 Project:

[jira] [Updated] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17733: --- Attachment: constraints.png > InferFiltersFromConstraints rule never terminates for query >

[jira] [Updated] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17733: --- Attachment:

[jira] [Created] (SPARK-17733) InferFiltersFromConstraints rule never terminates for query

2016-09-29 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-17733: -- Summary: InferFiltersFromConstraints rule never terminates for query Key: SPARK-17733 URL: https://issues.apache.org/jira/browse/SPARK-17733 Project: Spark

[jira] [Commented] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533935#comment-15533935 ] Apache Spark commented on SPARK-17732: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17732: Assignee: (was: Apache Spark) > ALTER TABLE DROP PARTITION should support comparators

[jira] [Assigned] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

2016-09-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17732: Assignee: Apache Spark > ALTER TABLE DROP PARTITION should support comparators >

[jira] [Created] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

2016-09-29 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-17732: - Summary: ALTER TABLE DROP PARTITION should support comparators Key: SPARK-17732 URL: https://issues.apache.org/jira/browse/SPARK-17732 Project: Spark

[jira] [Resolved] (SPARK-17699) from_json function for parsing json Strings into Structs

2016-09-29 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-17699. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15274

[jira] [Resolved] (SPARK-17715) Log INFO per task launch creates a large driver log

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17715. --- Resolution: Fixed Assignee: Brian Cho Fix Version/s: 2.1.0 Target

[jira] [Updated] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-17672: -- Assignee: Gang Wu > Spark 2.0 history server web Ui takes too long for a single application >

[jira] [Resolved] (SPARK-17672) Spark 2.0 history server web Ui takes too long for a single application

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17672. --- Resolution: Fixed Fix Version/s: 2.0.1 Target Version/s: 2.0.1 > Spark 2.0 history

[jira] [Assigned] (SPARK-17731) Metrics for Structured Streaming

2016-09-29 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-17731: - Assignee: Tathagata Das > Metrics for Structured Streaming >

[jira] [Created] (SPARK-17731) Metrics for Structured Streaming

2016-09-29 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-17731: - Summary: Metrics for Structured Streaming Key: SPARK-17731 URL: https://issues.apache.org/jira/browse/SPARK-17731 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

2016-09-29 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-17648. --- Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 >

[jira] [Updated] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17666: --- Target Version/s: 2.0.1, 2.1.0 (was: 2.0.2, 2.1.0) > take() or isEmpty() on dataset leaks s3a

[jira] [Assigned] (SPARK-17666) take() or isEmpty() on dataset leaks s3a connections

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-17666: -- Assignee: Josh Rosen > take() or isEmpty() on dataset leaks s3a connections >

[jira] [Updated] (SPARK-17712) Incorrect result due to invalid pushdown of data-independent filter beneath aggregate

2016-09-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-17712: --- Fix Version/s: 2.0.2 > Incorrect result due to invalid pushdown of data-independent filter beneath

  1   2   3   >