[jira] [Commented] (SPARK-21045) Spark executor blocked instead of throwing exception because exception occur when python worker send exception info to PythonRDD in Python 2+

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051485#comment-16051485 ] Apache Spark commented on SPARK-21045: -- User 'dataknocker' has created a pull reques

[jira] [Created] (SPARK-21118) OOM with 2 handred million vertex when mitrx multply

2017-06-15 Thread tao (JIRA)
tao created SPARK-21118: --- Summary: OOM with 2 handred million vertex when mitrx multply Key: SPARK-21118 URL: https://issues.apache.org/jira/browse/SPARK-21118 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21045) Spark executor blocked instead of throwing exception because exception occur when python worker send exception info to PythonRDD in Python 2+

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051471#comment-16051471 ] Apache Spark commented on SPARK-21045: -- User 'dataknocker' has created a pull reques

[jira] [Commented] (SPARK-21117) Built-in SQL Function Support - WIDTH_BUCKET

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051457#comment-16051457 ] Apache Spark commented on SPARK-21117: -- User 'wangyum' has created a pull request fo

[jira] [Assigned] (SPARK-21117) Built-in SQL Function Support - WIDTH_BUCKET

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21117: Assignee: Apache Spark > Built-in SQL Function Support - WIDTH_BUCKET > --

[jira] [Assigned] (SPARK-21117) Built-in SQL Function Support - WIDTH_BUCKET

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21117: Assignee: (was: Apache Spark) > Built-in SQL Function Support - WIDTH_BUCKET > ---

[jira] [Created] (SPARK-21117) Built-in SQL Function Support - WIDTH_BUCKET

2017-06-15 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21117: --- Summary: Built-in SQL Function Support - WIDTH_BUCKET Key: SPARK-21117 URL: https://issues.apache.org/jira/browse/SPARK-21117 Project: Spark Issue Type: Sub-ta

[jira] [Closed] (SPARK-20752) Build-in SQL Function Support - SQRT

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-20752. --- Resolution: Duplicate > Build-in SQL Function Support - SQRT > > >

[jira] [Resolved] (SPARK-20750) Built-in SQL Function Support - REPLACE

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20750. - Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.3.0 > Built-in SQL Function

[jira] [Commented] (SPARK-20752) Build-in SQL Function Support - SQRT

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051441#comment-16051441 ] Xiao Li commented on SPARK-20752: - Yes! > Build-in SQL Function Support - SQRT > ---

[jira] [Resolved] (SPARK-20749) Built-in SQL Function Support - all variants of LEN[GTH]

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-20749. - Resolution: Fixed Fix Version/s: 2.3.0 > Built-in SQL Function Support - all variants of LEN[GTH]

[jira] [Assigned] (SPARK-20749) Built-in SQL Function Support - all variants of LEN[GTH]

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-20749: --- Assignee: Kazuaki Ishizaki > Built-in SQL Function Support - all variants of LEN[GTH] >

[jira] [Created] (SPARK-21116) Support MapKeyContains function

2017-06-15 Thread darion yaphet (JIRA)
darion yaphet created SPARK-21116: - Summary: Support MapKeyContains function Key: SPARK-21116 URL: https://issues.apache.org/jira/browse/SPARK-21116 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-21115) If the cores left is less than the coresPerExecutor,the cores left will not be allocated, so it should not to check in every schedule

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21115: Assignee: Apache Spark > If the cores left is less than the coresPerExecutor,the cores lef

[jira] [Commented] (SPARK-21115) If the cores left is less than the coresPerExecutor,the cores left will not be allocated, so it should not to check in every schedule

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051427#comment-16051427 ] Apache Spark commented on SPARK-21115: -- User 'eatoncys' has created a pull request f

[jira] [Assigned] (SPARK-21115) If the cores left is less than the coresPerExecutor,the cores left will not be allocated, so it should not to check in every schedule

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21115: Assignee: (was: Apache Spark) > If the cores left is less than the coresPerExecutor,th

[jira] [Created] (SPARK-21115) If the cores left is less than the coresPerExecutor,the cores left will not be allocated, so it should not to check in every schedule

2017-06-15 Thread eaton (JIRA)
eaton created SPARK-21115: - Summary: If the cores left is less than the coresPerExecutor,the cores left will not be allocated, so it should not to check in every schedule Key: SPARK-21115 URL: https://issues.apache.org/ji

[jira] [Commented] (SPARK-21104) Support sort with index when parse LibSVM Record

2017-06-15 Thread darion yaphet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051388#comment-16051388 ] darion yaphet commented on SPARK-21104: --- we should make sure the array is in ordere

[jira] [Resolved] (SPARK-21114) Test failure in Spark 2.1 due to name mismatch

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21114. - Resolution: Fixed Fix Version/s: 2.1.2 > Test failure in Spark 2.1 due to name mismatch >

[jira] [Resolved] (SPARK-21072) `TreeNode.mapChildren` should only apply to the children node.

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21072. - Resolution: Fixed Assignee: coneyliu Fix Version/s: 2.2.0 2.1.2

[jira] [Commented] (SPARK-12552) Recovered driver's resource is not counted in the Master

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051322#comment-16051322 ] Apache Spark commented on SPARK-12552: -- User 'jerryshao' has created a pull request

[jira] [Assigned] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21093: Assignee: (was: Apache Spark) > Multiple gapply execution occasionally failed in Spark

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051308#comment-16051308 ] Apache Spark commented on SPARK-21093: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21093: Assignee: Apache Spark > Multiple gapply execution occasionally failed in SparkR > --

[jira] [Resolved] (SPARK-21112) ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21112. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18318 [https://githu

[jira] [Updated] (SPARK-21114) Test failure in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21114: Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.1-test-maven-

[jira] [Updated] (SPARK-21114) Test failure in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21114: Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.1-test-maven-

[jira] [Updated] (SPARK-21114) Test failure in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21114: Affects Version/s: 2.0.2 > Test failure in Spark 2.1 due to name mismatch > ---

[jira] [Updated] (SPARK-21114) Test failure in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21114: Summary: Test failure in Spark 2.1 due to name mismatch (was: Test failure fix in Spark 2.1 due to name mi

[jira] [Assigned] (SPARK-21114) Test failure fix in Spark 2.1 due to name mismatch

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21114: Assignee: Xiao Li (was: Apache Spark) > Test failure fix in Spark 2.1 due to name mismatc

[jira] [Assigned] (SPARK-21114) Test failure fix in Spark 2.1 due to name mismatch

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21114: Assignee: Apache Spark (was: Xiao Li) > Test failure fix in Spark 2.1 due to name mismatc

[jira] [Commented] (SPARK-21114) Test failure fix in Spark 2.1 due to name mismatch

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051290#comment-16051290 ] Apache Spark commented on SPARK-21114: -- User 'gatorsmile' has created a pull request

[jira] [Created] (SPARK-21114) Test failure fix in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21114: --- Summary: Test failure fix in Spark 2.1 due to name mismatch Key: SPARK-21114 URL: https://issues.apache.org/jira/browse/SPARK-21114 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-21114) Test failure fix in Spark 2.1 due to name mismatch

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21114: Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.1-test-maven-

[jira] [Commented] (SPARK-17237) DataFrame fill after pivot causing org.apache.spark.sql.AnalysisException

2017-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051288#comment-16051288 ] Takeshi Yamamuro commented on SPARK-17237: -- I checked the other behaviours and I

[jira] [Resolved] (SPARK-21111) Fix test failure in 2.2

2017-06-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-2. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 18316 [https://github.com/

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051264#comment-16051264 ] Hyukjin Kwon commented on SPARK-21093: -- BTW, {{mcfork}} in R looks opening a pipe ah

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051244#comment-16051244 ] Hyukjin Kwon commented on SPARK-21093: -- This does disappear in a certain condition w

[jira] [Resolved] (SPARK-21096) Pickle error when passing a member variable to Spark executors

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21096. -- Resolution: Not A Problem That's not an issue in Spark but maybe cloudpickle or Python. I wond

[jira] [Assigned] (SPARK-21112) ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21112: Assignee: Xiao Li (was: Apache Spark) > ALTER TABLE SET TBLPROPERTIES should not overwrit

[jira] [Commented] (SPARK-21112) ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051140#comment-16051140 ] Apache Spark commented on SPARK-21112: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-21112) ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21112: Assignee: Apache Spark (was: Xiao Li) > ALTER TABLE SET TBLPROPERTIES should not overwrit

[jira] [Assigned] (SPARK-21113) Support for read ahead input stream to amortize disk IO cost in the Spill reader

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21113: Assignee: (was: Apache Spark) > Support for read ahead input stream to amortize disk I

[jira] [Commented] (SPARK-21113) Support for read ahead input stream to amortize disk IO cost in the Spill reader

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051134#comment-16051134 ] Apache Spark commented on SPARK-21113: -- User 'sitalkedia' has created a pull request

[jira] [Assigned] (SPARK-21113) Support for read ahead input stream to amortize disk IO cost in the Spill reader

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21113: Assignee: Apache Spark > Support for read ahead input stream to amortize disk IO cost in t

[jira] [Created] (SPARK-21112) ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT

2017-06-15 Thread Xiao Li (JIRA)
Xiao Li created SPARK-21112: --- Summary: ALTER TABLE SET TBLPROPERTIES should not overwrite COMMENT Key: SPARK-21112 URL: https://issues.apache.org/jira/browse/SPARK-21112 Project: Spark Issue Type:

[jira] [Created] (SPARK-21113) Support for read ahead input stream to amortize disk IO cost in the Spill reader

2017-06-15 Thread Sital Kedia (JIRA)
Sital Kedia created SPARK-21113: --- Summary: Support for read ahead input stream to amortize disk IO cost in the Spill reader Key: SPARK-21113 URL: https://issues.apache.org/jira/browse/SPARK-21113 Projec

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051109#comment-16051109 ] Marcelo Vanzin commented on SPARK-18838: bq. If we're careful in performance opti

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051097#comment-16051097 ] Marcelo Vanzin commented on SPARK-18838: [~sitalke...@gmail.com] I requested your

[jira] [Commented] (SPARK-21111) Fix test failure in 2.2

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051095#comment-16051095 ] Apache Spark commented on SPARK-2: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-21111) Fix test failure in 2.2

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: Xiao Li (was: Apache Spark) > Fix test failure in 2.2 > --

[jira] [Assigned] (SPARK-21111) Fix test failure in 2.2

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: Apache Spark (was: Xiao Li) > Fix test failure in 2.2 > --

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051094#comment-16051094 ] Shivaram Venkataraman commented on SPARK-21093: --- Thanks [~hyukjin.kwon] --

[jira] [Commented] (SPARK-21025) missing data in jsc.union

2017-06-15 Thread meng xi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051090#comment-16051090 ] meng xi commented on SPARK-21025: - I create a new list for each rdd and it works, thanks!

[jira] [Created] (SPARK-21111) Fix test failure in 2.2

2017-06-15 Thread Xiao Li (JIRA)
Xiao Li created SPARK-2: --- Summary: Fix test failure in 2.2 Key: SPARK-2 URL: https://issues.apache.org/jira/browse/SPARK-2 Project: Spark Issue Type: Test Components: SQL

[jira] [Commented] (SPARK-21101) Error running Hive temporary UDTF on latest Spark 2.2

2017-06-15 Thread Dayou Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051017#comment-16051017 ] Dayou Zhou commented on SPARK-21101: Hi [~srowen], thanks for the helpful and constru

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2017-06-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051013#comment-16051013 ] Reynold Xin commented on SPARK-1: - But this ticket has nothing to do with SQL? >

[jira] [Comment Edited] (SPARK-20937) Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide

2017-06-15 Thread Neil Parker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050999#comment-16050999 ] Neil Parker edited comment on SPARK-20937 at 6/15/17 8:11 PM: -

[jira] [Commented] (SPARK-20937) Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide

2017-06-15 Thread Neil Parker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050999#comment-16050999 ] Neil Parker commented on SPARK-20937: - +1 I ran into the issue recently and took me a

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2017-06-15 Thread Taklon Stephen Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050993#comment-16050993 ] Taklon Stephen Wu commented on SPARK-19490: --- I pinged in the PR, but I didn't g

[jira] [Updated] (SPARK-21110) Structs should be usable in inequality filters

2017-06-15 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-21110: - Summary: Structs should be usable in inequality filters (was: Structs should be orderabl

[jira] [Created] (SPARK-21110) Structs should be orderable

2017-06-15 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-21110: Summary: Structs should be orderable Key: SPARK-21110 URL: https://issues.apache.org/jira/browse/SPARK-21110 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-21109) union two dataset[A] don't work as expected if one of the datasets is originated from a dataframe

2017-06-15 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry Lam updated SPARK-21109: -- Description: To reproduce the issue: {code} case class my_case(id0: Long, id1: Int, id2: Int, id3: Stri

[jira] [Updated] (SPARK-21109) union two dataset[A] don't work as expected if one of the datasets is originated from a dataframe

2017-06-15 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry Lam updated SPARK-21109: -- Description: To reproduce the issue: {code} case class my_case(id0: Long, id1: Int, id2: Int, id3: Stri

[jira] [Comment Edited] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2017-06-15 Thread Serge Vilvovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050940#comment-16050940 ] Serge Vilvovsky edited comment on SPARK-18649 at 6/15/17 6:57 PM: -

[jira] [Created] (SPARK-21109) union two dataset[A] don't work as expected if one of the datasets is originated from a dataframe

2017-06-15 Thread Jerry Lam (JIRA)
Jerry Lam created SPARK-21109: - Summary: union two dataset[A] don't work as expected if one of the datasets is originated from a dataframe Key: SPARK-21109 URL: https://issues.apache.org/jira/browse/SPARK-21109

[jira] [Comment Edited] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2017-06-15 Thread Serge Vilvovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050940#comment-16050940 ] Serge Vilvovsky edited comment on SPARK-18649 at 6/15/17 6:57 PM: -

[jira] [Commented] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2017-06-15 Thread Serge Vilvovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050940#comment-16050940 ] Serge Vilvovsky commented on SPARK-18649: - Does anybody work on the issue? Same p

[jira] [Resolved] (SPARK-20434) Move Hadoop delegation token code from yarn to core

2017-06-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20434. Resolution: Fixed Assignee: Michael Gummelt Fix Version/s: 2.3.0 > Move Had

[jira] [Assigned] (SPARK-21108) convert LinearSVC to aggregator framework

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21108: Assignee: Apache Spark > convert LinearSVC to aggregator framework > -

[jira] [Commented] (SPARK-21108) convert LinearSVC to aggregator framework

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050931#comment-16050931 ] Apache Spark commented on SPARK-21108: -- User 'hhbyyh' has created a pull request for

[jira] [Assigned] (SPARK-21108) convert LinearSVC to aggregator framework

2017-06-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21108: Assignee: (was: Apache Spark) > convert LinearSVC to aggregator framework > --

[jira] [Created] (SPARK-21108) convert LinearSVC to aggregator framework

2017-06-15 Thread yuhao yang (JIRA)
yuhao yang created SPARK-21108: -- Summary: convert LinearSVC to aggregator framework Key: SPARK-21108 URL: https://issues.apache.org/jira/browse/SPARK-21108 Project: Spark Issue Type: Sub-task

[jira] [Comment Edited] (SPARK-21081) Throw specific IllegalStateException subtype when asserting that SparkContext not stopped

2017-06-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050927#comment-16050927 ] Sean Owen edited comment on SPARK-21081 at 6/15/17 6:39 PM: [

[jira] [Commented] (SPARK-21081) Throw specific IllegalStateException subtype when asserting that SparkContext not stopped

2017-06-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050927#comment-16050927 ] Sean Owen commented on SPARK-21081: --- [~fzhinkin] when isn't it possible? you can {{ cat

[jira] [Created] (SPARK-21107) Pyspark: ISO-8859-1 column names inconsistently converted to UTF-8

2017-06-15 Thread Tavis Barr (JIRA)
Tavis Barr created SPARK-21107: -- Summary: Pyspark: ISO-8859-1 column names inconsistently converted to UTF-8 Key: SPARK-21107 URL: https://issues.apache.org/jira/browse/SPARK-21107 Project: Spark

[jira] [Comment Edited] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050896#comment-16050896 ] Hyukjin Kwon edited comment on SPARK-21093 at 6/15/17 6:19 PM:

[jira] [Comment Edited] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049944#comment-16049944 ] Hyukjin Kwon edited comment on SPARK-21093 at 6/15/17 6:17 PM:

[jira] [Commented] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050896#comment-16050896 ] Hyukjin Kwon commented on SPARK-21093: -- In case of my Mac, it looks the problem is h

[jira] [Commented] (SPARK-21104) Support sort with index when parse LibSVM Record

2017-06-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050850#comment-16050850 ] Sean Owen commented on SPARK-21104: --- I'm not sure what this is about. If the input isn'

[jira] [Commented] (SPARK-13210) NPE in Sort

2017-06-15 Thread Michael Smith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050807#comment-16050807 ] Michael Smith commented on SPARK-13210: --- I also saw this in a run on Spark 2.1.1. S

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2017-06-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050776#comment-16050776 ] Xiao Li commented on SPARK-1: - This function is still missing in the SQL interface.

[jira] [Commented] (SPARK-21101) Error running Hive temporary UDTF on latest Spark 2.2

2017-06-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050774#comment-16050774 ] Sean Owen commented on SPARK-21101: --- [~dyzhou] I see your reply. The thrift server shou

[jira] [Commented] (SPARK-21101) Error running Hive temporary UDTF on latest Spark 2.2

2017-06-15 Thread Dayou Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050762#comment-16050762 ] Dayou Zhou commented on SPARK-21101: Hi [~zhangzr1026], I'm still waiting for someone

[jira] [Updated] (SPARK-21097) Dynamic allocation will preserve cached data

2017-06-15 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad updated SPARK-21097: - Attachment: Preserving Cached Data with Dynamic Allocation.pdf > Dynamic allocation will preserve cached data > -

[jira] [Updated] (SPARK-21097) Dynamic allocation will preserve cached data

2017-06-15 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad updated SPARK-21097: - Attachment: (was: Preserving Cached Data with Dynamic Allocation.docx) > Dynamic allocation will preserve cac

[jira] [Updated] (SPARK-21097) Dynamic allocation will preserve cached data

2017-06-15 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad updated SPARK-21097: - Attachment: (was: Preserving Cached Data with Dynamic Allocation.docx) > Dynamic allocation will preserve cac

[jira] [Updated] (SPARK-21097) Dynamic allocation will preserve cached data

2017-06-15 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad updated SPARK-21097: - Attachment: Preserving Cached Data with Dynamic Allocation.docx > Dynamic allocation will preserve cached data >

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2017-06-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050730#comment-16050730 ] Reynold Xin commented on SPARK-1: - What's left in this ticket? Didn't we fix it a

[jira] [Updated] (SPARK-21097) Dynamic allocation will preserve cached data

2017-06-15 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad updated SPARK-21097: - Attachment: Preserving Cached Data with Dynamic Allocation.docx > Dynamic allocation will preserve cached data >

[jira] [Assigned] (SPARK-16251) LocalCheckpointSuite's - missing checkpoint block fails with informative message is flaky.

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-16251: --- Assignee: Jiang Xingbo > LocalCheckpointSuite's - missing checkpoint block fails with inform

[jira] [Assigned] (SPARK-20200) Flaky Test: org.apache.spark.rdd.LocalCheckpointSuite

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20200: --- Assignee: Jiang Xingbo > Flaky Test: org.apache.spark.rdd.LocalCheckpointSuite > ---

[jira] [Resolved] (SPARK-20200) Flaky Test: org.apache.spark.rdd.LocalCheckpointSuite

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20200. - Resolution: Fixed Fix Version/s: 2.1.2 2.2.0 2.0.3 I

[jira] [Resolved] (SPARK-16251) LocalCheckpointSuite's - missing checkpoint block fails with informative message is flaky.

2017-06-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16251. - Resolution: Fixed Fix Version/s: 2.1.2 2.2.0 2.0.3 I

[jira] [Reopened] (SPARK-21096) Pickle error when passing a member variable to Spark executors

2017-06-15 Thread Irina Truong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Irina Truong reopened SPARK-21096: -- The 2 methods I described should be equivalent, but they are not. > Pickle error when passing a me

[jira] [Commented] (SPARK-21096) Pickle error when passing a member variable to Spark executors

2017-06-15 Thread Irina Truong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050609#comment-16050609 ] Irina Truong commented on SPARK-21096: -- I am not passing in {{self}}. I am passing i

[jira] [Commented] (SPARK-21106) compile error

2017-06-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050445#comment-16050445 ] Hyukjin Kwon commented on SPARK-21106: -- It should build on Windows fine per AppVeyor

[jira] [Commented] (SPARK-21056) InMemoryFileIndex.listLeafFiles should create at most one spark job when listing files in parallel

2017-06-15 Thread Bertrand Bossy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050426#comment-16050426 ] Bertrand Bossy commented on SPARK-21056: cc [~michael] > InMemoryFileIndex.listL

[jira] [Updated] (SPARK-21098) Set lineseparator csv multiline and csv write to \n

2017-06-15 Thread Daniel van der Ende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel van der Ende updated SPARK-21098: Description: The Univocity-parser library uses the system line ending character as

[jira] [Updated] (SPARK-21098) Set lineseparator csv multiline and csv write to \n

2017-06-15 Thread Daniel van der Ende (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel van der Ende updated SPARK-21098: Description: sets the lineseparator for reading a multiline csv file or writing a c

  1   2   >