[jira] [Created] (SPARK-20577) Add REST API Documentation in Cluster Mode

2017-05-03 Thread guoxiaolongzte (JIRA)
guoxiaolongzte created SPARK-20577: -- Summary: Add REST API Documentation in Cluster Mode Key: SPARK-20577 URL: https://issues.apache.org/jira/browse/SPARK-20577 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-20546) spark-class gets syntax error in posix mode

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994464#comment-15994464 ] Sean Owen commented on SPARK-20546: --- Ok, seems fine to confine it to just this script w

[jira] [Resolved] (SPARK-20575) As a user,how do I know the return value type(double,bigint etc.) of a function

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20575. --- Resolution: Invalid Questions should go to u...@spark.apache.org, but I am not even sure this is a c

[jira] [Resolved] (SPARK-20572) Spark Streaming fail to read file on Hdfs

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20572. --- Resolution: Invalid It is not clear what you are describing, but this looks like a question that sho

[jira] [Resolved] (SPARK-6227) PCA and SVD for PySpark

2017-05-03 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-6227. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17621 [https://gi

[jira] [Commented] (SPARK-20577) Add REST API Documentation in Cluster Mode

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994499#comment-15994499 ] Apache Spark commented on SPARK-20577: -- User 'guoxiaolongzte' has created a pull req

[jira] [Assigned] (SPARK-20577) Add REST API Documentation in Cluster Mode

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20577: Assignee: (was: Apache Spark) > Add REST API Documentation in Cluster Mode > -

[jira] [Assigned] (SPARK-20577) Add REST API Documentation in Cluster Mode

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20577: Assignee: Apache Spark > Add REST API Documentation in Cluster Mode >

[jira] [Commented] (SPARK-20568) Delete files after processing

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994502#comment-15994502 ] Sean Owen commented on SPARK-20568: --- This conflicts with the general Spark model that o

[jira] [Resolved] (SPARK-20429) [GRAPHX] Strange results for personalized pagerank if node is involved in a cycle

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20429. --- Resolution: Duplicate > [GRAPHX] Strange results for personalized pagerank if node is involved in a

[jira] [Commented] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-05-03 Thread Nils Grabbert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994509#comment-15994509 ] Nils Grabbert commented on SPARK-19104: --- [~maropu] I think this issues is not relat

[jira] [Comment Edited] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-05-03 Thread Nils Grabbert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994509#comment-15994509 ] Nils Grabbert edited comment on SPARK-19104 at 5/3/17 9:05 AM:

[jira] [Commented] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread Umesh Chaudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994514#comment-15994514 ] Umesh Chaudhary commented on SPARK-20569: - Those functions seem to support additi

[jira] [Updated] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20570: -- Labels: (was: documentation) Indeed it seems like the 2.1.1 docs didn't deploy: http://spark.apache.o

[jira] [Resolved] (SPARK-20523) Clean up build warnings for 2.2.0 release

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20523. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17803 [https://github.co

[jira] [Updated] (SPARK-20523) Clean up build warnings for 2.2.0 release

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20523: -- Description: I'd like to fix up many small build warnings for 2.2.0. These generally are due to: - St

[jira] [Assigned] (SPARK-20523) Clean up build warnings for 2.2.0 release

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20523: - Assignee: Sean Owen > Clean up build warnings for 2.2.0 release > --

[jira] [Commented] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994548#comment-15994548 ] liuxian commented on SPARK-20569: - yes,it supports additional parameter. I think it is

[jira] [Commented] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread Umesh Chaudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994564#comment-15994564 ] Umesh Chaudhary commented on SPARK-20569: - I believe [~marmbrus] is the best pers

[jira] [Commented] (SPARK-18891) Support for specific collection types

2017-05-03 Thread Nils Grabbert (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994566#comment-15994566 ] Nils Grabbert commented on SPARK-18891: --- This will not work on v2.2.0-rc1: {code}

[jira] [Assigned] (SPARK-16957) Use weighted midpoints for split values.

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-16957: - Assignee: Yan Facai (颜发才) > Use weighted midpoints for split values. > -

[jira] [Updated] (SPARK-16957) Use weighted midpoints for split values.

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16957: -- Priority: Minor (was: Trivial) > Use weighted midpoints for split values. > --

[jira] [Resolved] (SPARK-16957) Use weighted midpoints for split values.

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16957. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 17556 [https://github.co

[jira] [Updated] (SPARK-19578) Poor pyspark performance

2017-05-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-19578: --- Summary: Poor pyspark performance (was: Poor pyspark performance + incorrect UI input-size m

[jira] [Updated] (SPARK-19578) Poor pyspark performance

2017-05-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-19578: --- Attachment: (was: pyspark_incorrect_inputsize.png) > Poor pyspark performance > -

[jira] [Updated] (SPARK-19578) Poor pyspark performance

2017-05-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-19578: --- Attachment: (was: spark_shell_correct_inputsize.png) > Poor pyspark performance > ---

[jira] [Updated] (SPARK-19578) Poor pyspark performance

2017-05-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-19578: --- Description: Simple job in pyspark takes 14 minutes to complete. The text file used to reprod

[jira] [Commented] (SPARK-19578) Poor pyspark performance

2017-05-03 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994617#comment-15994617 ] Artur Sukhenko commented on SPARK-19578: [~nchammas] I've opened separate issue f

[jira] [Commented] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994623#comment-15994623 ] liuxian commented on SPARK-20569: - ok,thanks Umesh Chaudhary > In spark-sql,Some functi

[jira] [Issue Comment Deleted] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-20569: Comment: was deleted (was: ok,thanks Umesh Chaudhary ) > In spark-sql,Some functions can execute successfu

[jira] [Commented] (SPARK-20569) In spark-sql,Some functions can execute successfully, when the number of input parameter is wrong

2017-05-03 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994627#comment-15994627 ] liuxian commented on SPARK-20569: - ok,thanks [~umesh9...@gmail.com] > In spark-sql,Some

[jira] [Commented] (SPARK-20433) Security issue with jackson-databind

2017-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994660#comment-15994660 ] Hyukjin Kwon commented on SPARK-20433: -- [~aash], What do you think about resolving t

[jira] [Created] (SPARK-20578) oozie submit spark job, the driver ui port is 0, instead of 4040

2017-05-03 Thread wuchang (JIRA)
wuchang created SPARK-20578: --- Summary: oozie submit spark job, the driver ui port is 0, instead of 4040 Key: SPARK-20578 URL: https://issues.apache.org/jira/browse/SPARK-20578 Project: Spark Issue

[jira] [Resolved] (SPARK-13802) Fields order in Row(**kwargs) is not consistent with Schema.toInternal method

2017-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13802. -- Resolution: Duplicate This looks a documented behaviour as mentioned above - https://github.co

[jira] [Updated] (SPARK-20568) Delete files after processing in structured streaming

2017-05-03 Thread Saul Shanabrook (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saul Shanabrook updated SPARK-20568: Summary: Delete files after processing in structured streaming (was: Delete files after pr

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994748#comment-15994748 ] Maciej Szymkiewicz commented on SPARK-12467: Python before 3.6 does not prese

[jira] [Resolved] (SPARK-20527) Schema issues when fields are queries in different order

2017-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20527. -- Resolution: Duplicate I tried to reproduces this but I could not reproduce as reported in the c

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994759#comment-15994759 ] Hyukjin Kwon commented on SPARK-12467: -- [~imachabeli] I will resolve this if there i

[jira] [Resolved] (SPARK-20578) oozie submit spark job, the driver ui port is 0, instead of 4040

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20578. --- Resolution: Invalid Questions should go to the user mailing list. JIRA isn't for tech support. > ooz

[jira] [Comment Edited] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994748#comment-15994748 ] Maciej Szymkiewicz edited comment on SPARK-12467 at 5/3/17 12:55 PM: --

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread John Berryman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994833#comment-15994833 ] John Berryman commented on SPARK-12467: --- I believe there is still something here th

[jira] [Comment Edited] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread John Berryman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994833#comment-15994833 ] John Berryman edited comment on SPARK-12467 at 5/3/17 1:21 PM:

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-03 Thread John Berryman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994849#comment-15994849 ] John Berryman commented on SPARK-12467: --- Here's a slightly different example that I

[jira] [Created] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread yao zhang (JIRA)
yao zhang created SPARK-20579: - Summary: large spark job hang on with many active stages/jobs Key: SPARK-20579 URL: https://issues.apache.org/jira/browse/SPARK-20579 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread yao zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994916#comment-15994916 ] yao zhang commented on SPARK-20579: --- I recently use scala spark to do complex data proc

[jira] [Updated] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread yao zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yao zhang updated SPARK-20579: -- Attachment: thread-dump-screen.png storage-screen.png stage-screen.png

[jira] [Updated] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread yao zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yao zhang updated SPARK-20579: -- Attachment: executor-err-log.txt > large spark job hang on with many active stages/jobs > -

[jira] [Commented] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread yao zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994919#comment-15994919 ] yao zhang commented on SPARK-20579: --- I attached one of executor error log file. > larg

[jira] [Resolved] (SPARK-20579) large spark job hang on with many active stages/jobs

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20579. --- Resolution: Invalid This is more of a tech support question, and JIRA isn't the place for it unless

[jira] [Created] (SPARK-20580) Allow RDD cache with unserializable objects

2017-05-03 Thread Fernando Pereira (JIRA)
Fernando Pereira created SPARK-20580: Summary: Allow RDD cache with unserializable objects Key: SPARK-20580 URL: https://issues.apache.org/jira/browse/SPARK-20580 Project: Spark Issue Typ

[jira] [Commented] (SPARK-20580) Allow RDD cache with unserializable objects

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994947#comment-15994947 ] Sean Owen commented on SPARK-20580: --- In general, such objects need to be serializable,

[jira] [Commented] (SPARK-20580) Allow RDD cache with unserializable objects

2017-05-03 Thread Fernando Pereira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994968#comment-15994968 ] Fernando Pereira commented on SPARK-20580: -- I try to avoid any operation involvi

[jira] [Created] (SPARK-20581) Using AVG or SUM on a INT/BIGINT column with fraction operator will yield BIGINT instead of DOUBLE

2017-05-03 Thread Dominic Ricard (JIRA)
Dominic Ricard created SPARK-20581: -- Summary: Using AVG or SUM on a INT/BIGINT column with fraction operator will yield BIGINT instead of DOUBLE Key: SPARK-20581 URL: https://issues.apache.org/jira/browse/SPARK-2

[jira] [Closed] (SPARK-20432) Unioning two identical Streaming DataFrames fails during attribute resolution

2017-05-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz closed SPARK-20432. --- Resolution: Duplicate > Unioning two identical Streaming DataFrames fails during attribute resolution

[jira] [Resolved] (SPARK-20441) Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation

2017-05-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20441. - Resolution: Fixed Resolved with https://github.com/apache/spark/pull/17735 > Within the same str

[jira] [Created] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

2017-05-03 Thread zhoukang (JIRA)
zhoukang created SPARK-20582: Summary: Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing Key: SPARK-20582 URL: https://issues.apache.org/jira/browse/SPARK-20582 Project: Spa

[jira] [Updated] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

2017-05-03 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-20582: - Description: In the current code of HistoryServer,jetty server will be started after all the logs

[jira] [Commented] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995149#comment-15995149 ] Felix Cheung commented on SPARK-20571: -- Thanks, I will look into today. > Flaky Spa

[jira] [Resolved] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing

2017-05-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20582. Resolution: Duplicate > Speed up the restart of HistoryServer using ApplicationAttemptInfo

[jira] [Created] (SPARK-20585) R generic hint support

2017-05-03 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-20585: --- Summary: R generic hint support Key: SPARK-20585 URL: https://issues.apache.org/jira/browse/SPARK-20585 Project: Spark Issue Type: Sub-task Component

[jira] [Created] (SPARK-20583) Scala/Java generic hint support

2017-05-03 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-20583: --- Summary: Scala/Java generic hint support Key: SPARK-20583 URL: https://issues.apache.org/jira/browse/SPARK-20583 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-20584) Python generic hint support

2017-05-03 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-20584: --- Summary: Python generic hint support Key: SPARK-20584 URL: https://issues.apache.org/jira/browse/SPARK-20584 Project: Spark Issue Type: Sub-task Comp

[jira] [Resolved] (SPARK-20583) Scala/Java generic hint support

2017-05-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-20583. - Resolution: Fixed Fix Version/s: 2.2.0 > Scala/Java generic hint support > ---

[jira] [Created] (SPARK-20586) Add deterministic and distinctLike to ScalaUDF

2017-05-03 Thread Xiao Li (JIRA)
Xiao Li created SPARK-20586: --- Summary: Add deterministic and distinctLike to ScalaUDF Key: SPARK-20586 URL: https://issues.apache.org/jira/browse/SPARK-20586 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995248#comment-15995248 ] Michael Armbrust commented on SPARK-20570: -- Hmmm, I did push them, and they show

[jira] [Commented] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995265#comment-15995265 ] Sean Owen commented on SPARK-20570: --- Oh, probably another git sync hiccup. It may 'flus

[jira] [Resolved] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20570. --- Resolution: Fixed Assignee: Michael Armbrust Fix Version/s: 2.1.1 Oh, [~marmbrus] alr

[jira] [Assigned] (SPARK-20548) Flaky Test: ReplSuite.newProductSeqEncoder with REPL defined class

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20548: Assignee: Apache Spark (was: Sameer Agarwal) > Flaky Test: ReplSuite.newProductSeqEncode

[jira] [Assigned] (SPARK-20548) Flaky Test: ReplSuite.newProductSeqEncoder with REPL defined class

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20548: Assignee: Sameer Agarwal (was: Apache Spark) > Flaky Test: ReplSuite.newProductSeqEncode

[jira] [Commented] (SPARK-20548) Flaky Test: ReplSuite.newProductSeqEncoder with REPL defined class

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995360#comment-15995360 ] Apache Spark commented on SPARK-20548: -- User 'cloud-fan' has created a pull request

[jira] [Resolved] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

2017-05-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19965. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 > DataFrame batch re

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-20564) a lot of executor failures when the executor number is more than 2000

2017-05-03 Thread Hua Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hua Liu updated SPARK-20564: Description: When we used more than 2000 executors in a spark application, we noticed a large number of ex

[jira] [Updated] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-19104: - Description: The following code will run with Spark 2.0.2 but not with Spark 2.1.0: {cod

[jira] [Updated] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-19104: - Affects Version/s: 2.2.0 Target Version/s: 2.2.0 > CompileException with Map and Ca

[jira] [Updated] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-20569: - Summary: RuntimeReplaceable functions accept invalid third parameter (was: In spark-sql,

[jira] [Commented] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995423#comment-15995423 ] Michael Armbrust commented on SPARK-20569: -- [~rxin] this does seem like a bug.

[jira] [Updated] (SPARK-20569) RuntimeReplaceable functions accept invalid third parameter

2017-05-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-20569: - Affects Version/s: 2.2.0 > RuntimeReplaceable functions accept invalid third parameter >

[jira] [Created] (SPARK-20587) Improve performance of ML ALS recommendForAll

2017-05-03 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-20587: -- Summary: Improve performance of ML ALS recommendForAll Key: SPARK-20587 URL: https://issues.apache.org/jira/browse/SPARK-20587 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20587) Improve performance of ML ALS recommendForAll

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995480#comment-15995480 ] Apache Spark commented on SPARK-20587: -- User 'MLnick' has created a pull request for

[jira] [Assigned] (SPARK-20587) Improve performance of ML ALS recommendForAll

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20587: Assignee: Apache Spark (was: Nick Pentreath) > Improve performance of ML ALS recommendFor

[jira] [Assigned] (SPARK-20587) Improve performance of ML ALS recommendForAll

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20587: Assignee: Nick Pentreath (was: Apache Spark) > Improve performance of ML ALS recommendFor

[jira] [Commented] (SPARK-19243) Error when selecting from DataFrame containing parsed data from files larger than 1MB

2017-05-03 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995489#comment-15995489 ] Harish commented on SPARK-19243: i am getting the same error in spark 2.1.0. I have 10 n

[jira] [Updated] (SPARK-20562) Support Maintenance by having a threshold for unavailability

2017-05-03 Thread Kamal Gurala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Gurala updated SPARK-20562: - Issue Type: Improvement (was: Bug) > Support Maintenance by having a threshold for unavailabilit

[jira] [Commented] (SPARK-20562) Support Maintenance by having a threshold for unavailability

2017-05-03 Thread Kamal Gurala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995493#comment-15995493 ] Kamal Gurala commented on SPARK-20562: -- Spark is not smart about offers that might b

[jira] [Commented] (SPARK-20562) Support Maintenance by having a threshold for unavailability

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995495#comment-15995495 ] Apache Spark commented on SPARK-20562: -- User 'gkc2104' has created a pull request fo

[jira] [Assigned] (SPARK-20562) Support Maintenance by having a threshold for unavailability

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20562: Assignee: Apache Spark > Support Maintenance by having a threshold for unavailability > --

[jira] [Assigned] (SPARK-20562) Support Maintenance by having a threshold for unavailability

2017-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20562: Assignee: (was: Apache Spark) > Support Maintenance by having a threshold for unavaila

[jira] [Created] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-03 Thread Ameen Tayyebi (JIRA)
Ameen Tayyebi created SPARK-20588: - Summary: from_utc_timestamp causes bottleneck Key: SPARK-20588 URL: https://issues.apache.org/jira/browse/SPARK-20588 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-03 Thread Ameen Tayyebi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995499#comment-15995499 ] Ameen Tayyebi commented on SPARK-20588: --- Hopefully more readable version of the cal

[jira] [Comment Edited] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-03 Thread Ameen Tayyebi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995499#comment-15995499 ] Ameen Tayyebi edited comment on SPARK-20588 at 5/3/17 7:40 PM:

[jira] [Commented] (SPARK-20568) Delete files after processing in structured streaming

2017-05-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995507#comment-15995507 ] Shixiong Zhu commented on SPARK-20568: -- [~srowen] Structured Streaming's Source has

[jira] [Comment Edited] (SPARK-19243) Error when selecting from DataFrame containing parsed data from files larger than 1MB

2017-05-03 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995489#comment-15995489 ] Harish edited comment on SPARK-19243 at 5/3/17 7:52 PM: i am gett

[jira] [Created] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20589: - Summary: Allow limiting task concurrency per stage Key: SPARK-20589 URL: https://issues.apache.org/jira/browse/SPARK-20589 Project: Spark Issue Type: Impro

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995550#comment-15995550 ] Mridul Muralidharan commented on SPARK-20589: - coalasce with shuffle=false mi

[jira] [Updated] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20588: -- Issue Type: Improvement (was: Bug) > from_utc_timestamp causes bottleneck > --

  1   2   >