[jira] [Assigned] (SPARK-21351) Update nullability based on children's output in optimized logical plan

[ https://issues.apache.org/jira/browse/SPARK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21351: Assignee: (was: Apache Spark) > Update nullability based on children's output in optim

[jira] [Assigned] (SPARK-21351) Update nullability based on children's output in optimized logical plan

[ https://issues.apache.org/jira/browse/SPARK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21351: Assignee: Apache Spark > Update nullability based on children's output in optimized logica

[jira] [Commented] (SPARK-21351) Update nullability based on children's output in optimized logical plan

[ https://issues.apache.org/jira/browse/SPARK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079480#comment-16079480 ] Apache Spark commented on SPARK-21351: -- User 'maropu' has created a pull request for

[jira] [Commented] (SPARK-21083) Store zero size and row count after analyzing empty table

[ https://issues.apache.org/jira/browse/SPARK-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079474#comment-16079474 ] Apache Spark commented on SPARK-21083: -- User 'wzhfy' has created a pull request for

[jira] [Updated] (SPARK-21351) Update nullability based on children's output in optimized logical plan

[ https://issues.apache.org/jira/browse/SPARK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21351: - Description: In the master, optimized plans do not respect the nullability that `Filter`

[jira] [Created] (SPARK-21351) Update nullability based on children's output in optimized logical plan

Takeshi Yamamuro created SPARK-21351: Summary: Update nullability based on children's output in optimized logical plan Key: SPARK-21351 URL: https://issues.apache.org/jira/browse/SPARK-21351 Proje

[jira] [Commented] (SPARK-21350) Fix the error message when the number of arguments is wrong when invoking a UDF

[ https://issues.apache.org/jira/browse/SPARK-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079433#comment-16079433 ] Apache Spark commented on SPARK-21350: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-21350) Fix the error message when the number of arguments is wrong when invoking a UDF

[ https://issues.apache.org/jira/browse/SPARK-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21350: Assignee: Apache Spark (was: Xiao Li) > Fix the error message when the number of argument

[jira] [Assigned] (SPARK-21350) Fix the error message when the number of arguments is wrong when invoking a UDF

[ https://issues.apache.org/jira/browse/SPARK-21350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21350: Assignee: Xiao Li (was: Apache Spark) > Fix the error message when the number of argument

[jira] [Created] (SPARK-21350) Fix the error message when the number of arguments is wrong when invoking a UDF

Xiao Li created SPARK-21350: --- Summary: Fix the error message when the number of arguments is wrong when invoking a UDF Key: SPARK-21350 URL: https://issues.apache.org/jira/browse/SPARK-21350 Project: Spark

[jira] [Updated] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

[ https://issues.apache.org/jira/browse/SPARK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21345: Fix Version/s: 2.1.2 > SparkSessionBuilderSuite should clean up stopped sessions >

[jira] [Comment Edited] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079421#comment-16079421 ] duyanghao edited comment on SPARK-21348 at 7/9/17 2:36 AM: --- [~j

[jira] [Commented] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079421#comment-16079421 ] duyanghao commented on SPARK-21348: --- @Saisai Shao ok, i will follow [SPARK-18085](http

[jira] [Comment Edited] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079421#comment-16079421 ] duyanghao edited comment on SPARK-21348 at 7/9/17 2:35 AM: --- [~j

[jira] [Closed] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duyanghao closed SPARK-21348. - Resolution: Duplicate > Spark history Server load too slow when eventlog is large > -

[jira] [Comment Edited] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079407#comment-16079407 ] Saisai Shao edited comment on SPARK-21346 at 7/9/17 1:29 AM: -

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079407#comment-16079407 ] Saisai Shao commented on SPARK-21346: - Jars, files can be downloaded from HTTPS if yo

[jira] [Comment Edited] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079405#comment-16079405 ] John edited comment on SPARK-21346 at 7/9/17 1:27 AM: -- Sorry, I wasn

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079405#comment-16079405 ] John commented on SPARK-21346: -- Sorry, I wasn't aware of that. Do you mind elaborating more

[jira] [Commented] (SPARK-20660) Not able to merge Dataframes with different column orders

[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079402#comment-16079402 ] Takeshi Yamamuro commented on SPARK-20660: -- You mean this function like? https:/

[jira] [Comment Edited] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079389#comment-16079389 ] Saisai Shao edited comment on SPARK-21334 at 7/9/17 12:43 AM: -

[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079389#comment-16079389 ] Saisai Shao commented on SPARK-21334: - Are you using external shuffle service with YA

[jira] [Commented] (SPARK-21346) Spark does not use SSL for HTTP File Server and Broadcast Server

[ https://issues.apache.org/jira/browse/SPARK-21346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079385#comment-16079385 ] Saisai Shao commented on SPARK-21346: - SSL is still used for fetch remote resources f

[jira] [Commented] (SPARK-21348) Spark history Server load too slow when eventlog is large

[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079381#comment-16079381 ] Saisai Shao commented on SPARK-21348: - SPARK-18085 is trying to address this issue.

[jira] [Assigned] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21349: Assignee: (was: Apache Spark) > Make TASK_SIZE_TO_WARN_KB configurable > -

[jira] [Assigned] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21349: Assignee: Apache Spark > Make TASK_SIZE_TO_WARN_KB configurable >

[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079349#comment-16079349 ] Apache Spark commented on SPARK-21349: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Updated] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21349: -- Description: Although this is just a warning message, this issue tries to make `TASK_SIZE_TO_WA

[jira] [Created] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB configurable

Dongjoon Hyun created SPARK-21349: - Summary: Make TASK_SIZE_TO_WARN_KB configurable Key: SPARK-21349 URL: https://issues.apache.org/jira/browse/SPARK-21349 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-20660) Not able to merge Dataframes with different column orders

[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079340#comment-16079340 ] Anton Okolnychyi commented on SPARK-20660: -- I used the following code to investi

[jira] [Commented] (SPARK-19568) Must include class/method documentation for CRAN check

[ https://issues.apache.org/jira/browse/SPARK-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079325#comment-16079325 ] Felix Cheung commented on SPARK-19568: -- separate from CRAN task > Must include clas

[jira] [Updated] (SPARK-19568) Must include class/method documentation for CRAN check

[ https://issues.apache.org/jira/browse/SPARK-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-19568: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-15799) > Must include class/meth

[jira] [Updated] (SPARK-21290) R document Programmatically Specifying the Schema in SQL guide

[ https://issues.apache.org/jira/browse/SPARK-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-21290: - Target Version/s: 2.3.0 > R document Programmatically Specifying the Schema in SQL guide > --

[jira] [Commented] (SPARK-21290) R document Programmatically Specifying the Schema in SQL guide

[ https://issues.apache.org/jira/browse/SPARK-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079323#comment-16079323 ] Felix Cheung commented on SPARK-21290: -- with some changes to schema string support o

[jira] [Resolved] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-21093. -- Resolution: Fixed Fix Version/s: 2.3.0 > Multiple gapply execution occasionally failed i

[jira] [Commented] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

[ https://issues.apache.org/jira/browse/SPARK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079308#comment-16079308 ] Apache Spark commented on SPARK-21345: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Closed] (SPARK-21307) Remove SQLConf parameters from the parser-related classes.

[ https://issues.apache.org/jira/browse/SPARK-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-21307. --- Resolution: Won't Fix > Remove SQLConf parameters from the parser-related classes. >

[jira] [Assigned] (SPARK-21343) Refine the document for spark.reducer.maxReqSizeShuffleToMem

[ https://issues.apache.org/jira/browse/SPARK-21343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21343: --- Assignee: jin xing > Refine the document for spark.reducer.maxReqSizeShuffleToMem >

[jira] [Resolved] (SPARK-21343) Refine the document for spark.reducer.maxReqSizeShuffleToMem

[ https://issues.apache.org/jira/browse/SPARK-21343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21343. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Assigned] (SPARK-20342) DAGScheduler sends SparkListenerTaskEnd before updating task's accumulators

[ https://issues.apache.org/jira/browse/SPARK-20342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20342: --- Assignee: Marcelo Vanzin > DAGScheduler sends SparkListenerTaskEnd before updating task's ac

[jira] [Resolved] (SPARK-20342) DAGScheduler sends SparkListenerTaskEnd before updating task's accumulators

[ https://issues.apache.org/jira/browse/SPARK-20342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20342. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Comment Edited] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

[ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076463#comment-16076463 ] Anton Okolnychyi edited comment on SPARK-18859 at 7/8/17 2:36 PM: -

[jira] [Commented] (SPARK-9476) Kafka stream loses leader after 2h of operation

[ https://issues.apache.org/jira/browse/SPARK-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079148#comment-16079148 ] Dharmas commented on SPARK-9476: Even I am getting the same issue !. any update on this.

[jira] [Commented] (SPARK-6227) PCA and SVD for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079132#comment-16079132 ] Abou Haydar Elias commented on SPARK-6227: -- I was also wondering if we can add PC

[jira] [Resolved] (SPARK-21083) Store zero size and row count after analyzing empty table

[ https://issues.apache.org/jira/browse/SPARK-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21083. - Resolution: Fixed Assignee: Zhenhua Wang Fix Version/s: 2.3.0 > Store zero size a

[jira] [Assigned] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

[ https://issues.apache.org/jira/browse/SPARK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21345: --- Assignee: Dongjoon Hyun > SparkSessionBuilderSuite should clean up stopped sessions > --

[jira] [Resolved] (SPARK-21345) SparkSessionBuilderSuite should clean up stopped sessions

[ https://issues.apache.org/jira/browse/SPARK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21345. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Updated] (SPARK-21228) InSet incorrect handling of structs

[ https://issues.apache.org/jira/browse/SPARK-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21228: Fix Version/s: 2.2.1 > InSet incorrect handling of structs > --- >

[jira] [Assigned] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21344: Assignee: Apache Spark > BinaryType comparison does signed byte array comparison > ---

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079065#comment-16079065 ] Apache Spark commented on SPARK-21344: -- User 'kiszk' has created a pull request for

[jira] [Assigned] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21344: Assignee: (was: Apache Spark) > BinaryType comparison does signed byte array compariso

[jira] [Commented] (SPARK-21100) Add summary method as alternative to describe that gives quartiles similar to Pandas

[ https://issues.apache.org/jira/browse/SPARK-21100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079057#comment-16079057 ] Apache Spark commented on SPARK-21100: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079032#comment-16079032 ] Kazuaki Ishizaki commented on SPARK-21337: -- I cannot reproduce this by using the

[jira] [Updated] (SPARK-21083) Store zero size and row count after analyzing empty table

[ https://issues.apache.org/jira/browse/SPARK-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-21083: - Summary: Store zero size and row count after analyzing empty table (was: Do not remove correct s

[jira] [Updated] (SPARK-21083) Store zero size and row count after analyzing empty table

[ https://issues.apache.org/jira/browse/SPARK-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-21083: - Description: (was: 1. When we first analyze without `noscan` and then analyze with `noscan`,

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079023#comment-16079023 ] Sean Owen commented on SPARK-21344: --- Bytes are signed, as are other int types in Spark

[jira] [Resolved] (SPARK-21347) Performance issues with KryoSerializer

[ https://issues.apache.org/jira/browse/SPARK-21347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21347. --- Resolution: Not A Problem I think this is more of question, and about a change or configuration in Kr

[jira] [Commented] (SPARK-21341) Spark 2.1.1: I want to be able to serialize wordVectors on Word2VecModel

[ https://issues.apache.org/jira/browse/SPARK-21341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079019#comment-16079019 ] Sean Owen commented on SPARK-21341: --- Hm, I don't think you normally save a model with s

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079017#comment-16079017 ] Dongjoon Hyun commented on SPARK-21344: --- +1 ! :) > BinaryType comparison does sign

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079010#comment-16079010 ] Kazuaki Ishizaki commented on SPARK-21344: -- I will work for this if anyone has f

[jira] [Assigned] (SPARK-20456) Add examples for functions collection for pyspark

[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-20456: Assignee: Michael Patterson > Add examples for functions collection for pyspark >

[jira] [Resolved] (SPARK-20456) Add examples for functions collection for pyspark

[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20456. -- Resolution: Fixed Fix Version/s: 2.3.0 Target Version/s: 2.3.0 > Add examples