[jira] [Assigned] (SPARK-26763) Using fileStatus cache when filterPartitions

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26763: Assignee: (was: Apache Spark) > Using fileStatus cache when filterPartitions > --

[jira] [Assigned] (SPARK-26763) Using fileStatus cache when filterPartitions

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26763: Assignee: Apache Spark > Using fileStatus cache when filterPartitions > -

[jira] [Created] (SPARK-26763) Using fileStatus cache when filterPartitions

2019-01-28 Thread Xianyang Liu (JIRA)
Xianyang Liu created SPARK-26763: Summary: Using fileStatus cache when filterPartitions Key: SPARK-26763 URL: https://issues.apache.org/jira/browse/SPARK-26763 Project: Spark Issue Type: Impr

[jira] [Comment Edited] (SPARK-26760) [Spark Incorrect display in YARN UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI]

2019-01-28 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754650#comment-16754650 ] shahid edited comment on SPARK-26760 at 1/29/19 6:37 AM: - [~abhi

[jira] [Commented] (SPARK-26760) [Spark Incorrect display in YARN UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI]

2019-01-28 Thread shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754650#comment-16754650 ] shahid commented on SPARK-26760: [~abhishek.akg] I would like to work on it, if no one i

[jira] [Updated] (SPARK-26760) [Spark Incorrect display in YARN UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI]

2019-01-28 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-26760: - Summary: [Spark Incorrect display in YARN UI Executor Tab when number of cores i

[jira] [Updated] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-26758: - Attachment: SPARK-26758.png > Idle Executors are not getting killed after > spa

[jira] [Updated] (SPARK-26760) [Spark Race condition when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI]

2019-01-28 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA updated SPARK-26760: - Attachment: SPARK-26760.png > [Spark Race condition when number of cores is 4 an

[jira] [Commented] (SPARK-26759) Arrow optimization in SparkR's interoperability

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754634#comment-16754634 ] Hyukjin Kwon commented on SPARK-26759: -- cc [~felixcheung], [~shivaram], [~bryanc],

[jira] [Commented] (SPARK-26761) Arrow optimization in native R function execution at gapply

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754636#comment-16754636 ] Hyukjin Kwon commented on SPARK-26761: -- I'm working on this. > Arrow optimization

[jira] [Assigned] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26566: Assignee: Bryan Cutler > Upgrade apache/arrow to 0.12.0 > --

[jira] [Resolved] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26566. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23657 [https://gi

[jira] [Commented] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754638#comment-16754638 ] sandeep katta commented on SPARK-26758: --- I would like to check this issue and fix

[jira] [Created] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

2019-01-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26762: Summary: Arrow optimization for conversion from Spark DataFrame to R DataFrame Key: SPARK-26762 URL: https://issues.apache.org/jira/browse/SPARK-26762 Project: Spark

[jira] [Commented] (SPARK-26752) Multiple aggregate methods in the same column in DataFrame

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754627#comment-16754627 ] Hyukjin Kwon commented on SPARK-26752: -- As you described, workaround is pretty easy

[jira] [Created] (SPARK-26761) Arrow optimization in native R function execution at gapply

2019-01-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26761: Summary: Arrow optimization in native R function execution at gapply Key: SPARK-26761 URL: https://issues.apache.org/jira/browse/SPARK-26761 Project: Spark

[jira] [Updated] (SPARK-26759) Arrow optimization in SparkR's interoperability

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26759: - Component/s: SQL > Arrow optimization in SparkR's interoperability > ---

[jira] [Updated] (SPARK-25981) Arrow optimization for conversion from R DataFrame to Spark DataFrame

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25981: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-26759 > Arrow optimization for co

[jira] [Commented] (SPARK-26739) Standardized Join Types for DataFrames

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754630#comment-16754630 ] Hyukjin Kwon commented on SPARK-26739: -- So, is it just to propose constant variable

[jira] [Updated] (SPARK-26746) Adaptive causes non-action operations to trigger computation

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26746: - Description: When we turn on the {{spark.sql.adaptive.enabled}} switch, the following actions t

[jira] [Created] (SPARK-26760) [Spark Race condition when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI]

2019-01-28 Thread ABHISHEK KUMAR GUPTA (JIRA)
ABHISHEK KUMAR GUPTA created SPARK-26760: Summary: [Spark Race condition when number of cores is 4 and Active Task display as 5 in Executor Tab of YARN UI] Key: SPARK-26760 URL: https://issues.apache.org/j

[jira] [Commented] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754626#comment-16754626 ] Hyukjin Kwon commented on SPARK-26758: -- Can you include UI screenshot to explain th

[jira] [Updated] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26758: - Component/s: (was: Spark Shell) YARN > Idle Executors are not getting kille

[jira] [Updated] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26758: - Summary: Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout

[jira] [Updated] (SPARK-26758) Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26758: - Description: Steps: 1. Submit Spark shell with below initial Executor 3, minimum Executor=0 and

[jira] [Created] (SPARK-26759) Arrow optimization in SparkR's interoperability

2019-01-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26759: Summary: Arrow optimization in SparkR's interoperability Key: SPARK-26759 URL: https://issues.apache.org/jira/browse/SPARK-26759 Project: Spark Issue Type: U

[jira] [Created] (SPARK-26758) [Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value

2019-01-28 Thread ABHISHEK KUMAR GUPTA (JIRA)
ABHISHEK KUMAR GUPTA created SPARK-26758: Summary: [Idle Executors are not getting killed after spark.dynamicAllocation.executorIdleTimeout value Key: SPARK-26758 URL: https://issues.apache.org/jira/browse

[jira] [Assigned] (SPARK-26757) GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26757: Assignee: (was: Apache Spark) > GraphX EdgeRDDImpl and VertexRDDImpl `count` method c

[jira] [Assigned] (SPARK-26757) GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26757: Assignee: Apache Spark > GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handl

[jira] [Updated] (SPARK-26757) GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs

2019-01-28 Thread Huon Wilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huon Wilson updated SPARK-26757: Priority: Minor (was: Major) > GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle

[jira] [Created] (SPARK-26757) GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs

2019-01-28 Thread Huon Wilson (JIRA)
Huon Wilson created SPARK-26757: --- Summary: GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs Key: SPARK-26757 URL: https://issues.apache.org/jira/browse/SPARK-26757 Project: S

[jira] [Assigned] (SPARK-26756) Support session conf for thriftserver

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26756: Assignee: Apache Spark > Support session conf for thriftserver >

[jira] [Assigned] (SPARK-26756) Support session conf for thriftserver

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26756: Assignee: (was: Apache Spark) > Support session conf for thriftserver > -

[jira] [Updated] (SPARK-26756) Support session conf for thriftserver

2019-01-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-26756: - Description: We can add support for session conf.like: {code:java} set spark.sql.xxx.xxx=xxx {code} whi

[jira] [Created] (SPARK-26756) Support session conf for thriftserver

2019-01-28 Thread zhoukang (JIRA)
zhoukang created SPARK-26756: Summary: Support session conf for thriftserver Key: SPARK-26756 URL: https://issues.apache.org/jira/browse/SPARK-26756 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23516: Description: Now `StaticMemoryManager` mode has been removed. And for `UnifiedMemoryManager`, unroll me

[jira] [Assigned] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23516: Assignee: (was: Apache Spark) > I think it is unnecessary to transfer unroll memory t

[jira] [Assigned] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23516: Assignee: Apache Spark > I think it is unnecessary to transfer unroll memory to storage m

[jira] [Commented] (SPARK-26748) CLONE - Autoencoder

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754505#comment-16754505 ] Hyukjin Kwon commented on SPARK-26748: -- That's fine. Let me leave this one resolved

[jira] [Resolved] (SPARK-26748) CLONE - Autoencoder

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26748. -- Resolution: Duplicate > CLONE - Autoencoder > --- > > Key: SPA

[jira] [Commented] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2019-01-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754499#comment-16754499 ] Takeshi Yamamuro commented on SPARK-26708: -- Yea, thanks for the answer! > Inco

[jira] [Reopened] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian reopened SPARK-23516: - > I think it is unnecessary to transfer unroll memory to storage memory > -

[jira] [Updated] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23516: Description: Now _StaticMemoryManager_ mode has been removed. And for _UnifiedMemoryManager_,  unroll mem

[jira] [Updated] (SPARK-23516) I think it is unnecessary to transfer unroll memory to storage memory

2019-01-28 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-23516: Affects Version/s: (was: 2.3.0) 3.0.0 > I think it is unnecessary to transfer u

[jira] [Commented] (SPARK-26651) Use Proleptic Gregorian calendar

2019-01-28 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754437#comment-16754437 ] Maxim Gekk commented on SPARK-26651: [~cloud_fan] [~hvanhovell] [~srowen] [~hyukjin.

[jira] [Commented] (SPARK-26651) Use Proleptic Gregorian calendar

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754438#comment-16754438 ] Sean Owen commented on SPARK-26651: --- You can just tag this with release-notes and add

[jira] [Assigned] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26755: Assignee: Apache Spark > Optimize Spark Scheduler to dequeue speculative tasks more effic

[jira] [Assigned] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26755: Assignee: (was: Apache Spark) > Optimize Spark Scheduler to dequeue speculative tasks

[jira] [Updated] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26755: - Attachment: Screen Shot 2019-01-28 at 11.21.05 AM.png > Optimize Spark Scheduler to dequeue spec

[jira] [Updated] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26755: - Attachment: Screen Shot 2019-01-28 at 11.22.42 AM.png > Optimize Spark Scheduler to dequeue spec

[jira] [Created] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Parth Gandhi (JIRA)
Parth Gandhi created SPARK-26755: Summary: Optimize Spark Scheduler to dequeue speculative tasks more efficiently Key: SPARK-26755 URL: https://issues.apache.org/jira/browse/SPARK-26755 Project: Spark

[jira] [Updated] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

2019-01-28 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26755: - Attachment: Screen Shot 2019-01-28 at 11.21.25 AM.png > Optimize Spark Scheduler to dequeue spec

[jira] [Assigned] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26754: Assignee: (was: Apache Spark) > Add hasTrainingSummary to replace duplicate code in P

[jira] [Assigned] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26754: Assignee: Apache Spark > Add hasTrainingSummary to replace duplicate code in PySpark > --

[jira] [Commented] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

2019-01-28 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754420#comment-16754420 ] Huaxin Gao commented on SPARK-26754: I will submit a PR soon. Thanks! > Add hasTrai

[jira] [Created] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

2019-01-28 Thread Huaxin Gao (JIRA)
Huaxin Gao created SPARK-26754: -- Summary: Add hasTrainingSummary to replace duplicate code in PySpark Key: SPARK-26754 URL: https://issues.apache.org/jira/browse/SPARK-26754 Project: Spark Issu

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite.fetchBothChunks

2019-01-28 Thread Sanket Reddy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754406#comment-16754406 ] Sanket Reddy commented on SPARK-25692: -- I had a few observations regarding this tes

[jira] [Resolved] (SPARK-26747) Makes GetMapValue nullability more precise

2019-01-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26747. --- Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 3.0.0 This is re

[jira] [Assigned] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26595: -- Assignee: Marcelo Vanzin > Allow delegation token renewal without a keytab >

[jira] [Resolved] (SPARK-26595) Allow delegation token renewal without a keytab

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26595. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23525 [https:

[jira] [Commented] (SPARK-26154) Stream-stream joins - left outer join gives inconsistent output

2019-01-28 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754335#comment-16754335 ] Jungtaek Lim commented on SPARK-26154: -- [~tdas] [~zsxwing] [~joseph.torres] Could w

[jira] [Updated] (SPARK-26379) Use dummy TimeZoneId for CurrentTimestamp to avoid UnresolvedException in CurrentBatchTimestamp

2019-01-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26379: -- Fix Version/s: 2.3.3 > Use dummy TimeZoneId for CurrentTimestamp to avoid UnresolvedException

[jira] [Assigned] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26753: Assignee: (was: Apache Spark) > Log4j customization not working for spark-shell > ---

[jira] [Assigned] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26753: Assignee: Apache Spark > Log4j customization not working for spark-shell > --

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2019-01-28 Thread Nannan Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754319#comment-16754319 ] Nannan Yu commented on SPARK-21287: --- [~smilegator] Ok. Anything we should follow up to

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2019-01-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754318#comment-16754318 ] Xiao Li commented on SPARK-21287: - [~bestcastor] Feel free to submit a PR > Cannot use

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2019-01-28 Thread Nannan Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754316#comment-16754316 ] Nannan Yu commented on SPARK-21287: --- Do we have any following up for this issue? MySQL

[jira] [Commented] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2019-01-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754300#comment-16754300 ] Xiao Li commented on SPARK-26708: - I do not think this affects 2.3 > Incorrect result c

[jira] [Created] (SPARK-26753) Log4j customization not working for spark-shell

2019-01-28 Thread Ankur Gupta (JIRA)
Ankur Gupta created SPARK-26753: --- Summary: Log4j customization not working for spark-shell Key: SPARK-26753 URL: https://issues.apache.org/jira/browse/SPARK-26753 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-26731) remove EOLed spark jobs from jenkins

2019-01-28 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754215#comment-16754215 ] shane knapp commented on SPARK-26731: - uhhh... did [~Thatboix45] get hacked? i'm g

[jira] [Updated] (SPARK-26731) remove EOLed spark jobs from jenkins

2019-01-28 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26731: Attachment: (was: activemq-cli-tools-4a984ec.tar.gz) > remove EOLed spark jobs from jenkins >

[jira] [Updated] (SPARK-26731) remove EOLed spark jobs from jenkins

2019-01-28 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26731: Attachment: (was: LICENSE) > remove EOLed spark jobs from jenkins > --

[jira] [Updated] (SPARK-26722) Set SPARK_TEST_KEY to pull request builder and spark-master-test-sbt-hadoop-2.7

2019-01-28 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26722: Attachment: (was: SPARK-26731.doc) > Set SPARK_TEST_KEY to pull request builder and > spark-m

[jira] [Resolved] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26432. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23429 [https:

[jira] [Assigned] (SPARK-26432) Not able to connect Hbase 2.1 service Getting NoSuchMethodException while trying to obtain token from Hbase 2.1 service.

2019-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26432: -- Assignee: Sujith > Not able to connect Hbase 2.1 service Getting NoSuchMethodExceptio

[jira] [Updated] (SPARK-26739) Standardized Join Types for DataFrames

2019-01-28 Thread Skyler Lehan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Skyler Lehan updated SPARK-26739: - Description: h3. *Q1.* What are you trying to do? Articulate your objectives using absolutely n

[jira] [Assigned] (SPARK-26713) PipedRDD may holds stdin writer and stdout read threads even if the task is finished

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26713: - Assignee: Xianjin YE > PipedRDD may holds stdin writer and stdout read threads even if the task

[jira] [Resolved] (SPARK-26713) PipedRDD may holds stdin writer and stdout read threads even if the task is finished

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26713. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23638 [https://github.c

[jira] [Assigned] (SPARK-26651) Use Proleptic Gregorian calendar

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26651: - Assignee: Maxim Gekk > Use Proleptic Gregorian calendar > > >

[jira] [Assigned] (SPARK-26719) Get rid of java.util.Calendar in DateTimeUtils

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26719: - Assignee: Maxim Gekk > Get rid of java.util.Calendar in DateTimeUtils > ---

[jira] [Resolved] (SPARK-26719) Get rid of java.util.Calendar in DateTimeUtils

2019-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26719. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23641 [https://github.c

[jira] [Resolved] (SPARK-26700) enable fetch-big-block-to-memory by default

2019-01-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26700. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23625 [https://gith

[jira] [Created] (SPARK-26752) Multiple aggregate methods in the same column in DataFrame

2019-01-28 Thread Guilherme Beltramini (JIRA)
Guilherme Beltramini created SPARK-26752: Summary: Multiple aggregate methods in the same column in DataFrame Key: SPARK-26752 URL: https://issues.apache.org/jira/browse/SPARK-26752 Project: Sp

[jira] [Updated] (SPARK-26752) Multiple aggregate methods in the same column in DataFrame

2019-01-28 Thread Guilherme Beltramini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guilherme Beltramini updated SPARK-26752: - Description: The agg function in  [org.apache.spark.sql.RelationalGroupedDataset

[jira] [Assigned] (SPARK-26656) Benchmark for date/time functions and expressions

2019-01-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell reassigned SPARK-26656: - Assignee: Maxim Gekk > Benchmark for date/time functions and expressions >

[jira] [Resolved] (SPARK-26656) Benchmark for date/time functions and expressions

2019-01-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-26656. --- Resolution: Fixed Fix Version/s: 3.0.0 > Benchmark for date/time functions an

[jira] [Commented] (SPARK-26709) OptimizeMetadataOnlyQuery does not correctly handle the files with zero record

2019-01-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753948#comment-16753948 ] Takeshi Yamamuro commented on SPARK-26709: -- Resolved by https://github.com/apac

[jira] [Updated] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-26751: - Description: When we run in background and we get exception which is not HiveSQLException, we may encoun

[jira] [Assigned] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26751: Assignee: (was: Apache Spark) > HiveSessionImpl might have memory leak since Operatio

[jira] [Assigned] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26751: Assignee: Apache Spark > HiveSessionImpl might have memory leak since Operation do not cl

[jira] [Updated] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-26751: - Description: When we run in background and we get exception which is not HiveSQLException, we may encoun

[jira] [Updated] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-26751: - Attachment: 26751.png > HiveSessionImpl might have memory leak since Operation do not close properly > -

[jira] [Created] (SPARK-26751) HiveSessionImpl might have memory leak since Operation do not close properly

2019-01-28 Thread zhoukang (JIRA)
zhoukang created SPARK-26751: Summary: HiveSessionImpl might have memory leak since Operation do not close properly Key: SPARK-26751 URL: https://issues.apache.org/jira/browse/SPARK-26751 Project: Spark

[jira] [Assigned] (SPARK-26750) Estimate memory overhead should taking multi-cores into account

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26750: Assignee: (was: Apache Spark) > Estimate memory overhead should taking multi-cores in

[jira] [Assigned] (SPARK-26750) Estimate memory overhead should taking multi-cores into account

2019-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26750: Assignee: Apache Spark > Estimate memory overhead should taking multi-cores into account

[jira] [Updated] (SPARK-26750) Estimate memory overhead should taking multi-cores into account

2019-01-28 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26750: - Summary: Estimate memory overhead should taking multi-cores into account (was: Estimate memory

[jira] [Commented] (SPARK-26748) CLONE - Autoencoder

2019-01-28 Thread Chris Bogan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753891#comment-16753891 ] Chris Bogan commented on SPARK-26748: - My mistake I am terribly sorry > CLONE - Aut

[jira] [Commented] (SPARK-26749) spark streaming kafka verison for high version

2019-01-28 Thread Chang Quanyou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753870#comment-16753870 ] Chang Quanyou commented on SPARK-26749: --- where is a mailing list? > spark streami

[jira] [Resolved] (SPARK-26738) Pyspark random forest classifier feature importance with column names

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26738. -- Resolution: Invalid > Pyspark random forest classifier feature importance with column names >

[jira] [Commented] (SPARK-26738) Pyspark random forest classifier feature importance with column names

2019-01-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753867#comment-16753867 ] Hyukjin Kwon commented on SPARK-26738: -- Questions should go to mailing list. Let's

  1   2   >