[jira] [Created] (SPARK-24408) Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

2018-05-29 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-24408: --- Summary: Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group Key: SPARK-24408 URL: https://issues.apache.org/jira/browse/SPARK-24408 Project: Spark

[jira] [Assigned] (SPARK-24408) Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24408: Assignee: Apache Spark > Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

[jira] [Assigned] (SPARK-24408) Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24408: Assignee: (was: Apache Spark) > Move abs, bitwiseNOT, isnan, nanvl functions to math_

[jira] [Commented] (SPARK-24408) Move abs, bitwiseNOT, isnan, nanvl functions to math_funcs group

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493195#comment-16493195 ] Apache Spark commented on SPARK-24408: -- User 'jaceklaskowski' has created a pull re

[jira] [Commented] (SPARK-19250) In security cluster, spark beeline connect to hive metastore failed

2018-05-29 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493254#comment-16493254 ] zuotingbing commented on SPARK-19250: - Same problem in spark 2.2.1 . But We add kini

[jira] [Updated] (SPARK-24353) Add support for pod affinity/anti-affinity

2018-05-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-24353: Description: Spark on K8s allows to place driver/executor pods on specific k8s nod

[jira] [Created] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Janet Levin (JIRA)
Janet Levin created SPARK-24409: --- Summary: exception when sending large list in filter(col(x).isin(list)) Key: SPARK-24409 URL: https://issues.apache.org/jira/browse/SPARK-24409 Project: Spark

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493323#comment-16493323 ] Izek Greenfield commented on SPARK-23904: - [~RBerenguel] Did you manage to repro

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493327#comment-16493327 ] Ruben Berenguel commented on SPARK-23904: - I could not (but had no time to dive

[jira] [Updated] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Janet Levin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janet Levin updated SPARK-24409: Affects Version/s: (was: 2.3.0) 2.2.1 > exception when sending large li

[jira] [Assigned] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23991: --- Assignee: Gabor Somogyi > data loss when allocateBlocksToBatch > --

[jira] [Resolved] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23991. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull re

[jira] [Commented] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493486#comment-16493486 ] Apache Spark commented on SPARK-24385: -- User 'mgaido91' has created a pull request

[jira] [Assigned] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24385: Assignee: Apache Spark > Trivially-true EqualNullSafe should be handled like EqualTo in D

[jira] [Assigned] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24385: Assignee: (was: Apache Spark) > Trivially-true EqualNullSafe should be handled like E

[jira] [Commented] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2018-05-29 Thread SemanticBeeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493498#comment-16493498 ] SemanticBeeng commented on SPARK-18649: --- Should this hard coded timeout value not

[jira] [Comment Edited] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2018-05-29 Thread SemanticBeeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493498#comment-16493498 ] SemanticBeeng edited comment on SPARK-18649 at 5/29/18 12:51 PM: -

[jira] [Commented] (SPARK-24370) spark checkpoint creates many 0 byte empty files(partitions) in checkpoint directory

2018-05-29 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493520#comment-16493520 ] Gabor Somogyi commented on SPARK-24370: --- In general it's not an issue to have empt

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493545#comment-16493545 ] Wenbo Zhao commented on SPARK-24373: Same question as [~icexelloss]. Also, any plan

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493548#comment-16493548 ] Marco Gaido commented on SPARK-24373: - [~wbzhao] as I answered on the PR, the fix is

[jira] [Updated] (SPARK-24073) DataSourceV2: Rename DataReaderFactory to InputPartition.

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24073: Summary: DataSourceV2: Rename DataReaderFactory to InputPartition. (was: DataSourceV2: Rename Dat

[jira] [Created] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Ohad Raviv (JIRA)
Ohad Raviv created SPARK-24410: -- Summary: Missing optimization for Union on bucketed tables Key: SPARK-24410 URL: https://issues.apache.org/jira/browse/SPARK-24410 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493561#comment-16493561 ] Ohad Raviv commented on SPARK-24410: [~sowen], [~cloud_fan] - could you please check

[jira] [Issue Comment Deleted] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenbo Zhao updated SPARK-24373: --- Comment: was deleted (was: Same question as [~icexelloss]. Also, any plan to make your fix into a m

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493569#comment-16493569 ] Wenbo Zhao commented on SPARK-24373: [~mgaido] Thanks. I didn't look the comment car

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493637#comment-16493637 ] Wenchen Fan commented on SPARK-24410: - The `UnionExec#outputPartitioning` should be

[jira] [Assigned] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24319: Assignee: (was: Apache Spark) > run-example can not print usage > ---

[jira] [Commented] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493759#comment-16493759 ] Apache Spark commented on SPARK-24319: -- User 'gaborgsomogyi' has created a pull req

[jira] [Assigned] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24319: Assignee: Apache Spark > run-example can not print usage > --

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493815#comment-16493815 ] Li Jin commented on SPARK-22947: Hi [~TomaszGaweda] thanks for your interest! Yes I am w

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449935#comment-16449935 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:33 PM: - I came

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493815#comment-16493815 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:34 PM: - Hi [~T

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493815#comment-16493815 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:34 PM: - Hi [~T

[jira] [Commented] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493864#comment-16493864 ] Apache Spark commented on SPARK-24296: -- User 'squito' has created a pull request fo

[jira] [Assigned] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24296: Assignee: Apache Spark > Support replicating blocks larger than 2 GB > --

[jira] [Assigned] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24296: Assignee: (was: Apache Spark) > Support replicating blocks larger than 2 GB > ---

[jira] [Resolved] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24371. - Resolution: Fixed > Added isInCollection in DataFrame API for Scala and Java. >

[jira] [Created] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24411: --- Summary: Adding native Java tests for `isInCollection` Key: SPARK-24411 URL: https://issues.apache.org/jira/browse/SPARK-24411 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-24412) Adding docs about automagical type casting in `isin` and `isInCollection` APIs

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24412: --- Summary: Adding docs about automagical type casting in `isin` and `isInCollection` APIs Key: SPARK-24412 URL: https://issues.apache.org/jira/browse/SPARK-24412 Project: Spark

[jira] [Created] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24413: - Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors Key: SPARK-24413 URL:

[jira] [Updated] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24413: -- Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocatio

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493978#comment-16493978 ] Thomas Graves commented on SPARK-24413: --- [~imranr]  thoughts on this? > Executor

[jira] [Assigned] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24337: Assignee: Apache Spark (was: Xiao Li) > Improve the error message for invalid SQL conf v

[jira] [Assigned] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24337: Assignee: Xiao Li (was: Apache Spark) > Improve the error message for invalid SQL conf v

[jira] [Commented] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494038#comment-16494038 ] Apache Spark commented on SPARK-24337: -- User 'PenguinToast' has created a pull requ

[jira] [Updated] (SPARK-22666) Spark datasource for image format

2018-05-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22666: -- Summary: Spark datasource for image format (was: Spark reader source for image format

[jira] [Commented] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494058#comment-16494058 ] Ruslan Dautkhanov commented on SPARK-24356: --- Another improvement for YARN Node

[jira] [Created] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24414: - Summary: Stages page doesn't show all task attempts when failures Key: SPARK-24414 URL: https://issues.apache.org/jira/browse/SPARK-24414 Project: Spark Is

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494083#comment-16494083 ] Thomas Graves commented on SPARK-24414: --- to reproduce this simply start a shell:

[jira] [Created] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24415: - Summary: Stage page aggregated executor metrics wrong when failures Key: SPARK-24415 URL: https://issues.apache.org/jira/browse/SPARK-24415 Project: Spark

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Attachment: Screen Shot 2018-05-29 at 2.15.38 PM.png > Stage page aggregated executor metrics

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the ag

[jira] [Assigned] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24093: Assignee: (was: Apache Spark) > Make some fields of KafkaStreamWriter/InternalRowMicr

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494152#comment-16494152 ] Apache Spark commented on SPARK-24093: -- User 'merlintang' has created a pull reques

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494154#comment-16494154 ] Mingjie Tang commented on SPARK-24093: -- I made a PR for this https://github.com/ap

[jira] [Assigned] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24093: Assignee: Apache Spark > Make some fields of KafkaStreamWriter/InternalRowMicroBatchWrite

[jira] [Commented] (SPARK-24395) Fix Behavior of NOT IN with Literals Containing NULL

2018-05-29 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494181#comment-16494181 ] Juliusz Sompolski commented on SPARK-24395: --- The question is whether the liter

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494183#comment-16494183 ] Imran Rashid commented on SPARK-24413: -- yeah I agree about this. I linked two rela

[jira] [Assigned] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24356: Assignee: (was: Apache Spark) > Duplicate strings in File.path managed by FileSegment

[jira] [Assigned] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24356: Assignee: Apache Spark > Duplicate strings in File.path managed by FileSegmentManagedBuff

[jira] [Commented] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494184#comment-16494184 ] Apache Spark commented on SPARK-24356: -- User 'countmdm' has created a pull request

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494199#comment-16494199 ] Thomas Graves commented on SPARK-24414: --- looks like this was broken by SPARK-23147

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494206#comment-16494206 ] Marcelo Vanzin commented on SPARK-24414: I was going to take a stab at this next

[jira] [Created] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-05-29 Thread Sanket Reddy (JIRA)
Sanket Reddy created SPARK-24416: Summary: Update configuration definition for spark.blacklist.killBlacklistedExecutors Key: SPARK-24416 URL: https://issues.apache.org/jira/browse/SPARK-24416 Project:

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494237#comment-16494237 ] Thomas Graves commented on SPARK-24414: --- I am looking to see if we can just return

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494239#comment-16494239 ] Marcelo Vanzin commented on SPARK-24414: Yeah that's the direction I ended up in

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494265#comment-16494265 ] Thomas Graves commented on SPARK-24414: --- also just an fyi I also filed SPARK-24415

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494276#comment-16494276 ] Marcelo Vanzin commented on SPARK-24414: After a quick look at the code they don

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494279#comment-16494279 ] Thomas Graves commented on SPARK-24413: --- thanks for linking those we can just dup

[jira] [Updated] (SPARK-24392) Mark pandas_udf as Experimental

2018-05-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-24392: - Fix Version/s: 2.4.0 > Mark pandas_udf as Experimental > --- > >

[jira] [Assigned] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24414: Assignee: (was: Apache Spark) > Stages page doesn't show all task attempts when failu

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494300#comment-16494300 ] Apache Spark commented on SPARK-24414: -- User 'vanzin' has created a pull request fo

[jira] [Assigned] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24414: Assignee: Apache Spark > Stages page doesn't show all task attempts when failures > -

[jira] [Resolved] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24413. --- Resolution: Duplicate > Executor Blacklisting shouldn't immediately fail the application if

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494391#comment-16494391 ] Liang-Chi Hsieh commented on SPARK-24410: - [~cloud_fan] Thanks for pinging me. I

[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming

2018-05-29 Thread sivanesh selvanataraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494403#comment-16494403 ] sivanesh selvanataraj commented on SPARK-18165: --- [~itsvikramagr] I got thi

[jira] [Created] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24417: --- Summary: Build and Run Spark on JDK9+ Key: SPARK-24417 URL: https://issues.apache.org/jira/browse/SPARK-24417 Project: Spark Issue Type: New Feature Componen

[jira] [Created] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24418: --- Summary: Upgrade to Scala 2.11.12 Key: SPARK-24418 URL: https://issues.apache.org/jira/browse/SPARK-24418 Project: Spark Issue Type: Sub-task Components: Bui

[jira] [Created] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24419: --- Summary: Upgrade SBT to 0.13.17 with Scala 2.10.7 Key: SPARK-24419 URL: https://issues.apache.org/jira/browse/SPARK-24419 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494462#comment-16494462 ] Ruben Berenguel commented on SPARK-23904: - Finally, managed to reproduce (takes

[jira] [Created] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24420: --- Summary: Upgrade ASM to 6.x to support JDK9+ Key: SPARK-24420 URL: https://issues.apache.org/jira/browse/SPARK-24420 Project: Spark Issue Type: Sub-task Comp

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494513#comment-16494513 ] Ruben Berenguel commented on SPARK-23904: - [~igreenfi] after a few more tries at

[jira] [Created] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24421: --- Summary: sun.misc.Unsafe in JDK9+ Key: SPARK-24421 URL: https://issues.apache.org/jira/browse/SPARK-24421 Project: Spark Issue Type: Sub-task Components: Bui

[jira] [Assigned] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24418: --- Assignee: DB Tsai > Upgrade to Scala 2.11.12 > > > Key: SP

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24419: --- Assignee: DB Tsai > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24420: --- Assignee: DB Tsai > Upgrade ASM to 6.x to support JDK9+ > --- > >

[jira] [Created] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24422: --- Summary: Add JDK9+ in our Jenkins' build servers Key: SPARK-24422 URL: https://issues.apache.org/jira/browse/SPARK-24422 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24417: Description: This is an umbrella JIRA for Apache Spark to support JDK9+ As Java 8 is going way soon, JDK1

[jira] [Resolved] (SPARK-24376) compiling spark with scala-2.10 should use the -P parameter instead of -D

2018-05-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24376. -- Resolution: Won't Fix > compiling spark with scala-2.10 should use the -P parameter instead of

[jira] [Commented] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494590#comment-16494590 ] Hyukjin Kwon commented on SPARK-24409: -- Mind sharing a reproducer if you already ha

[jira] [Commented] (SPARK-8659) Spark SQL Thrift Server does NOT honour hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

2018-05-29 Thread L (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494620#comment-16494620 ] L commented on SPARK-8659: -- I meet the problems,too.And i choose the sentry as the authorisation

[jira] [Commented] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494647#comment-16494647 ] Liang-Chi Hsieh commented on SPARK-24409: - Seems you use AWS Glue Data Catalog a

[jira] [Updated] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-29 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jongyoul Lee updated SPARK-24120: - Fix Version/s: (was: 0.9.0) (was: 0.8.1) > Show `Jobs` page when `job

[jira] [Updated] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-29 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jongyoul Lee updated SPARK-24120: - Fix Version/s: 0.8.1 0.9.0 > Show `Jobs` page when `jobId` is missing > -

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494701#comment-16494701 ] Izek Greenfield commented on SPARK-23904: - [~RBerenguel] `setting completeString

[jira] [Commented] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494705#comment-16494705 ] Apache Spark commented on SPARK-24419: -- User 'dbtsai' has created a pull request fo

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24419: Assignee: DB Tsai (was: Apache Spark) > Upgrade SBT to 0.13.17 with Scala 2.10.7 > -

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24419: Assignee: Apache Spark (was: DB Tsai) > Upgrade SBT to 0.13.17 with Scala 2.10.7 > -

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24420: Assignee: Apache Spark (was: DB Tsai) > Upgrade ASM to 6.x to support JDK9+ > --

[jira] [Commented] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494706#comment-16494706 ] Apache Spark commented on SPARK-24420: -- User 'dbtsai' has created a pull request fo

  1   2   >