[jira] [Resolved] (SPARK-24365) Add data source write benchmark

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24365. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21409

[jira] [Assigned] (SPARK-24365) Add data source write benchmark

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24365: --- Assignee: Gengliang Wang > Add data source write benchmark >

[jira] [Comment Edited] (SPARK-24403) reuse r worker

2018-05-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494707#comment-16494707 ] Felix Cheung edited comment on SPARK-24403 at 5/30/18 5:38 AM: --- Reuse

[jira] [Commented] (SPARK-23650) Slow SparkR udf (dapply)

2018-05-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494708#comment-16494708 ] Felix Cheung commented on SPARK-23650: -- sorry, I really don't have time/resource to investigate

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24420: Assignee: DB Tsai (was: Apache Spark) > Upgrade ASM to 6.x to support JDK9+ >

[jira] [Commented] (SPARK-24403) reuse r worker

2018-05-29 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494707#comment-16494707 ] Felix Cheung commented on SPARK-24403: -- Reuse worker (daemon process) is actually supported and the

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24420: Assignee: Apache Spark (was: DB Tsai) > Upgrade ASM to 6.x to support JDK9+ >

[jira] [Commented] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494706#comment-16494706 ] Apache Spark commented on SPARK-24420: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24419: Assignee: DB Tsai (was: Apache Spark) > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24419: Assignee: Apache Spark (was: DB Tsai) > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Commented] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494705#comment-16494705 ] Apache Spark commented on SPARK-24419: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494701#comment-16494701 ] Izek Greenfield commented on SPARK-23904: - [~RBerenguel] `setting completeString to no-op` what

[jira] [Updated] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-29 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jongyoul Lee updated SPARK-24120: - Fix Version/s: 0.8.1 0.9.0 > Show `Jobs` page when `jobId` is missing >

[jira] [Updated] (SPARK-24120) Show `Jobs` page when `jobId` is missing

2018-05-29 Thread Jongyoul Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jongyoul Lee updated SPARK-24120: - Fix Version/s: (was: 0.9.0) (was: 0.8.1) > Show `Jobs` page when

[jira] [Commented] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494647#comment-16494647 ] Liang-Chi Hsieh commented on SPARK-24409: - Seems you use AWS Glue Data Catalog as the Metastore

[jira] [Commented] (SPARK-8659) Spark SQL Thrift Server does NOT honour hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

2018-05-29 Thread L (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494620#comment-16494620 ] L commented on SPARK-8659: -- I meet the problems,too.And i choose the sentry as the authorisation for Spark

[jira] [Commented] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494590#comment-16494590 ] Hyukjin Kwon commented on SPARK-24409: -- Mind sharing a reproducer if you already have? > exception

[jira] [Resolved] (SPARK-24376) compiling spark with scala-2.10 should use the -P parameter instead of -D

2018-05-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24376. -- Resolution: Won't Fix > compiling spark with scala-2.10 should use the -P parameter instead

[jira] [Updated] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-24417: Description: This is an umbrella JIRA for Apache Spark to support JDK9+ As Java 8 is going way soon,

[jira] [Created] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24422: --- Summary: Add JDK9+ in our Jenkins' build servers Key: SPARK-24422 URL: https://issues.apache.org/jira/browse/SPARK-24422 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24419: --- Assignee: DB Tsai > Upgrade SBT to 0.13.17 with Scala 2.10.7 >

[jira] [Assigned] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24420: --- Assignee: DB Tsai > Upgrade ASM to 6.x to support JDK9+ > --- > >

[jira] [Assigned] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-24418: --- Assignee: DB Tsai > Upgrade to Scala 2.11.12 > > > Key:

[jira] [Created] (SPARK-24421) sun.misc.Unsafe in JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24421: --- Summary: sun.misc.Unsafe in JDK9+ Key: SPARK-24421 URL: https://issues.apache.org/jira/browse/SPARK-24421 Project: Spark Issue Type: Sub-task Components:

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494513#comment-16494513 ] Ruben Berenguel commented on SPARK-23904: - [~igreenfi] after a few more tries at reproducing,

[jira] [Created] (SPARK-24420) Upgrade ASM to 6.x to support JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24420: --- Summary: Upgrade ASM to 6.x to support JDK9+ Key: SPARK-24420 URL: https://issues.apache.org/jira/browse/SPARK-24420 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494462#comment-16494462 ] Ruben Berenguel commented on SPARK-23904: - Finally, managed to reproduce (takes a long while,

[jira] [Created] (SPARK-24419) Upgrade SBT to 0.13.17 with Scala 2.10.7

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24419: --- Summary: Upgrade SBT to 0.13.17 with Scala 2.10.7 Key: SPARK-24419 URL: https://issues.apache.org/jira/browse/SPARK-24419 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-24418) Upgrade to Scala 2.11.12

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24418: --- Summary: Upgrade to Scala 2.11.12 Key: SPARK-24418 URL: https://issues.apache.org/jira/browse/SPARK-24418 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-24417) Build and Run Spark on JDK9+

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24417: --- Summary: Build and Run Spark on JDK9+ Key: SPARK-24417 URL: https://issues.apache.org/jira/browse/SPARK-24417 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming

2018-05-29 Thread sivanesh selvanataraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494403#comment-16494403 ] sivanesh selvanataraj commented on SPARK-18165: --- [~itsvikramagr] I got this error when i

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494391#comment-16494391 ] Liang-Chi Hsieh commented on SPARK-24410: - [~cloud_fan] Thanks for pinging me. I'll look into

[jira] [Resolved] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24413. --- Resolution: Duplicate > Executor Blacklisting shouldn't immediately fail the application if

[jira] [Assigned] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24414: Assignee: Apache Spark > Stages page doesn't show all task attempts when failures >

[jira] [Assigned] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24414: Assignee: (was: Apache Spark) > Stages page doesn't show all task attempts when

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494300#comment-16494300 ] Apache Spark commented on SPARK-24414: -- User 'vanzin' has created a pull request for this issue:

[jira] [Updated] (SPARK-24392) Mark pandas_udf as Experimental

2018-05-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-24392: - Fix Version/s: 2.4.0 > Mark pandas_udf as Experimental > --- > >

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494279#comment-16494279 ] Thomas Graves commented on SPARK-24413: --- thanks for linking those we can just dup this to

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494276#comment-16494276 ] Marcelo Vanzin commented on SPARK-24414: After a quick look at the code they don't seem related.

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494265#comment-16494265 ] Thomas Graves commented on SPARK-24414: --- also just an fyi I also filed SPARK-24415, not sure if

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494239#comment-16494239 ] Marcelo Vanzin commented on SPARK-24414: Yeah that's the direction I ended up in. Taking the

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494237#comment-16494237 ] Thomas Graves commented on SPARK-24414: --- I am looking to see if we can just return an empty table

[jira] [Created] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-05-29 Thread Sanket Reddy (JIRA)
Sanket Reddy created SPARK-24416: Summary: Update configuration definition for spark.blacklist.killBlacklistedExecutors Key: SPARK-24416 URL: https://issues.apache.org/jira/browse/SPARK-24416

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494206#comment-16494206 ] Marcelo Vanzin commented on SPARK-24414: I was going to take a stab at this next. > Stages page

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494199#comment-16494199 ] Thomas Graves commented on SPARK-24414: --- looks like this was broken by SPARK-23147, so we probably

[jira] [Assigned] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24356: Assignee: Apache Spark > Duplicate strings in File.path managed by

[jira] [Commented] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494184#comment-16494184 ] Apache Spark commented on SPARK-24356: -- User 'countmdm' has created a pull request for this issue:

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494183#comment-16494183 ] Imran Rashid commented on SPARK-24413: -- yeah I agree about this. I linked two related jiras that

[jira] [Assigned] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24356: Assignee: (was: Apache Spark) > Duplicate strings in File.path managed by

[jira] [Commented] (SPARK-24395) Fix Behavior of NOT IN with Literals Containing NULL

2018-05-29 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494181#comment-16494181 ] Juliusz Sompolski commented on SPARK-24395: --- The question is whether the literals should be

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494152#comment-16494152 ] Apache Spark commented on SPARK-24093: -- User 'merlintang' has created a pull request for this

[jira] [Commented] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494154#comment-16494154 ] Mingjie Tang commented on SPARK-24093: -- I made a PR for this

[jira] [Assigned] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24093: Assignee: Apache Spark > Make some fields of

[jira] [Assigned] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24093: Assignee: (was: Apache Spark) > Make some fields of

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Description: Running with spark 2.3 on yarn and having task failures and blacklisting, the

[jira] [Created] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24415: - Summary: Stage page aggregated executor metrics wrong when failures Key: SPARK-24415 URL: https://issues.apache.org/jira/browse/SPARK-24415 Project: Spark

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Attachment: Screen Shot 2018-05-29 at 2.15.38 PM.png > Stage page aggregated executor metrics

[jira] [Commented] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494083#comment-16494083 ] Thomas Graves commented on SPARK-24414: --- to reproduce this simply start a shell:

[jira] [Created] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24414: - Summary: Stages page doesn't show all task attempts when failures Key: SPARK-24414 URL: https://issues.apache.org/jira/browse/SPARK-24414 Project: Spark

[jira] [Commented] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-29 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494058#comment-16494058 ] Ruslan Dautkhanov commented on SPARK-24356: --- Another improvement for YARN NodeManagers we saw

[jira] [Updated] (SPARK-22666) Spark datasource for image format

2018-05-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22666: -- Summary: Spark datasource for image format (was: Spark reader source for image

[jira] [Commented] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494038#comment-16494038 ] Apache Spark commented on SPARK-24337: -- User 'PenguinToast' has created a pull request for this

[jira] [Assigned] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24337: Assignee: Apache Spark (was: Xiao Li) > Improve the error message for invalid SQL conf

[jira] [Assigned] (SPARK-24337) Improve the error message for invalid SQL conf value

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24337: Assignee: Xiao Li (was: Apache Spark) > Improve the error message for invalid SQL conf

[jira] [Commented] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493978#comment-16493978 ] Thomas Graves commented on SPARK-24413: --- [~imranr]  thoughts on this? > Executor Blacklisting

[jira] [Updated] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors

2018-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24413: -- Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic

[jira] [Created] (SPARK-24413) Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors

2018-05-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24413: - Summary: Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and it doesn't have any other active executors Key: SPARK-24413

[jira] [Created] (SPARK-24412) Adding docs about automagical type casting in `isin` and `isInCollection` APIs

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24412: --- Summary: Adding docs about automagical type casting in `isin` and `isInCollection` APIs Key: SPARK-24412 URL: https://issues.apache.org/jira/browse/SPARK-24412 Project: Spark

[jira] [Created] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-05-29 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24411: --- Summary: Adding native Java tests for `isInCollection` Key: SPARK-24411 URL: https://issues.apache.org/jira/browse/SPARK-24411 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-24371) Added isInCollection in DataFrame API for Scala and Java.

2018-05-29 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-24371. - Resolution: Fixed > Added isInCollection in DataFrame API for Scala and Java. >

[jira] [Assigned] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24296: Assignee: (was: Apache Spark) > Support replicating blocks larger than 2 GB >

[jira] [Commented] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493864#comment-16493864 ] Apache Spark commented on SPARK-24296: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24296) Support replicating blocks larger than 2 GB

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24296: Assignee: Apache Spark > Support replicating blocks larger than 2 GB >

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493815#comment-16493815 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:34 PM: - Hi [~TomaszGaweda]

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493815#comment-16493815 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:34 PM: - Hi [~TomaszGaweda]

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493815#comment-16493815 ] Li Jin commented on SPARK-22947: Hi [~TomaszGaweda] thanks for your interest! Yes I am willing to work

[jira] [Comment Edited] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-05-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449935#comment-16449935 ] Li Jin edited comment on SPARK-22947 at 5/29/18 4:33 PM: - I came across this

[jira] [Commented] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493759#comment-16493759 ] Apache Spark commented on SPARK-24319: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Assigned] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24319: Assignee: Apache Spark > run-example can not print usage >

[jira] [Assigned] (SPARK-24319) run-example can not print usage

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24319: Assignee: (was: Apache Spark) > run-example can not print usage >

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493637#comment-16493637 ] Wenchen Fan commented on SPARK-24410: - The `UnionExec#outputPartitioning` should be smarter and

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493569#comment-16493569 ] Wenbo Zhao commented on SPARK-24373: [~mgaido] Thanks. I didn't look the comment carefully.  >

[jira] [Issue Comment Deleted] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenbo Zhao updated SPARK-24373: --- Comment: was deleted (was: Same question as [~icexelloss]. Also, any plan to make your fix into a

[jira] [Commented] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493561#comment-16493561 ] Ohad Raviv commented on SPARK-24410: [~sowen], [~cloud_fan] - could you please check if my

[jira] [Created] (SPARK-24410) Missing optimization for Union on bucketed tables

2018-05-29 Thread Ohad Raviv (JIRA)
Ohad Raviv created SPARK-24410: -- Summary: Missing optimization for Union on bucketed tables Key: SPARK-24410 URL: https://issues.apache.org/jira/browse/SPARK-24410 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24073) DataSourceV2: Rename DataReaderFactory to InputPartition.

2018-05-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24073: Summary: DataSourceV2: Rename DataReaderFactory to InputPartition. (was: DataSourceV2: Rename

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493548#comment-16493548 ] Marco Gaido commented on SPARK-24373: - [~wbzhao] as I answered on the PR, the fix is complete and

[jira] [Commented] (SPARK-24373) "df.cache() df.count()" no longer eagerly caches data when the analyzed plans are different after re-analyzing the plans

2018-05-29 Thread Wenbo Zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493545#comment-16493545 ] Wenbo Zhao commented on SPARK-24373: Same question as [~icexelloss]. Also, any plan to make your fix

[jira] [Commented] (SPARK-24370) spark checkpoint creates many 0 byte empty files(partitions) in checkpoint directory

2018-05-29 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493520#comment-16493520 ] Gabor Somogyi commented on SPARK-24370: --- In general it's not an issue to have empty partitions

[jira] [Comment Edited] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2018-05-29 Thread SemanticBeeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493498#comment-16493498 ] SemanticBeeng edited comment on SPARK-18649 at 5/29/18 12:51 PM: - Should

[jira] [Commented] (SPARK-18649) sc.textFile(my_file).collect() raises socket.timeout on large files

2018-05-29 Thread SemanticBeeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493498#comment-16493498 ] SemanticBeeng commented on SPARK-18649: --- Should this hard coded timeout value not be configurable?

[jira] [Assigned] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24385: Assignee: Apache Spark > Trivially-true EqualNullSafe should be handled like EqualTo in

[jira] [Assigned] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24385: Assignee: (was: Apache Spark) > Trivially-true EqualNullSafe should be handled like

[jira] [Commented] (SPARK-24385) Trivially-true EqualNullSafe should be handled like EqualTo in Dataset.join

2018-05-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493486#comment-16493486 ] Apache Spark commented on SPARK-24385: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Resolved] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23991. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Assigned] (SPARK-23991) data loss when allocateBlocksToBatch

2018-05-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23991: --- Assignee: Gabor Somogyi > data loss when allocateBlocksToBatch >

[jira] [Updated] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Janet Levin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janet Levin updated SPARK-24409: Affects Version/s: (was: 2.3.0) 2.2.1 > exception when sending large

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493327#comment-16493327 ] Ruben Berenguel commented on SPARK-23904: - I could not (but had no time to dive too deep), but

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-29 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493323#comment-16493323 ] Izek Greenfield commented on SPARK-23904: - [~RBerenguel] Did you manage to reproduce? In my side

[jira] [Created] (SPARK-24409) exception when sending large list in filter(col(x).isin(list))

2018-05-29 Thread Janet Levin (JIRA)
Janet Levin created SPARK-24409: --- Summary: exception when sending large list in filter(col(x).isin(list)) Key: SPARK-24409 URL: https://issues.apache.org/jira/browse/SPARK-24409 Project: Spark

  1   2   >