[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/20/18 7:33 AM: -- [~skonto] so

[jira] [Commented] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369750#comment-16369750 ] Manan Bakshi commented on SPARK-23463: -- Hi Marco, I realized that if you used following code for

[jira] [Assigned] (SPARK-23436) Incorrect Date column Inference in partition discovery

[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23436: --- Assignee: Marco Gaido > Incorrect Date column Inference in partition discovery >

[jira] [Resolved] (SPARK-23436) Incorrect Date column Inference in partition discovery

[ https://issues.apache.org/jira/browse/SPARK-23436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23436. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20621

[jira] [Resolved] (SPARK-23457) Register task completion listeners first for ParquetFileFormat

[ https://issues.apache.org/jira/browse/SPARK-23457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23457. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20619

[jira] [Assigned] (SPARK-23457) Register task completion listeners first for ParquetFileFormat

[ https://issues.apache.org/jira/browse/SPARK-23457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23457: --- Assignee: Dongjoon Hyun > Register task completion listeners first for ParquetFileFormat >

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369724#comment-16369724 ] Dongjoon Hyun commented on SPARK-23443: --- Hi, [~ameen.tayy...@gmail.com]. I have the same need for

[jira] [Updated] (SPARK-23443) Spark with Glue as external catalog

[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23443: -- Target Version/s: (was: 2.4.0) > Spark with Glue as external catalog >

[jira] [Commented] (SPARK-23466) Remove redundant null checks in generated Java code by GenerateUnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369623#comment-16369623 ] Apache Spark commented on SPARK-23466: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23466) Remove redundant null checks in generated Java code by GenerateUnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23466: Assignee: Apache Spark > Remove redundant null checks in generated Java code by >

[jira] [Assigned] (SPARK-23466) Remove redundant null checks in generated Java code by GenerateUnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23466: Assignee: (was: Apache Spark) > Remove redundant null checks in generated Java code

[jira] [Commented] (SPARK-15348) Hive ACID

[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369602#comment-16369602 ] Dongjoon Hyun commented on SPARK-15348: --- Since Hive 3.0, Hive ACID 2.0 (HIVE-14035) table becomes

[jira] [Updated] (SPARK-23467) Enable way to create DataFrame from pre-partitioned files (Parquet/ORC/etc.) with each in-memory partition mapped to 1 physical file partition

[ https://issues.apache.org/jira/browse/SPARK-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V Luong updated SPARK-23467: Description: I would like to echo the need described here:

[jira] [Updated] (SPARK-23467) Enable way to create DataFrame from pre-partitioned files (Parquet/ORC/etc.) with each in-memory partition mapped to 1 physical file partition

[ https://issues.apache.org/jira/browse/SPARK-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V Luong updated SPARK-23467: Description: I would like to echo the need described here:

[jira] [Updated] (SPARK-23467) Enable way to create DataFrame from pre-partitioned files (Parquet/ORC/etc.) with each in-memory partition mapped to 1 physical file partition

[ https://issues.apache.org/jira/browse/SPARK-23467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] V Luong updated SPARK-23467: Description: I would like to echo the need described here:

[jira] [Created] (SPARK-23467) Enable way to create DataFrame from pre-partitioned files (Parquet/ORC/etc.) with each in-memory partition mapped to 1 physical file partition

V Luong created SPARK-23467: --- Summary: Enable way to create DataFrame from pre-partitioned files (Parquet/ORC/etc.) with each in-memory partition mapped to 1 physical file partition Key: SPARK-23467 URL:

[jira] [Commented] (SPARK-23346) Failed tasks reported as success if the failure reason is not ExceptionFailure

[ https://issues.apache.org/jira/browse/SPARK-23346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369600#comment-16369600 ] Dongjoon Hyun commented on SPARK-23346: --- Thank you for reporting. I adjusted the priority to

[jira] [Updated] (SPARK-23346) Failed tasks reported as success if the failure reason is not ExceptionFailure

[ https://issues.apache.org/jira/browse/SPARK-23346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23346: -- Priority: Critical (was: Blocker) > Failed tasks reported as success if the failure reason is

[jira] [Commented] (SPARK-23449) Extra java options lose order in Docker context

[ https://issues.apache.org/jira/browse/SPARK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369567#comment-16369567 ] Dongjoon Hyun commented on SPARK-23449: --- I removed the fixed version from this issue. We can update

[jira] [Updated] (SPARK-23449) Extra java options lose order in Docker context

[ https://issues.apache.org/jira/browse/SPARK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23449: -- Fix Version/s: (was: 2.3.0) > Extra java options lose order in Docker context >

[jira] [Updated] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23094: -- Fix Version/s: (was: 2.3.0) > Json Readers choose wrong encoding when bad records are

[jira] [Commented] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369565#comment-16369565 ] Dongjoon Hyun commented on SPARK-23094: --- Since this is reverted and not a regression, I'll remove

[jira] [Commented] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369564#comment-16369564 ] Dongjoon Hyun commented on SPARK-23390: --- Since this is not a regression for both Parquet(not a

[jira] [Updated] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Target Version/s: (was: 2.3.0) > Flaky Test Suite: FileBasedDataSourceSuite in Spark

[jira] [Updated] (SPARK-23390) Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

[ https://issues.apache.org/jira/browse/SPARK-23390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23390: -- Fix Version/s: (was: 2.3.0) > Flaky Test Suite: FileBasedDataSourceSuite in Spark

[jira] [Assigned] (SPARK-23417) pyspark tests give wrong sbt instructions

[ https://issues.apache.org/jira/browse/SPARK-23417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23417: Assignee: Apache Spark > pyspark tests give wrong sbt instructions >

[jira] [Commented] (SPARK-23417) pyspark tests give wrong sbt instructions

[ https://issues.apache.org/jira/browse/SPARK-23417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369500#comment-16369500 ] Apache Spark commented on SPARK-23417: -- User 'bersprockets' has created a pull request for this

[jira] [Assigned] (SPARK-23417) pyspark tests give wrong sbt instructions

[ https://issues.apache.org/jira/browse/SPARK-23417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23417: Assignee: (was: Apache Spark) > pyspark tests give wrong sbt instructions >

[jira] [Commented] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369424#comment-16369424 ] Manan Bakshi commented on SPARK-23463: -- Hi Marco, I have attached a text file along with this

[jira] [Updated] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manan Bakshi updated SPARK-23463: - Attachment: sample > Filter operation fails to handle blank values and evicts rows that even >

[jira] [Created] (SPARK-23466) Remove redundant null checks in generated Java code by GenerateUnsafeProjection

Kazuaki Ishizaki created SPARK-23466: Summary: Remove redundant null checks in generated Java code by GenerateUnsafeProjection Key: SPARK-23466 URL: https://issues.apache.org/jira/browse/SPARK-23466

[jira] [Commented] (SPARK-17430) Spark task Hangs after OOM while DAG scheduler tries to serialize a task

[ https://issues.apache.org/jira/browse/SPARK-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369367#comment-16369367 ] Furcy Pin commented on SPARK-17430: --- Same problem here in spark 2.2.1 The job hangs forever, the GUI

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Attachment: (was: no-dyn-allocation-failed-no-statusUpdate.txt) > Application declines any

[jira] [Commented] (SPARK-23459) Improve the error message when unknown column is specified in partition columns

[ https://issues.apache.org/jira/browse/SPARK-23459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369346#comment-16369346 ] Kazuaki Ishizaki commented on SPARK-23459: -- Have you started working for this? If not, I can

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369330#comment-16369330 ] Steve Loughran commented on SPARK-23420: Can I note that if there's a colon in the path, it'd

[jira] [Commented] (SPARK-23465) Dataset.withAllColumnsRenamed should map all column names to a new one

[ https://issues.apache.org/jira/browse/SPARK-23465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369275#comment-16369275 ] Mihaly Toth commented on SPARK-23465: - I have started working on this. >

[jira] [Created] (SPARK-23465) Dataset.withAllColumnsRenamed should map all column names to a new one

Mihaly Toth created SPARK-23465: --- Summary: Dataset.withAllColumnsRenamed should map all column names to a new one Key: SPARK-23465 URL: https://issues.apache.org/jira/browse/SPARK-23465 Project: Spark

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman edited comment on SPARK-23423 at 2/19/18 3:47 PM: -- [~skonto] so

[jira] [Issue Comment Deleted] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Comment: was deleted (was: [~skonto] today I haven't managed to run with dynamic allocation on,

[jira] [Commented] (SPARK-19755) Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.

[ https://issues.apache.org/jira/browse/SPARK-19755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369227#comment-16369227 ] Igor Berman commented on SPARK-19755: - This Jira is very relevant for the case when running with  

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369221#comment-16369221 ] Igor Berman commented on SPARK-23423: - [~skonto] so I've run application today with relevant logs at

[jira] [Commented] (SPARK-8529) Set metadata for MinMaxScaler

[ https://issues.apache.org/jira/browse/SPARK-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369194#comment-16369194 ] Barry Becker commented on SPARK-8529: - Complementing the output metadata in what way? Need more info

[jira] [Commented] (SPARK-5377) Dynamically add jar into Spark Driver's classpath.

[ https://issues.apache.org/jira/browse/SPARK-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369112#comment-16369112 ] Shay Elbaz commented on SPARK-5377: --- +1 This seems like a very useful improvement and will save us many

[jira] [Updated] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command

[ https://issues.apache.org/jira/browse/SPARK-23464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Kurczych updated SPARK-23464: Description: Parameters passed to driver launching command in Mesos container are escaped

[jira] [Created] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command

Marcin Kurczych created SPARK-23464: --- Summary: MesosClusterScheduler double-escapes parameters to bash command Key: SPARK-23464 URL: https://issues.apache.org/jira/browse/SPARK-23464 Project: Spark

[jira] [Updated] (SPARK-23464) MesosClusterScheduler double-escapes parameters to bash command

[ https://issues.apache.org/jira/browse/SPARK-23464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Kurczych updated SPARK-23464: Description: Parameters passed to driver launching command in Mesos container are escaped

[jira] [Commented] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368968#comment-16368968 ] Marco Gaido commented on SPARK-23463: - sorry, what do you mean by blank values? Which is the type of

[jira] [Commented] (SPARK-23450) jars option in spark submit is documented in misleading way

[ https://issues.apache.org/jira/browse/SPARK-23450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368953#comment-16368953 ] Gregory Reshetniak commented on SPARK-23450: Hi guys, any update on this, please? > jars

[jira] [Assigned] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23415: Assignee: Apache Spark > BufferHolderSparkSubmitSuite is flaky >

[jira] [Assigned] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23415: Assignee: (was: Apache Spark) > BufferHolderSparkSubmitSuite is flaky >

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368895#comment-16368895 ] Apache Spark commented on SPARK-23415: -- User 'kiszk' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manan Bakshi updated SPARK-23463: - Comment: was deleted (was: I believe that the bug has something to do with CBO that was

[jira] [Updated] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manan Bakshi updated SPARK-23463: - Description: Filter operations were updated in Spark 2.2.0. Cost Based Optimizer was introduced

[jira] [Updated] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manan Bakshi updated SPARK-23463: - Description: I have a simple dataframe with some blank values as below ||dev||val|| |ALL|0.01|

[jira] [Commented] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368879#comment-16368879 ] Manan Bakshi commented on SPARK-23463: -- I believe that the bug has something to do with CBO that was

[jira] [Updated] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

[ https://issues.apache.org/jira/browse/SPARK-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manan Bakshi updated SPARK-23463: - Description: I have a simple dataframe with some blank values as below ||dev||val|| |ALL|0.01|

[jira] [Created] (SPARK-23463) Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition

Manan Bakshi created SPARK-23463: Summary: Filter operation fails to handle blank values and evicts rows that even satisfy the filtering condition Key: SPARK-23463 URL:

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368877#comment-16368877 ] Paul Doran commented on SPARK-21962: I have a working local version for Jaeger implementing the