[jira] [Commented] (SPARK-11058) failed spark job reports on YARN as successful

[ https://issues.apache.org/jira/browse/SPARK-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16218994#comment-16218994 ] Nick Dimiduk commented on SPARK-11058: -- [~srowen], [~vanzin] this is still considere

[jira] [Commented] (SPARK-11058) failed spark job reports on YARN as successful

[ https://issues.apache.org/jira/browse/SPARK-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219280#comment-16219280 ] Nick Dimiduk commented on SPARK-11058: -- This is a Spark issue though? Spark could fi

[jira] [Commented] (SPARK-17593) list files on s3 very slow

[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244887#comment-16244887 ] Nick Dimiduk commented on SPARK-17593: -- So the fix in Hadoop 2.8 is for any variant

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864798#comment-15864798 ] Nick Dimiduk commented on SPARK-13219: -- Now that all the subtasks on SPARK-12957 are

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864796#comment-15864796 ] Nick Dimiduk commented on SPARK-12957: -- I'm trying to understand the current state o

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866210#comment-15866210 ] Nick Dimiduk commented on SPARK-13219: -- bq. The result is that no matter the number

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866239#comment-15866239 ] Nick Dimiduk commented on SPARK-12957: -- [~sameerag] thanks for the comment. From a n

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866262#comment-15866262 ] Nick Dimiduk commented on SPARK-13219: -- I would implement this manually by materiali

[jira] [Created] (SPARK-19609) Broadcast joins should pushdown join constraints as Filter to the larger relation

Nick Dimiduk created SPARK-19609: Summary: Broadcast joins should pushdown join constraints as Filter to the larger relation Key: SPARK-19609 URL: https://issues.apache.org/jira/browse/SPARK-19609 Pro

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868366#comment-15868366 ] Nick Dimiduk commented on SPARK-12957: -- Filed SPARK-19609. IMHO, it would be another

[jira] [Created] (SPARK-19614) add type-preserving null function

Nick Dimiduk created SPARK-19614: Summary: add type-preserving null function Key: SPARK-19614 URL: https://issues.apache.org/jira/browse/SPARK-19614 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-19615) Provide Dataset union convenience for divergent schema

Nick Dimiduk created SPARK-19615: Summary: Provide Dataset union convenience for divergent schema Key: SPARK-19615 URL: https://issues.apache.org/jira/browse/SPARK-19615 Project: Spark Issue

[jira] [Updated] (SPARK-19615) Provide Dataset union convenience for divergent schema

[ https://issues.apache.org/jira/browse/SPARK-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated SPARK-19615: - Priority: Major (was: Minor) > Provide Dataset union convenience for divergent schema >

[jira] [Commented] (SPARK-19615) Provide Dataset union convenience for divergent schema

[ https://issues.apache.org/jira/browse/SPARK-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870513#comment-15870513 ] Nick Dimiduk commented on SPARK-19615: -- IMHO, a union operation should be as generou

[jira] [Closed] (SPARK-19614) add type-preserving null function

[ https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk closed SPARK-19614. Resolution: Invalid > add type-preserving null function > - > >

[jira] [Commented] (SPARK-19614) add type-preserving null function

[ https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870525#comment-15870525 ] Nick Dimiduk commented on SPARK-19614: -- {{lit(null).cast(type)}} does exactly what I

[jira] [Created] (SPARK-19638) Filter pushdown not working for struct fields

Nick Dimiduk created SPARK-19638: Summary: Filter pushdown not working for struct fields Key: SPARK-19638 URL: https://issues.apache.org/jira/browse/SPARK-19638 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19638) Filter pushdown not working for struct fields

[ https://issues.apache.org/jira/browse/SPARK-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872060#comment-15872060 ] Nick Dimiduk commented on SPARK-19638: -- [~maropu] I have placed a breakpoint in the

[jira] [Commented] (SPARK-19638) Filter pushdown not working for struct fields

[ https://issues.apache.org/jira/browse/SPARK-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872413#comment-15872413 ] Nick Dimiduk commented on SPARK-19638: -- Debugging. I'm looking at the match expressi

[jira] [Commented] (SPARK-19615) Provide Dataset union convenience for divergent schema

[ https://issues.apache.org/jira/browse/SPARK-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875137#comment-15875137 ] Nick Dimiduk commented on SPARK-19615: -- Thanks for taking a look [~hyukjin.kwon]. Th

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875178#comment-15875178 ] Nick Dimiduk commented on SPARK-17636: -- Per [~maropu]'s comments on SPARK-19638, thi

[jira] [Commented] (SPARK-19638) Filter pushdown not working for struct fields

[ https://issues.apache.org/jira/browse/SPARK-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875184#comment-15875184 ] Nick Dimiduk commented on SPARK-19638: -- Okay [~maropu], I'll continue discussion ove

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875188#comment-15875188 ] Nick Dimiduk commented on SPARK-17636: -- I've proposed a PR for fixing this issue. Us

[jira] [Commented] (SPARK-11373) Add metrics to the History Server and providers

[ https://issues.apache.org/jira/browse/SPARK-11373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252776#comment-16252776 ] Nick Dimiduk commented on SPARK-11373: -- I'm chasing a goose through the wild and hav

[jira] [Commented] (SPARK-27599) DataFrameWriter.partitionBy should be optional when writing to a hive table

[ https://issues.apache.org/jira/browse/SPARK-27599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845378#comment-16845378 ] Nick Dimiduk commented on SPARK-27599: -- Sure [~Alexander_Fedosov]. Hive DDL let's y

[jira] [Commented] (SPARK-19609) Broadcast joins should pushdown join constraints as Filter to the larger relation

[ https://issues.apache.org/jira/browse/SPARK-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947001#comment-16947001 ] Nick Dimiduk commented on SPARK-19609: -- Hi [~hyukjin.kwon], mind adding a comment a

[jira] [Created] (SPARK-27597) RuntimeConfig should be serializable

Nick Dimiduk created SPARK-27597: Summary: RuntimeConfig should be serializable Key: SPARK-27597 URL: https://issues.apache.org/jira/browse/SPARK-27597 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27597) RuntimeConfig should be serializable

[ https://issues.apache.org/jira/browse/SPARK-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829612#comment-16829612 ] Nick Dimiduk commented on SPARK-27597: -- It would nice nice if there was an API buil

[jira] [Updated] (SPARK-27599) DataFrameWriter.partitionBy should be optional when writing to a hive table

[ https://issues.apache.org/jira/browse/SPARK-27599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated SPARK-27599: - Summary: DataFrameWriter.partitionBy should be optional when writing to a hive table (was: Data

[jira] [Created] (SPARK-27599) DataFrameWriter$partitionBy should be optional when writing to a hive table

Nick Dimiduk created SPARK-27599: Summary: DataFrameWriter$partitionBy should be optional when writing to a hive table Key: SPARK-27599 URL: https://issues.apache.org/jira/browse/SPARK-27599 Project:

[jira] [Commented] (SPARK-27597) RuntimeConfig should be serializable

[ https://issues.apache.org/jira/browse/SPARK-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830564#comment-16830564 ] Nick Dimiduk commented on SPARK-27597: -- {quote}bq. Do you want to access {{SparkSes

[jira] [Commented] (SPARK-39753) Broadcast joins should pushdown join constraints as Filter to the larger relation

[ https://issues.apache.org/jira/browse/SPARK-39753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17576643#comment-17576643 ] Nick Dimiduk commented on SPARK-39753: -- Linking to the original issue. > Broadcast