[jira] [Commented] (SPARK-22456) Add new function dayofweek

[ https://issues.apache.org/jira/browse/SPARK-22456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240331#comment-16240331 ] Michael Styles commented on SPARK-22456: This functionality is part of the SQL st

[jira] [Created] (SPARK-22456) Add new function dayofweek

Michael Styles created SPARK-22456: -- Summary: Add new function dayofweek Key: SPARK-22456 URL: https://issues.apache.org/jira/browse/SPARK-22456 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-21692) Modify PythonUDF to support nullability

[ https://issues.apache.org/jira/browse/SPARK-21692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123461#comment-16123461 ] Michael Styles commented on SPARK-21692: https://github.com/apache/spark/pull/189

[jira] [Created] (SPARK-21692) Modify PythonUDF to support nullability

Michael Styles created SPARK-21692: -- Summary: Modify PythonUDF to support nullability Key: SPARK-21692 URL: https://issues.apache.org/jira/browse/SPARK-21692 Project: Spark Issue Type: Impro

[jira] [Commented] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066363#comment-16066363 ] Michael Styles commented on SPARK-17091: n Parquet 1.7, there as a bug involving

[jira] [Resolved] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles resolved SPARK-21218. Resolution: Duplicate > Convert IN predicate to equivalent Parquet filter > ---

[jira] [Comment Edited] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065413#comment-16065413 ] Michael Styles edited comment on SPARK-17091 at 6/27/17 8:26 PM: --

[jira] [Updated] (SPARK-17091) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17091: --- Summary: Convert IN predicate to equivalent Parquet filter (was: ParquetFilters rewrite IN t

[jira] [Updated] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17091: --- Attachment: IN Predicate.png OR Predicate.png > ParquetFilters rewrite IN to

[jira] [Commented] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065413#comment-16065413 ] Michael Styles commented on SPARK-17091: By not pushing the filter to Parquet, ar

[jira] [Reopened] (SPARK-17091) ParquetFilters rewrite IN to OR of Eq

[ https://issues.apache.org/jira/browse/SPARK-17091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles reopened SPARK-17091: > ParquetFilters rewrite IN to OR of Eq > - > >

[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065210#comment-16065210 ] Michael Styles commented on SPARK-21218: In Parquet 1.7, there as a bug involving

[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065142#comment-16065142 ] Michael Styles commented on SPARK-21218: [~hyukjin.kwon] Not sure I understand wh

[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 12:17 PM: -

[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 10:30 AM: -

[jira] [Comment Edited] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles edited comment on SPARK-21218 at 6/27/17 10:25 AM: -

[jira] [Commented] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064608#comment-16064608 ] Michael Styles commented on SPARK-21218: By not pushing the filter to Parquet, ar

[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: (was: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Det

[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: (was: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Det

[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: IN Predicate.png OR Predicate.png > Convert IN predicate to equiv

[jira] [Updated] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

[ https://issues.apache.org/jira/browse/SPARK-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-21218: --- Attachment: Starscream Console on OTT---Michael-Styles---MBP-15-inch-Mid-2015 - Details for Q

[jira] [Created] (SPARK-21218) Convert IN predicate to equivalent Parquet filter

Michael Styles created SPARK-21218: -- Summary: Convert IN predicate to equivalent Parquet filter Key: SPARK-21218 URL: https://issues.apache.org/jira/browse/SPARK-21218 Project: Spark Issue T

[jira] [Created] (SPARK-20636) Eliminate unnecessary shuffle with adjacent Window expressions

Michael Styles created SPARK-20636: -- Summary: Eliminate unnecessary shuffle with adjacent Window expressions Key: SPARK-20636 URL: https://issues.apache.org/jira/browse/SPARK-20636 Project: Spark

[jira] [Updated] (SPARK-20463) Add support for IS [NOT] DISTINCT FROM to SPARK SQL

[ https://issues.apache.org/jira/browse/SPARK-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-20463: --- Description: Add support for the SQL standard distinct predicate to SPARK SQL. {noformat} I

[jira] [Updated] (SPARK-20463) Add support for IS [NOT] DISTINCT FROM to SPARK SQL

[ https://issues.apache.org/jira/browse/SPARK-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-20463: --- Component/s: (was: PySpark) SQL Summary: Add support for IS [NOT]

[jira] [Created] (SPARK-20463) Expose SPARK SQL <=> operator in PySpark

Michael Styles created SPARK-20463: -- Summary: Expose SPARK SQL <=> operator in PySpark Key: SPARK-20463 URL: https://issues.apache.org/jira/browse/SPARK-20463 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-20413) New Optimizer Hint to prevent collapsing of adjacent projections

[ https://issues.apache.org/jira/browse/SPARK-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-20413: --- Description: I am proposing that a new optimizer hint called NO_COLLAPSE be introduced. This

[jira] [Updated] (SPARK-20413) New Optimizer Hint to prevent collapsing of adjacent projections

[ https://issues.apache.org/jira/browse/SPARK-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-20413: --- Description: I am proposing that a new optimizer hint called NO_COLLAPSE be introduced. This

[jira] [Updated] (SPARK-20413) New Optimizer Hint to prevent collapsing of adjacent projections

[ https://issues.apache.org/jira/browse/SPARK-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-20413: --- Description: I am proposing that a new optimizer hint called NO_COLLAPSE be introduced. This

[jira] [Created] (SPARK-20413) New Optimizer Hint to prevent collapsing of adjacent projections

Michael Styles created SPARK-20413: -- Summary: New Optimizer Hint to prevent collapsing of adjacent projections Key: SPARK-20413 URL: https://issues.apache.org/jira/browse/SPARK-20413 Project: Spark

[jira] [Created] (SPARK-20350) Apply Complementation Laws during boolean expression simplification

Michael Styles created SPARK-20350: -- Summary: Apply Complementation Laws during boolean expression simplification Key: SPARK-20350 URL: https://issues.apache.org/jira/browse/SPARK-20350 Project: Spar

[jira] [Updated] (SPARK-19851) Add support for EVERY and ANY (SOME) aggregates

[ https://issues.apache.org/jira/browse/SPARK-19851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-19851: --- Description: Add support for EVERY and ANY (SOME) aggregates. - EVERY returns true if all in

[jira] [Commented] (SPARK-19851) Add support for EVERY and ANY (SOME) aggregates

[ https://issues.apache.org/jira/browse/SPARK-19851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15899988#comment-15899988 ] Michael Styles commented on SPARK-19851: https://github.com/apache/spark/pull/171

[jira] [Created] (SPARK-19851) Add support for EVERY and ANY (SOME) aggregates

Michael Styles created SPARK-19851: -- Summary: Add support for EVERY and ANY (SOME) aggregates Key: SPARK-19851 URL: https://issues.apache.org/jira/browse/SPARK-19851 Project: Spark Issue Typ

[jira] [Commented] (SPARK-17035) Conversion of datetime.max to microseconds produces incorrect value

[ https://issues.apache.org/jira/browse/SPARK-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420534#comment-15420534 ] Michael Styles commented on SPARK-17035: I have a fix for this issue if you would

[jira] [Created] (SPARK-17037) distinct() operator fails on Dataframe with column names containing periods

Michael Styles created SPARK-17037: -- Summary: distinct() operator fails on Dataframe with column names containing periods Key: SPARK-17037 URL: https://issues.apache.org/jira/browse/SPARK-17037 Proje

[jira] [Updated] (SPARK-17035) Conversion of datetime.max to microseconds produces incorrect value

[ https://issues.apache.org/jira/browse/SPARK-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17035: --- Description: Conversion of datetime.max to microseconds produces incorrect value. For exampl

[jira] [Updated] (SPARK-17035) Conversion of datetime.max to microseconds produces incorrect value

[ https://issues.apache.org/jira/browse/SPARK-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Styles updated SPARK-17035: --- Priority: Minor (was: Major) > Conversion of datetime.max to microseconds produces incorrect

[jira] [Created] (SPARK-17035) Conversion of datetime.max to microseconds produces incorrect value

Michael Styles created SPARK-17035: -- Summary: Conversion of datetime.max to microseconds produces incorrect value Key: SPARK-17035 URL: https://issues.apache.org/jira/browse/SPARK-17035 Project: Spar