[jira] [Commented] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-30 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265288#comment-15265288 ] Liang-Chi Hsieh commented on SPARK-14906: - Yes. > Move VectorUDT and MatrixUDT in PySpark to new

[jira] [Commented] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263537#comment-15263537 ] Liang-Chi Hsieh commented on SPARK-14906: - Are you working on this? If not, I will work on this

[jira] [Commented] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-27 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260220#comment-15260220 ] Liang-Chi Hsieh commented on SPARK-14906: - This should be able to do only after SPARK-14487 is

[jira] [Created] (SPARK-14951) Subexpression elimination in wholestage codegen version of TungstenAggregate

2016-04-27 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14951: --- Summary: Subexpression elimination in wholestage codegen version of TungstenAggregate Key: SPARK-14951 URL: https://issues.apache.org/jira/browse/SPARK-14951

[jira] [Updated] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14906: Issue Type: Sub-task (was: Improvement) Parent: SPARK-13944 > Move VectorUDT and

[jira] [Updated] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14906: Description: As we move VectorUDT and MatrixUDT in Scala to new ml package, the PySpark

[jira] [Created] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-04-25 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14906: --- Summary: Move VectorUDT and MatrixUDT in PySpark to new ML package Key: SPARK-14906 URL: https://issues.apache.org/jira/browse/SPARK-14906 Project: Spark

[jira] [Updated] (SPARK-14838) Implement statistics in SerializeFromObject to avoid failure when estimating sizeInBytes for ObjectType

2016-04-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14838: Summary: Implement statistics in SerializeFromObject to avoid failure when estimating

[jira] [Created] (SPARK-14838) Skip automatically broadcast a plan when it contains ObjectProducer

2016-04-21 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14838: --- Summary: Skip automatically broadcast a plan when it contains ObjectProducer Key: SPARK-14838 URL: https://issues.apache.org/jira/browse/SPARK-14838 Project:

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-04-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245427#comment-15245427 ] Liang-Chi Hsieh commented on SPARK-14083: - Based on [~joshrosen]'s code, I added some comments

[jira] [Closed] (SPARK-14432) Add API to calculate the approximate quantiles for multiple columns

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14432. --- Resolution: Duplicate > Add API to calculate the approximate quantiles for multiple columns

[jira] [Closed] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14627. --- Resolution: Won't Fix > Avoid shilfting encoder when delta is zero >

[jira] [Updated] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14627: Summary: Avoid shilfting encoder when delta is zero (was: In TypedAggregateExpression

[jira] [Updated] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14627: Description: We can also improve encoder's shift method to return itself when shift delta

[jira] [Reopened] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reopened SPARK-14627: - > Avoid shilfting encoder when delta is zero > -- >

[jira] [Closed] (SPARK-14627) In TypedAggregateExpression update method we call encoder.shift many times

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14627. --- Resolution: Won't Fix > In TypedAggregateExpression update method we call encoder.shift many

[jira] [Created] (SPARK-14627) In TypedAggregateExpression update method we call encoder.shift many times

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14627: --- Summary: In TypedAggregateExpression update method we call encoder.shift many times Key: SPARK-14627 URL: https://issues.apache.org/jira/browse/SPARK-14627

[jira] [Issue Comment Deleted] (SPARK-14592) Create table like

2016-04-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14592: Comment: was deleted (was: I am working on this...) > Create table like >

[jira] [Issue Comment Deleted] (SPARK-14592) Create table like

2016-04-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14592: Comment: was deleted (was: Will submit PR soon.) > Create table like > -

[jira] [Commented] (SPARK-14592) Create table like

2016-04-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239034#comment-15239034 ] Liang-Chi Hsieh commented on SPARK-14592: - I am working on this... > Create table like >

[jira] [Commented] (SPARK-14592) Create table like

2016-04-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239035#comment-15239035 ] Liang-Chi Hsieh commented on SPARK-14592: - Will submit PR soon. > Create table like >

[jira] [Created] (SPARK-14593) Make currentVars work with splitExpressions to enable whole stage codegen for large input columns

2016-04-13 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14593: --- Summary: Make currentVars work with splitExpressions to enable whole stage codegen for large input columns Key: SPARK-14593 URL:

[jira] [Issue Comment Deleted] (SPARK-14495) Distinct aggregation cannot be used in the having clause

2016-04-11 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14495: Comment: was deleted (was: I've tried this query in my PR

[jira] [Commented] (SPARK-14495) Distinct aggregation cannot be used in the having clause

2016-04-11 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235188#comment-15235188 ] Liang-Chi Hsieh commented on SPARK-14495: - I can' reproduce this bug with current master branch.

[jira] [Commented] (SPARK-14495) Distinct aggregation cannot be used in the having clause

2016-04-11 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234538#comment-15234538 ] Liang-Chi Hsieh commented on SPARK-14495: - I've tried this query in my PR

[jira] [Comment Edited] (SPARK-14520) ClasscastException thrown with spark.sql.parquet.enableVectorizedReader=true

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234496#comment-15234496 ] Liang-Chi Hsieh edited comment on SPARK-14520 at 4/11/16 5:09 AM: -- Hi

[jira] [Commented] (SPARK-14520) ClasscastException thrown with spark.sql.parquet.enableVectorizedReader=true

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234496#comment-15234496 ] Liang-Chi Hsieh commented on SPARK-14520: - Hi [~Rajesh Balamohan], I submitted a PR for this

[jira] [Commented] (SPARK-14253) Avoid registering temporary functions in Hive

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234492#comment-15234492 ] Liang-Chi Hsieh commented on SPARK-14253: - This can be closed now. > Avoid registering temporary

[jira] [Closed] (SPARK-9882) Priority-based scheduling for Spark applications

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-9882. -- Resolution: Won't Fix > Priority-based scheduling for Spark applications >

[jira] [Comment Edited] (SPARK-9882) Priority-based scheduling for Spark applications

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234028#comment-15234028 ] Liang-Chi Hsieh edited comment on SPARK-9882 at 4/10/16 9:54 AM: - This PR

[jira] [Commented] (SPARK-9882) Priority-based scheduling for Spark applications

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234028#comment-15234028 ] Liang-Chi Hsieh commented on SPARK-9882: This PR stays for a while. As the PR doesn't get the

[jira] [Commented] (SPARK-9882) Priority-based scheduling for Spark applications

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234027#comment-15234027 ] Liang-Chi Hsieh commented on SPARK-9882: I've updated the description. Thanks! > Priority-based

[jira] [Updated] (SPARK-9882) Priority-based scheduling for Spark applications

2016-04-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-9882: --- Description: We implement this patch because in our daily usage of Spark we found that

[jira] [Created] (SPARK-14487) User Defined Type registration without SQLUserDefinedType annotation

2016-04-08 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14487: --- Summary: User Defined Type registration without SQLUserDefinedType annotation Key: SPARK-14487 URL: https://issues.apache.org/jira/browse/SPARK-14487 Project:

[jira] [Created] (SPARK-14427) Support persisting partitioned data source relations in Hive compatible format

2016-04-06 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14427: --- Summary: Support persisting partitioned data source relations in Hive compatible format Key: SPARK-14427 URL: https://issues.apache.org/jira/browse/SPARK-14427

[jira] [Created] (SPARK-14354) Let Expand take name expressions and infer output attributes

2016-04-03 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14354: --- Summary: Let Expand take name expressions and infer output attributes Key: SPARK-14354 URL: https://issues.apache.org/jira/browse/SPARK-14354 Project: Spark

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222648#comment-15222648 ] Liang-Chi Hsieh commented on SPARK-14083: - I think this optimization now just consider the

[jira] [Updated] (SPARK-13321) Add nested union test cases

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13321: Description: The following SQL can not be parsed with current parser: {code} SELECT

[jira] [Updated] (SPARK-13321) Add nested union test cases

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13321: Issue Type: Test (was: Bug) > Add nested union test cases > --- >

[jira] [Updated] (SPARK-13321) Add nested union test cases

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13321: Summary: Add nested union test cases (was: Support nested UNION in parser) > Add nested

[jira] [Updated] (SPARK-13321) Add nested union test cases

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13321: Priority: Minor (was: Major) > Add nested union test cases > ---

[jira] [Commented] (SPARK-14129) [Table related commands] Alter table

2016-04-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221224#comment-15221224 ] Liang-Chi Hsieh commented on SPARK-14129: - TOK_ALTERTABLE_CLUSTER_SORT is also listed in

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-31 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219702#comment-15219702 ] Liang-Chi Hsieh commented on SPARK-14083: - Very interested in this work too. I would like to

[jira] [Commented] (SPARK-14253) Avoid registering temporary functions in Hive

2016-03-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217047#comment-15217047 ] Liang-Chi Hsieh commented on SPARK-14253: - In fact, current HiveFunctionRegistry doesn't handle

[jira] [Commented] (SPARK-14253) Avoid registering temporary functions in Hive

2016-03-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217044#comment-15217044 ] Liang-Chi Hsieh commented on SPARK-14253: - I already add this support in

[jira] [Created] (SPARK-14191) Fix Expand operator constraints

2016-03-28 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14191: --- Summary: Fix Expand operator constraints Key: SPARK-14191 URL: https://issues.apache.org/jira/browse/SPARK-14191 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-14157) Parse Drop Function DDL command

2016-03-25 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14157: --- Summary: Parse Drop Function DDL command Key: SPARK-14157 URL: https://issues.apache.org/jira/browse/SPARK-14157 Project: Spark Issue Type:

[jira] [Created] (SPARK-14156) Use executedPlan for HiveComparisonTest

2016-03-25 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14156: --- Summary: Use executedPlan for HiveComparisonTest Key: SPARK-14156 URL: https://issues.apache.org/jira/browse/SPARK-14156 Project: Spark Issue Type:

[jira] [Created] (SPARK-14111) Correct output nullability with constraints for logical plans

2016-03-23 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-14111: --- Summary: Correct output nullability with constraints for logical plans Key: SPARK-14111 URL: https://issues.apache.org/jira/browse/SPARK-14111 Project: Spark

[jira] [Closed] (SPARK-13903) Modify output nullability with constraints for Join and Filter operators

2016-03-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-13903. --- Resolution: Won't Fix > Modify output nullability with constraints for Join and Filter

[jira] [Updated] (SPARK-13995) Extract correct IsNotNull constraints for Expression

2016-03-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13995: Description: We infer relative `IsNotNull` constraints from logical plan's expressions in

[jira] [Updated] (SPARK-13995) Extract correct IsNotNull constraints for Expression

2016-03-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13995: Summary: Extract correct IsNotNull constraints for Expression (was: Constraints should

[jira] [Commented] (SPARK-13943) The behavior of sum(booleantype) in Spark DataFrames is not intuitive

2016-03-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204219#comment-15204219 ] Liang-Chi Hsieh commented on SPARK-13943: - Yes, I think so. > The behavior of sum(booleantype)

[jira] [Commented] (SPARK-13943) The behavior of sum(booleantype) in Spark DataFrames is not intuitive

2016-03-21 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203941#comment-15203941 ] Liang-Chi Hsieh commented on SPARK-13943: - Currently we seems don't support implicit type

[jira] [Closed] (SPARK-13839) Defer input evaluation and fix Cast issue in IsNotNull filtering for Filter codegen

2016-03-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-13839. --- Resolution: Won't Fix > Defer input evaluation and fix Cast issue in IsNotNull filtering for

[jira] [Created] (SPARK-13995) Constraints should take care of Cast

2016-03-19 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13995: --- Summary: Constraints should take care of Cast Key: SPARK-13995 URL: https://issues.apache.org/jira/browse/SPARK-13995 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-13996) Add more not null attributes for Filter codegen

2016-03-19 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13996: --- Summary: Add more not null attributes for Filter codegen Key: SPARK-13996 URL: https://issues.apache.org/jira/browse/SPARK-13996 Project: Spark Issue

[jira] [Comment Edited] (SPARK-13908) Limit not pushed down

2016-03-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202604#comment-15202604 ] Liang-Chi Hsieh edited comment on SPARK-13908 at 3/19/16 7:32 AM: --

[jira] [Commented] (SPARK-13908) Limit not pushed down

2016-03-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202604#comment-15202604 ] Liang-Chi Hsieh commented on SPARK-13908: - Rethink this issue, I think it should not related to

[jira] [Created] (SPARK-13930) Apply fast serialization on collect limit

2016-03-16 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13930: --- Summary: Apply fast serialization on collect limit Key: SPARK-13930 URL: https://issues.apache.org/jira/browse/SPARK-13930 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13903) Modify output nullability with constraints for Join and Filter operators

2016-03-15 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13903: Summary: Modify output nullability with constraints for Join and Filter operators (was:

[jira] [Updated] (SPARK-13903) Modify output nullability with constraints for Join and Filter operators

2016-03-15 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13903: Description: With constraints and optimization, we can make sure some outputs of a Join

[jira] [Created] (SPARK-13903) Modify output nullability with constraints for Join operator

2016-03-15 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13903: --- Summary: Modify output nullability with constraints for Join operator Key: SPARK-13903 URL: https://issues.apache.org/jira/browse/SPARK-13903 Project: Spark

[jira] [Closed] (SPARK-13854) Add constraints to outer join

2016-03-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-13854. --- Resolution: Not A Problem > Add constraints to outer join > - >

[jira] [Created] (SPARK-13854) Add constraints to outer join

2016-03-14 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13854: --- Summary: Add constraints to outer join Key: SPARK-13854 URL: https://issues.apache.org/jira/browse/SPARK-13854 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-13249) Filter null keys for inner join

2016-03-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192723#comment-15192723 ] Liang-Chi Hsieh commented on SPARK-13249: - I think this can be closed? > Filter null keys for

[jira] [Closed] (SPARK-13847) Defer the variable evaluation for Limit codegen

2016-03-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-13847. --- Resolution: Not A Problem > Defer the variable evaluation for Limit codegen >

[jira] [Created] (SPARK-13847) Defer the variable evaluation for Limit codegen

2016-03-13 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13847: --- Summary: Defer the variable evaluation for Limit codegen Key: SPARK-13847 URL: https://issues.apache.org/jira/browse/SPARK-13847 Project: Spark Issue

[jira] [Created] (SPARK-13839) Defer input evaluation and fix Cast issue in IsNotNull filtering for Filter codegen

2016-03-12 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13839: --- Summary: Defer input evaluation and fix Cast issue in IsNotNull filtering for Filter codegen Key: SPARK-13839 URL: https://issues.apache.org/jira/browse/SPARK-13839

[jira] [Created] (SPARK-13838) Clear variable code to prevent it to be re-evaluated in BoundAttribute

2016-03-12 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13838: --- Summary: Clear variable code to prevent it to be re-evaluated in BoundAttribute Key: SPARK-13838 URL: https://issues.apache.org/jira/browse/SPARK-13838

[jira] [Resolved] (SPARK-13771) Eliminate child of project if the project with no references to its child

2016-03-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-13771. - Resolution: Won't Fix > Eliminate child of project if the project with no references to

[jira] [Commented] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2016-03-09 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188593#comment-15188593 ] Liang-Chi Hsieh commented on SPARK-12117: - As I revisit this PR and find that this bug is already

[jira] [Created] (SPARK-13771) Eliminate child of project if the project with no references to its child

2016-03-09 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13771: --- Summary: Eliminate child of project if the project with no references to its child Key: SPARK-13771 URL: https://issues.apache.org/jira/browse/SPARK-13771

[jira] [Created] (SPARK-13742) Add non-iterator interface to RandomSampler

2016-03-08 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13742: --- Summary: Add non-iterator interface to RandomSampler Key: SPARK-13742 URL: https://issues.apache.org/jira/browse/SPARK-13742 Project: Spark Issue

[jira] [Created] (SPARK-13717) Let RandomSampler can sample with Java iterator

2016-03-07 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13717: --- Summary: Let RandomSampler can sample with Java iterator Key: SPARK-13717 URL: https://issues.apache.org/jira/browse/SPARK-13717 Project: Spark Issue

[jira] [Created] (SPARK-13674) Add wholestage codegen support to Sample

2016-03-04 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13674: --- Summary: Add wholestage codegen support to Sample Key: SPARK-13674 URL: https://issues.apache.org/jira/browse/SPARK-13674 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13635) Enable LimitPushdown optimizer rule because we have whole-stage codegen for Limit

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177445#comment-15177445 ] Liang-Chi Hsieh commented on SPARK-13635: - [~davies] Can you help update the Assignee field?

[jira] [Commented] (SPARK-13589) Flaky test: ParquetHadoopFsRelationSuite.test all data types - ByteType

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177437#comment-15177437 ] Liang-Chi Hsieh commented on SPARK-13589: - [~lian cheng] I think this is already solved in

[jira] [Comment Edited] (SPARK-13612) Multiplication of BigDecimal columns not working as expected

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177428#comment-15177428 ] Liang-Chi Hsieh edited comment on SPARK-13612 at 3/3/16 7:35 AM: - Because

[jira] [Commented] (SPARK-13612) Multiplication of BigDecimal columns not working as expected

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177428#comment-15177428 ] Liang-Chi Hsieh commented on SPARK-13612: - Because the internal type for BigDecimal would be

[jira] [Created] (SPARK-13636) Direct consume UnsafeRow in wholestage codegen plans

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13636: --- Summary: Direct consume UnsafeRow in wholestage codegen plans Key: SPARK-13636 URL: https://issues.apache.org/jira/browse/SPARK-13636 Project: Spark

[jira] [Created] (SPARK-13635) Enable LimitPushdown optimizer rule because we have whole-stage codegen for Limit

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13635: --- Summary: Enable LimitPushdown optimizer rule because we have whole-stage codegen for Limit Key: SPARK-13635 URL: https://issues.apache.org/jira/browse/SPARK-13635

[jira] [Created] (SPARK-13616) Let SQLBuilder convert logical plan without a Project on top of it

2016-03-02 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13616: --- Summary: Let SQLBuilder convert logical plan without a Project on top of it Key: SPARK-13616 URL: https://issues.apache.org/jira/browse/SPARK-13616 Project:

[jira] [Commented] (SPARK-13511) Add wholestage codegen for limit

2016-03-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174919#comment-15174919 ] Liang-Chi Hsieh commented on SPARK-13511: - [~davies] Can you help update the Assignee field?

[jira] [Created] (SPARK-13537) Fix readBytes in VectorizedPlainValuesReader

2016-02-28 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13537: --- Summary: Fix readBytes in VectorizedPlainValuesReader Key: SPARK-13537 URL: https://issues.apache.org/jira/browse/SPARK-13537 Project: Spark Issue

[jira] [Updated] (SPARK-13530) Add ShortType support to UnsafeRowParquetRecordReader

2016-02-27 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-13530: Description: By enabling vectorized parquet scanner by default, the unit test

[jira] [Created] (SPARK-13530) Add ShortType support to UnsafeRowParquetRecordReader

2016-02-27 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13530: --- Summary: Add ShortType support to UnsafeRowParquetRecordReader Key: SPARK-13530 URL: https://issues.apache.org/jira/browse/SPARK-13530 Project: Spark

[jira] [Created] (SPARK-13511) Add wholestage codegen for limit

2016-02-26 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13511: --- Summary: Add wholestage codegen for limit Key: SPARK-13511 URL: https://issues.apache.org/jira/browse/SPARK-13511 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13383) Keep broadcast hint after column pruning

2016-02-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166611#comment-15166611 ] Liang-Chi Hsieh commented on SPARK-13383: - [~marmbrus] Can you help set the Assignee field?

[jira] [Created] (SPARK-13472) Fix unstable Kmeans test in R

2016-02-24 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13472: --- Summary: Fix unstable Kmeans test in R Key: SPARK-13472 URL: https://issues.apache.org/jira/browse/SPARK-13472 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-13466) Don't introduce redundant project with colum pruning rule

2016-02-23 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13466: --- Summary: Don't introduce redundant project with colum pruning rule Key: SPARK-13466 URL: https://issues.apache.org/jira/browse/SPARK-13466 Project: Spark

[jira] [Commented] (SPARK-13358) Retrieve grep path when doing Benchmark

2016-02-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160145#comment-15160145 ] Liang-Chi Hsieh commented on SPARK-13358: - [~davies] Can you help update the Assignee field?

[jira] [Created] (SPARK-13464) Fix failed test test_reduce_by_key_and_window_with_none_invFunc in pyspark/streaming

2016-02-23 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13464: --- Summary: Fix failed test test_reduce_by_key_and_window_with_none_invFunc in pyspark/streaming Key: SPARK-13464 URL: https://issues.apache.org/jira/browse/SPARK-13464

[jira] [Created] (SPARK-13384) Keep attribute qualifiers after dedup in Analyzer

2016-02-18 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13384: --- Summary: Keep attribute qualifiers after dedup in Analyzer Key: SPARK-13384 URL: https://issues.apache.org/jira/browse/SPARK-13384 Project: Spark

[jira] [Created] (SPARK-13383) Keep broadcast hint after column pruning

2016-02-18 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-13383: --- Summary: Keep broadcast hint after column pruning Key: SPARK-13383 URL: https://issues.apache.org/jira/browse/SPARK-13383 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2016-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151720#comment-15151720 ] Liang-Chi Hsieh edited comment on SPARK-1 at 2/18/16 4:13 AM: -- The

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2016-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151720#comment-15151720 ] Liang-Chi Hsieh commented on SPARK-1: - Yes. I agree that when user provides a specific seed

[jira] [Comment Edited] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2016-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150072#comment-15150072 ] Liang-Chi Hsieh edited comment on SPARK-1 at 2/17/16 8:07 AM: -- Isn't

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2016-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150072#comment-15150072 ] Liang-Chi Hsieh commented on SPARK-1: - Isn't weird? Suppose each partition should have

[jira] [Commented] (SPARK-13333) DataFrame filter + randn + unionAll has bad interaction

2016-02-16 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150050#comment-15150050 ] Liang-Chi Hsieh commented on SPARK-1: - But when you set deterministic to true, your each data

<    6   7   8   9   10   11   12   13   14   >