[jira] [Commented] (SPARK-1888) enhance MEMORY_AND_DISK mode by dropping blocks in parallel

2014-05-21 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004538#comment-14004538 ] Wenchen Fan commented on SPARK-1888: I introduce a new member for `Entry` (var

[jira] [Created] (SPARK-1912) Compression memory issue during shuffle

2014-05-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-1912: -- Summary: Compression memory issue during shuffle Key: SPARK-1912 URL: https://issues.apache.org/jira/browse/SPARK-1912 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-1912) Compression memory issue during reduce

2014-05-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-1912: --- Summary: Compression memory issue during reduce (was: Compression memory issue during shuffle)

[jira] [Commented] (SPARK-2133) FileNotFoundException in BlockObjectWriter

2014-06-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030480#comment-14030480 ] Wenchen Fan commented on SPARK-2133: The exception happened when spark want to write a

[jira] [Created] (SPARK-3698) Case sensitive check in spark sql is incompleted.

2014-09-26 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-3698: -- Summary: Case sensitive check in spark sql is incompleted. Key: SPARK-3698 URL: https://issues.apache.org/jira/browse/SPARK-3698 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-5278: -- Summary: ambiguous reference to fields in Spark SQL is incompleted Key: SPARK-5278 URL: https://issues.apache.org/jira/browse/SPARK-5278 Project: Spark Issue

[jira] [Updated] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5278: --- Description: for json string like {a: {b: 1, B: 2}} The SQL `SELECT a.b from t` will report error for

[jira] [Updated] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5278: --- Description: for json string like {a: {b: 1, B: 2}} The SQL `SELECT a.b from t` will report error for

[jira] [Updated] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5278: --- Description: for json string like {a: {b: 1, B: 2}} The SQL `SELECT a.b from t` will report error for

[jira] [Updated] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5278: --- Description: at hive context for json string like {a: {b: 1, B: 2}} The SQL `SELECT a.b from t` will

[jira] [Updated] (SPARK-5278) ambiguous reference to fields in Spark SQL is incompleted

2015-01-15 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5278: --- Description: at hive context for json string like {code}{a: {b: 1, B: 2}}{code} The SQL `SELECT a.b

[jira] [Updated] (SPARK-5374) abstract RDD's DAG graph iteration in DAGScheduler

2015-01-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5374: --- Summary: abstract RDD's DAG graph iteration in DAGScheduler (was: abstract RDD's DAG graph iteration

[jira] [Updated] (SPARK-5374) abstract RDD's DAG graph iteration in

2015-01-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-5374: --- Summary: abstract RDD's DAG graph iteration in (was: abstract RDD's DAG graph iteration) abstract

[jira] [Created] (SPARK-5374) abstract RDD's DAG graph iteration

2015-01-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-5374: -- Summary: abstract RDD's DAG graph iteration Key: SPARK-5374 URL: https://issues.apache.org/jira/browse/SPARK-5374 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-6394) cleanup BlockManager companion object and improve the getCacheLocs method in DAGScheduler

2015-03-17 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-6394: -- Summary: cleanup BlockManager companion object and improve the getCacheLocs method in DAGScheduler Key: SPARK-6394 URL: https://issues.apache.org/jira/browse/SPARK-6394

[jira] [Created] (SPARK-6138) Enhance the `toArray` method in `SizeTrackingVector`

2015-03-03 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-6138: -- Summary: Enhance the `toArray` method in `SizeTrackingVector` Key: SPARK-6138 URL: https://issues.apache.org/jira/browse/SPARK-6138 Project: Spark Issue Type:

[jira] [Updated] (SPARK-6898) Special chars in column names is broken

2015-04-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-6898: --- Description: This function is added a long time ago, but it's not complete, it will fail if we have

[jira] [Updated] (SPARK-6898) Special chars in column names is broken

2015-04-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-6898: --- Description: This function is added a long time ago, but it's not complete, it will fail if we have

[jira] [Updated] (SPARK-6898) Special chars in column names is broken

2015-04-14 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-6898: --- Description: This function is added a long time ago, but it's not complete, it will fail if we have

[jira] [Created] (SPARK-6898) Special chars in column names is broken

2015-04-14 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-6898: -- Summary: Special chars in column names is broken Key: SPARK-6898 URL: https://issues.apache.org/jira/browse/SPARK-6898 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-7133) Implement struct, array, and map field accessor using apply in Scala and __getitem__ in Python

2015-04-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512968#comment-14512968 ] Wenchen Fan edited comment on SPARK-7133 at 4/26/15 10:47 AM: --

[jira] [Comment Edited] (SPARK-7133) Implement struct, array, and map field accessor using apply in Scala and __getitem__ in Python

2015-04-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512968#comment-14512968 ] Wenchen Fan edited comment on SPARK-7133 at 4/26/15 10:57 AM: --

[jira] [Created] (SPARK-7153) support Long type ordinal in GetItem

2015-04-26 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7153: -- Summary: support Long type ordinal in GetItem Key: SPARK-7153 URL: https://issues.apache.org/jira/browse/SPARK-7153 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7133) Implement struct, array, and map field accessor using apply in Scala and __getitem__ in Python

2015-04-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512968#comment-14512968 ] Wenchen Fan commented on SPARK-7133: Hi [~rxin] , the reason why we didn't generalize

[jira] [Updated] (SPARK-7153) support Long type ordinal in GetItem

2015-04-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-7153: --- Description: In GetItem, we will cast the ordinal into Int first. However, if the ordinal is Long

[jira] [Created] (SPARK-7067) Can't resolve nested column in ORDER BY

2015-04-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7067: -- Summary: Can't resolve nested column in ORDER BY Key: SPARK-7067 URL: https://issues.apache.org/jira/browse/SPARK-7067 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-7067) Can't resolve nested column in ORDER BY

2015-04-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-7067: --- Description: In order to avoid breaking existing HiveQL queries, the current way we resolve column

[jira] [Updated] (SPARK-7067) Can't resolve nested column in ORDER BY

2015-04-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-7067: --- Description: In order to avoid breaking existing HiveQL queries, the current way we resolve column

[jira] [Updated] (SPARK-7067) Can't resolve nested column in ORDER BY

2015-04-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-7067: --- Description: In order to avoid breaking existing HiveQL queries, the current way we resolve column

[jira] [Created] (SPARK-7114) parse error for DataFrame.filter after aggregate

2015-04-24 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7114: -- Summary: parse error for DataFrame.filter after aggregate Key: SPARK-7114 URL: https://issues.apache.org/jira/browse/SPARK-7114 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-7114) parse error for DataFrame.filter after aggregate

2015-04-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-7114: --- Description: DataFrame.filter has 2 overloaded versions. One of it accept String parameter to

[jira] [Commented] (SPARK-7114) parse error for DataFrame.filter after aggregate

2015-04-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510592#comment-14510592 ] Wenchen Fan commented on SPARK-7114: cc [~rxin] parse error for DataFrame.filter

[jira] [Created] (SPARK-7734) make explode support struct type

2015-05-19 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7734: -- Summary: make explode support struct type Key: SPARK-7734 URL: https://issues.apache.org/jira/browse/SPARK-7734 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-8164) transformExpressions should support nested expression sequence

2015-06-08 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8164: -- Summary: transformExpressions should support nested expression sequence Key: SPARK-8164 URL: https://issues.apache.org/jira/browse/SPARK-8164 Project: Spark

[jira] [Created] (SPARK-8075) apply type checking interface to more expressions

2015-06-03 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8075: -- Summary: apply type checking interface to more expressions Key: SPARK-8075 URL: https://issues.apache.org/jira/browse/SPARK-8075 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8075) apply type checking interface to more expressions

2015-06-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8075: --- Description: As https://github.com/apache/spark/pull/6405 has been merged, we need to apply the type

[jira] [Created] (SPARK-8104) move the auto alias logic into Analyzer

2015-06-04 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8104: -- Summary: move the auto alias logic into Analyzer Key: SPARK-8104 URL: https://issues.apache.org/jira/browse/SPARK-8104 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8588) Could not use concat with UDF in where clause

2015-06-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602401#comment-14602401 ] Wenchen Fan commented on SPARK-8588: cc [~marmbrus] this issue has already been fixed

[jira] [Created] (SPARK-8635) improve performance of CatalystTypeConverters

2015-06-25 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8635: -- Summary: improve performance of CatalystTypeConverters Key: SPARK-8635 URL: https://issues.apache.org/jira/browse/SPARK-8635 Project: Spark Issue Type:

[jira] [Created] (SPARK-8589) cleanup DateTimeUtils

2015-06-24 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8589: -- Summary: cleanup DateTimeUtils Key: SPARK-8589 URL: https://issues.apache.org/jira/browse/SPARK-8589 Project: Spark Issue Type: Bug Components: SQL

[jira] [Created] (SPARK-8590) add code gen for ExtractValue

2015-06-24 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8590: -- Summary: add code gen for ExtractValue Key: SPARK-8590 URL: https://issues.apache.org/jira/browse/SPARK-8590 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8589) cleanup DateTimeUtils

2015-06-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8589: --- Issue Type: Improvement (was: Bug) cleanup DateTimeUtils -

[jira] [Updated] (SPARK-8590) add code gen for ExtractValue

2015-06-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8590: --- Issue Type: Improvement (was: Bug) add code gen for ExtractValue -

[jira] [Created] (SPARK-8620) cleanup CodeGenContext

2015-06-24 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8620: -- Summary: cleanup CodeGenContext Key: SPARK-8620 URL: https://issues.apache.org/jira/browse/SPARK-8620 Project: Spark Issue Type: Bug Components: SQL

[jira] [Updated] (SPARK-8620) cleanup CodeGenContext

2015-06-24 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8620: --- Issue Type: Improvement (was: Bug) cleanup CodeGenContext --

[jira] [Created] (SPARK-8679) remove dummy java class BaseRow and BaseMutableRow

2015-06-27 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8679: -- Summary: remove dummy java class BaseRow and BaseMutableRow Key: SPARK-8679 URL: https://issues.apache.org/jira/browse/SPARK-8679 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-8371) improve unit test for MaxOf and MinOf

2015-06-15 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8371: -- Summary: improve unit test for MaxOf and MinOf Key: SPARK-8371 URL: https://issues.apache.org/jira/browse/SPARK-8371 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-7964) The BooleanCasts rule in HiveTypeCoercion is useless

2015-05-30 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7964: -- Summary: The BooleanCasts rule in HiveTypeCoercion is useless Key: SPARK-7964 URL: https://issues.apache.org/jira/browse/SPARK-7964 Project: Spark Issue Type:

[jira] [Created] (SPARK-7984) do type coercion for key and when expressions in CaseKeyWhen

2015-05-31 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7984: -- Summary: do type coercion for key and when expressions in CaseKeyWhen Key: SPARK-7984 URL: https://issues.apache.org/jira/browse/SPARK-7984 Project: Spark

[jira] [Created] (SPARK-7952) equality check between boolean type and numeric type is broken.

2015-05-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-7952: -- Summary: equality check between boolean type and numeric type is broken. Key: SPARK-7952 URL: https://issues.apache.org/jira/browse/SPARK-7952 Project: Spark

[jira] [Created] (SPARK-8759) add default eval to binary and unary expression according to default behavior of nullable

2015-07-01 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8759: -- Summary: add default eval to binary and unary expression according to default behavior of nullable Key: SPARK-8759 URL: https://issues.apache.org/jira/browse/SPARK-8759

[jira] [Updated] (SPARK-8796) make sure SparkPlan is only executed at driver side

2015-07-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8796: --- Summary: make sure SparkPlan is only executed at driver side (was: mark child as transient in

[jira] [Issue Comment Deleted] (SPARK-8796) make sure SparkPlan is only executed at driver side

2015-07-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8796: --- Comment: was deleted (was: we can still avoid transient in `TakeOrderedAndProject`) make sure

[jira] [Created] (SPARK-8692) improve date and timestamp handling

2015-06-28 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8692: -- Summary: improve date and timestamp handling Key: SPARK-8692 URL: https://issues.apache.org/jira/browse/SPARK-8692 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-8837) support using keyword in column name

2015-07-06 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8837: -- Summary: support using keyword in column name Key: SPARK-8837 URL: https://issues.apache.org/jira/browse/SPARK-8837 Project: Spark Issue Type: Improvement

[jira] [Reopened] (SPARK-8796) clean closure at InsertIntoHiveTable and remove transient

2015-07-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-8796: we can still avoid transient for `TakeOrderedAndProject` clean closure at InsertIntoHiveTable and

[jira] [Comment Edited] (SPARK-8796) mark child as transient in InsertIntoHiveTable

2015-07-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612265#comment-14612265 ] Wenchen Fan edited comment on SPARK-8796 at 7/2/15 5:34 PM: we

[jira] [Updated] (SPARK-8796) mark child as transient in InsertIntoHiveTable

2015-07-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8796: --- Summary: mark child as transient in InsertIntoHiveTable (was: clean closure at InsertIntoHiveTable

[jira] [Created] (SPARK-8796) clean closure at InsertIntoHiveTable and remove transient

2015-07-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8796: -- Summary: clean closure at InsertIntoHiveTable and remove transient Key: SPARK-8796 URL: https://issues.apache.org/jira/browse/SPARK-8796 Project: Spark Issue

[jira] [Closed] (SPARK-8796) clean closure at InsertIntoHiveTable and remove transient

2015-07-02 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan closed SPARK-8796. -- Resolution: Not A Problem clean closure at InsertIntoHiveTable and remove transient

[jira] [Updated] (SPARK-8692) refactor the handling of date and timestamp

2015-06-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-8692: --- Summary: refactor the handling of date and timestamp (was: improve date and timestamp handling)

[jira] [Created] (SPARK-8747) fix EqualNullSafe for binary type

2015-06-30 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8747: -- Summary: fix EqualNullSafe for binary type Key: SPARK-8747 URL: https://issues.apache.org/jira/browse/SPARK-8747 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7562) Improve error reporting for expression data type mismatch

2015-05-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558836#comment-14558836 ] Wenchen Fan commented on SPARK-7562: After thinking about it, I think we can't just

[jira] [Updated] (SPARK-9738) remove FromUnsafe and add its codegen version to GenerateSafe

2015-08-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9738: --- Description: In https://github.com/apache/spark/pull/7752 we added `FromUnsafe` to convert nexted

[jira] [Commented] (SPARK-9738) remove FromUnsafe and add its codegen version to GenerateSafe

2015-08-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662763#comment-14662763 ] Wenchen Fan commented on SPARK-9738: [~joshrosen] sorry about the rush, I've filled in

[jira] [Commented] (SPARK-9604) Unsafe ArrayData and MapData is very very slow

2015-08-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663335#comment-14663335 ] Wenchen Fan commented on SPARK-9604: cc [~rxin] [~davies] looks like the tests is fast

[jira] [Created] (SPARK-9929) support adding metadata in withColumn

2015-08-12 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9929: -- Summary: support adding metadata in withColumn Key: SPARK-9929 URL: https://issues.apache.org/jira/browse/SPARK-9929 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-10034) Can't analyze Sort on Aggregate with aggregation expression named _aggOrdering

2015-08-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-10034: Description: {code=scala} val df = Seq(1 - 2).toDF(i, j) val query = df.groupBy('i)

[jira] [Updated] (SPARK-10034) Can't analyze Sort on Aggregate with aggregation expression named _aggOrdering

2015-08-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-10034: Description: {code} val df = Seq(1 - 2).toDF(i, j) val query = df.groupBy('i)

[jira] [Created] (SPARK-10033) Sort on

2015-08-16 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-10033: --- Summary: Sort on Key: SPARK-10033 URL: https://issues.apache.org/jira/browse/SPARK-10033 Project: Spark Issue Type: Bug Reporter: Wenchen Fan

[jira] [Created] (SPARK-10034) Can't analyze Sort on Aggregate with aggregation expression named _aggOrdering

2015-08-16 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-10034: --- Summary: Can't analyze Sort on Aggregate with aggregation expression named _aggOrdering Key: SPARK-10034 URL: https://issues.apache.org/jira/browse/SPARK-10034

[jira] [Closed] (SPARK-10033) Sort on

2015-08-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan closed SPARK-10033. --- Resolution: Invalid Sort on Key: SPARK-10033 URL:

[jira] [Created] (SPARK-10174) refactor out project, filter, ordering generator from SparkPlan

2015-08-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-10174: --- Summary: refactor out project, filter, ordering generator from SparkPlan Key: SPARK-10174 URL: https://issues.apache.org/jira/browse/SPARK-10174 Project: Spark

[jira] [Created] (SPARK-10154) remove the no-longer-necessary CatalystScan

2015-08-21 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-10154: --- Summary: remove the no-longer-necessary CatalystScan Key: SPARK-10154 URL: https://issues.apache.org/jira/browse/SPARK-10154 Project: Spark Issue Type:

[jira] [Created] (SPARK-8723) improve code gen for divide and remainder

2015-06-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-8723: -- Summary: improve code gen for divide and remainder Key: SPARK-8723 URL: https://issues.apache.org/jira/browse/SPARK-8723 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-9542) create unsafe version of map type

2015-08-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9542: -- Summary: create unsafe version of map type Key: SPARK-9542 URL: https://issues.apache.org/jira/browse/SPARK-9542 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-9598) do not expose generic getter in internal row

2015-08-04 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9598: -- Summary: do not expose generic getter in internal row Key: SPARK-9598 URL: https://issues.apache.org/jira/browse/SPARK-9598 Project: Spark Issue Type:

[jira] [Commented] (SPARK-9404) UnsafeArrayData

2015-07-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645435#comment-14645435 ] Wenchen Fan commented on SPARK-9404: I have a new idea about the encoding. Since array

[jira] [Created] (SPARK-9632) update InternalRow.toSeq to make it accept data type info

2015-08-05 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9632: -- Summary: update InternalRow.toSeq to make it accept data type info Key: SPARK-9632 URL: https://issues.apache.org/jira/browse/SPARK-9632 Project: Spark Issue

[jira] [Created] (SPARK-9634) resolve UnresolvedAlias in DataFrame.resolve

2015-08-05 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9634: -- Summary: resolve UnresolvedAlias in DataFrame.resolve Key: SPARK-9634 URL: https://issues.apache.org/jira/browse/SPARK-9634 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-9604) Unsafe ArrayData and MapData is very very slow

2015-08-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654696#comment-14654696 ] Wenchen Fan commented on SPARK-9604: there is a known issue:

[jira] [Created] (SPARK-9551) add copyTo for UnsafeRow to reuse a copy buffer

2015-08-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9551: -- Summary: add copyTo for UnsafeRow to reuse a copy buffer Key: SPARK-9551 URL: https://issues.apache.org/jira/browse/SPARK-9551 Project: Spark Issue Type:

[jira] [Created] (SPARK-9553) remove the no-longer-necessary createCode and createStructCode, and replace the usage of them by createStructCode

2015-08-03 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9553: -- Summary: remove the no-longer-necessary createCode and createStructCode, and replace the usage of them by createStructCode Key: SPARK-9553 URL:

[jira] [Updated] (SPARK-9553) remove the createCode and createStructCode, and replace the usage of them by createStructCode

2015-08-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9553: --- Summary: remove the createCode and createStructCode, and replace the usage of them by

[jira] [Created] (SPARK-9480) Create an map abstract class MapData and a default implementation backed by 2 ArrayData

2015-07-30 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9480: -- Summary: Create an map abstract class MapData and a default implementation backed by 2 ArrayData Key: SPARK-9480 URL: https://issues.apache.org/jira/browse/SPARK-9480

[jira] [Updated] (SPARK-9634) cleanup unnecessary Aliases in LogicalPlan

2015-08-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9634: --- Summary: cleanup unnecessary Aliases in LogicalPlan (was: resolve UnresolvedAlias in

[jira] [Updated] (SPARK-9634) cleanup unnecessary Aliases in LogicalPlan at the end of analysis

2015-08-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9634: --- Summary: cleanup unnecessary Aliases in LogicalPlan at the end of analysis (was: cleanup unnecessary

[jira] [Updated] (SPARK-9634) cleanup unnecessary Aliases in LogicalPlan

2015-08-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9634: --- Description: Also alias the ExtractValue instead of wrapping it with UnresolvedAlias when resolve

[jira] [Created] (SPARK-9683) deep copy UTF8String when convert unsafe row to safe row

2015-08-06 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9683: -- Summary: deep copy UTF8String when convert unsafe row to safe row Key: SPARK-9683 URL: https://issues.apache.org/jira/browse/SPARK-9683 Project: Spark Issue

[jira] [Created] (SPARK-9732) remove the unsafe - safe conversion

2015-08-06 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9732: -- Summary: remove the unsafe - safe conversion Key: SPARK-9732 URL: https://issues.apache.org/jira/browse/SPARK-9732 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-9683) copy UTF8String when convert unsafe array/map to safe

2015-08-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9683: --- Summary: copy UTF8String when convert unsafe array/map to safe (was: deep copy UTF8String when

[jira] [Created] (SPARK-9738) remove FromUnsafe and add its codegen version to GenerateSafe

2015-08-07 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9738: -- Summary: remove FromUnsafe and add its codegen version to GenerateSafe Key: SPARK-9738 URL: https://issues.apache.org/jira/browse/SPARK-9738 Project: Spark

[jira] [Commented] (SPARK-8670) Nested columns can't be referenced (but they can be selected)

2015-08-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696356#comment-14696356 ] Wenchen Fan commented on SPARK-8670: We support backticks in select and technically we

[jira] [Commented] (SPARK-9950) Wrong Analysis Error for grouping/aggregating on struct fields

2015-08-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14696429#comment-14696429 ] Wenchen Fan commented on SPARK-9950: this is caused by the extra `UnresolvedAlias` in

[jira] [Created] (SPARK-9778) remove unnecessary evaluation from SortOrder

2015-08-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9778: -- Summary: remove unnecessary evaluation from SortOrder Key: SPARK-9778 URL: https://issues.apache.org/jira/browse/SPARK-9778 Project: Spark Issue Type:

[jira] [Updated] (SPARK-9192) add initialization phase for nondeterministic expression

2015-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9192: --- Summary: add initialization phase for nondeterministic expression (was: add initialization phase for

[jira] [Updated] (SPARK-9192) add initialization phase for nondeterministic expression

2015-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-9192: --- Description: Currently nondeterministic expression is broken without a explicit initialization

[jira] [Commented] (SPARK-9192) add initialization phase for nondeterministic expression

2015-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637124#comment-14637124 ] Wenchen Fan commented on SPARK-9192: hi [~lian cheng], sorry about leaving the

[jira] [Created] (SPARK-9294) cleanup comments, code style, naming typo for the new aggregation

2015-07-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-9294: -- Summary: cleanup comments, code style, naming typo for the new aggregation Key: SPARK-9294 URL: https://issues.apache.org/jira/browse/SPARK-9294 Project: Spark

[jira] [Commented] (SPARK-6898) Special chars in column names is broken

2015-07-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642232#comment-14642232 ] Wenchen Fan commented on SPARK-6898: It's really weird that I can run this case in

  1   2   3   4   5   6   7   8   9   10   >