[jira] [Commented] (SPARK-7428) DataFrame.join() could create a new df with duplicate column name

2015-10-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968575#comment-14968575 ] Xiao Li commented on SPARK-7428: IMO, it is impossible to fix the problem if we use dataFrames. Spark SQL

[jira] [Created] (SPARK-11360) Loss of nullability when writing parquet files

2015-10-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-11360: --- Summary: Loss of nullability when writing parquet files Key: SPARK-11360 URL: https://issues.apache.org/jira/browse/SPARK-11360 Project: Spark Issue Type:

[jira] [Commented] (SPARK-8658) AttributeReference equals method only compare name, exprId and dataType

2015-10-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967898#comment-14967898 ] Xiao Li commented on SPARK-8658: Hi, Michael, Thank you! It sounds a trivial work. Let me try it. Send a

[jira] [Updated] (SPARK-11275) [SQL] Regression in rollup/cube

2015-10-23 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-11275: Summary: [SQL] Regression in rollup/cube (was: Regression in rollup/cube ) > [SQL] Regression in

[jira] [Created] (SPARK-11275) Regression in rollup/cube

2015-10-23 Thread Xiao Li (JIRA)
Xiao Li created SPARK-11275: --- Summary: Regression in rollup/cube Key: SPARK-11275 URL: https://issues.apache.org/jira/browse/SPARK-11275 Project: Spark Issue Type: Bug Components: SQL

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2015-10-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968427#comment-14968427 ] Xiao Li commented on SPARK-10925: - There is another ongoing JIRA that has a big impact on the fix for

[jira] [Created] (SPARK-11576) [SQL] Incorrect results when using the nested self-join

2015-11-08 Thread Xiao Li (JIRA)
Xiao Li created SPARK-11576: --- Summary: [SQL] Incorrect results when using the nested self-join Key: SPARK-11576 URL: https://issues.apache.org/jira/browse/SPARK-11576 Project: Spark Issue Type:

[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002986#comment-15002986 ] Xiao Li commented on SPARK-11637: - The fix is ready. Will submit a PR soon. > Alias do not work with

[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003536#comment-15003536 ] Xiao Li commented on SPARK-11637: - https://github.com/apache/spark/pull/9343 has already fixed the

[jira] [Issue Comment Deleted] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node

2015-11-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-11633: Comment: was deleted (was: Which version are you using? I did hit an error, but it is a different error:

[jira] [Commented] (SPARK-11753) Understand why allowNonNumericNumbers JSON option doesn't work

2015-11-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007999#comment-15007999 ] Xiao Li commented on SPARK-11753: - I see. Will do. Thanks! > Understand why allowNonNumericNumbers JSON

[jira] [Commented] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node

2015-11-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007668#comment-15007668 ] Xiao Li commented on SPARK-11633: - The fix is ready. Will deliver it soon. > HiveContext throws

[jira] [Commented] (SPARK-11753) Understand why allowNonNumericNumbers JSON option doesn't work

2015-11-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007950#comment-15007950 ] Xiao Li commented on SPARK-11753: - I can try it if this is not an urgent issue. > Understand why

[jira] [Closed] (SPARK-11433) [SQL] Rule EliminateSubQueries does not clean the parent Project's qualifiers

2015-11-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-11433. --- Resolution: Won't Fix > [SQL] Rule EliminateSubQueries does not clean the parent Project's qualifiers >

[jira] [Commented] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node

2015-11-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006283#comment-15006283 ] Xiao Li commented on SPARK-11633: - Which version are you using? I did hit an error, but it is a different

[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002460#comment-15002460 ] Xiao Li commented on SPARK-11637: - After using hiveContext, I can reproduce your problem: Exception in

[jira] [Commented] (SPARK-11633) HiveContext throws TreeNode Exception : Failed to Copy Node

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1542#comment-1542 ] Xiao Li commented on SPARK-11633: - I am interested in this issue. Can you post your exception? >

[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1577#comment-1577 ] Xiao Li commented on SPARK-11637: - In 1.4.1, it works well. > Alias do not work with udf with *

[jira] [Comment Edited] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1522#comment-1522 ] Xiao Li edited comment on SPARK-11637 at 11/11/15 6:53 AM: --- In 1.5.1, the

[jira] [Commented] (SPARK-11637) Alias do not work with udf with * parameter

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1522#comment-1522 ] Xiao Li commented on SPARK-11637: - In 1.5.1, the output of your query is: ''' Exception in thread "main"

[jira] [Commented] (SPARK-11231) join returns schema with duplicated and ambiguous join columns

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530 ] Xiao Li commented on SPARK-11231: - Your join is not natural join. It could return duplicate names. You

[jira] [Comment Edited] (SPARK-11231) join returns schema with duplicated and ambiguous join columns

2015-11-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530 ] Xiao Li edited comment on SPARK-11231 at 11/11/15 6:59 AM: --- Your join is not

[jira] [Updated] (SPARK-11275) [SQL] Incorrect results when using rollup/cube

2015-11-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-11275: Summary: [SQL] Incorrect results when using rollup/cube (was: [SQL] Regression in rollup/cube ) > [SQL]

[jira] [Updated] (SPARK-11275) [SQL] Incorrect results when using rollup/cube

2015-11-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-11275: Affects Version/s: 1.3.0 1.4.0 > [SQL] Incorrect results when using rollup/cube >

[jira] [Commented] (SPARK-11275) [SQL] Regression in rollup/cube

2015-10-30 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983324#comment-14983324 ] Xiao Li commented on SPARK-11275: - Hi, Andrew, Expression is not the root cause. The implementation of

[jira] [Commented] (SPARK-11275) [SQL] Regression in rollup/cube

2015-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984539#comment-14984539 ] Xiao Li commented on SPARK-11275: - A simple fix can resolve this issue by using subquery. Thus, trying to

[jira] [Commented] (SPARK-11360) Loss of nullability when writing parquet files

2015-10-30 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983738#comment-14983738 ] Xiao Li commented on SPARK-11360: - Hi, Sean, I see. Thank you! Xiao Li > Loss of nullability when

[jira] [Commented] (SPARK-11275) [SQL] Regression in rollup/cube

2015-11-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984881#comment-14984881 ] Xiao Li commented on SPARK-11275: - Agree. It becomes more complex when you need to resolve both cases at

[jira] [Commented] (SPARK-11275) [SQL] Regression in rollup/cube

2015-11-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984814#comment-14984814 ] Xiao Li commented on SPARK-11275: - My fix is ready. Now, trying to add the test cases. Hopefully, I can

[jira] [Commented] (SPARK-10838) Repeat to join one DataFrame twice,there will be AnalysisException.

2015-11-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990544#comment-14990544 ] Xiao Li commented on SPARK-10838: - In 1.5.1, both failed with the same exception. Exception in thread

[jira] [Comment Edited] (SPARK-5068) When the path not found in the hdfs,we can't get the result

2015-11-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990352#comment-14990352 ] Xiao Li edited comment on SPARK-5068 at 11/4/15 8:34 PM: - Now, the default value

[jira] [Commented] (SPARK-5068) When the path not found in the hdfs,we can't get the result

2015-11-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990352#comment-14990352 ] Xiao Li commented on SPARK-5068: Now, the default value of this feature is off. You can turn it on and do

[jira] [Commented] (SPARK-10838) Repeat to join one DataFrame twice,there will be AnalysisException.

2015-11-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991108#comment-14991108 ] Xiao Li commented on SPARK-10838: - The fix is ready. Writing unit test cases now. > Repeat to join one

[jira] [Created] (SPARK-11433) [SQL] Rule EliminateSubQueries does not clean the parent Project's qualifiers

2015-10-30 Thread Xiao Li (JIRA)
Xiao Li created SPARK-11433: --- Summary: [SQL] Rule EliminateSubQueries does not clean the parent Project's qualifiers Key: SPARK-11433 URL: https://issues.apache.org/jira/browse/SPARK-11433 Project: Spark

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2015-10-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957170#comment-14957170 ] Xiao Li commented on SPARK-10925: - Hi, Alexis, The schema of your query results has the duplicate

[jira] [Comment Edited] (SPARK-10925) Exception when joining DataFrames

2015-10-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956367#comment-14956367 ] Xiao Li edited comment on SPARK-10925 at 10/14/15 7:16 AM: --- Also hit the same

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2015-10-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956367#comment-14956367 ] Xiao Li commented on SPARK-10925: - Also hit the same problem. Trying to narrow down the root cause of the

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2015-10-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958054#comment-14958054 ] Xiao Li commented on SPARK-10925: - I have not tried Spark 1.4, but inner joining 2 tables with the same

[jira] [Commented] (SPARK-8658) AttributeReference equals method only compare name, exprId and dataType

2015-10-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949946#comment-14949946 ] Xiao Li commented on SPARK-8658: Hi, Michael and Antonio, Trying to understand the problem and fix it if

[jira] [Commented] (SPARK-10217) Spark SQL cannot handle ordering directive in ORDER BY clauses with expressions

2015-10-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955723#comment-14955723 ] Xiao Li commented on SPARK-10217: - Hi, Simeon, I am trying to reproduce your problem on Spark 1.5.1. I

[jira] [Commented] (SPARK-11803) Dataset self join returns incorrect result

2015-11-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011193#comment-15011193 ] Xiao Li commented on SPARK-11803: - No problem. Actually, I have a couple of test cases. You can try it.

[jira] [Commented] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009916#comment-15009916 ] Xiao Li commented on SPARK-11770: - I can take a look at this problem. > Spark SQL field resolution

[jira] [Commented] (SPARK-7286) Precedence of operator not behaving properly

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009898#comment-15009898 ] Xiao Li commented on SPARK-7286: This is affected by the operator precedence in Scala. There does not

[jira] [Commented] (SPARK-6929) Alias for more complex expression causes attribute not been able to resolve

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009340#comment-15009340 ] Xiao Li commented on SPARK-6929: [~mwaciega] The default Alias name generation was changed to _c$i in the

[jira] [Commented] (SPARK-11803) Dataset self join returns incorrect result

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010267#comment-15010267 ] Xiao Li commented on SPARK-11803: - We need to detect if this is a self join in the function joinWith. >

[jira] [Commented] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li commented on SPARK-11770: - Hi, [~simeons] I am unable to reproduce your issue. Could you try it

[jira] [Comment Edited] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li edited comment on SPARK-11770 at 11/18/15 3:47 AM: --- Hi, [~simeons] I

[jira] [Comment Edited] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li edited comment on SPARK-11770 at 11/18/15 3:47 AM: --- Hi, [~simeons] I

[jira] [Commented] (SPARK-11803) Dataset self join returns incorrect result

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010177#comment-15010177 ] Xiao Li commented on SPARK-11803: - Not sure if this has been assigned. I can try it tonight and tomorrow.

[jira] [Comment Edited] (SPARK-11803) Dataset self join returns incorrect result

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010181#comment-15010181 ] Xiao Li edited comment on SPARK-11803 at 11/18/15 4:14 AM: --- The optimized plan

[jira] [Comment Edited] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li edited comment on SPARK-11770 at 11/18/15 3:48 AM: --- Hi, [~simeons] I

[jira] [Comment Edited] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li edited comment on SPARK-11770 at 11/18/15 3:48 AM: --- Hi, [~simeons] I

[jira] [Comment Edited] (SPARK-11770) Spark SQL field resolution error in GROUP BY HAVING clause

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010157#comment-15010157 ] Xiao Li edited comment on SPARK-11770 at 11/18/15 3:49 AM: --- Hi, [~simeons] I

[jira] [Commented] (SPARK-11803) Dataset self join returns incorrect result

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010230#comment-15010230 ] Xiao Li commented on SPARK-11803: - We need to assign a new expression ID the conflicting attribute in

[jira] [Commented] (SPARK-11803) Dataset self join returns incorrect result

2015-11-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010181#comment-15010181 ] Xiao Li commented on SPARK-11803: - The optimized plan is wrong. Project [value#1 AS _1#4,value#1 AS

[jira] [Created] (SPARK-12028) [SQL] get_json_object is unable to return a correct result for null literals

2015-11-27 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12028: --- Summary: [SQL] get_json_object is unable to return a correct result for null literals Key: SPARK-12028 URL: https://issues.apache.org/jira/browse/SPARK-12028 Project: Spark

[jira] [Updated] (SPARK-11980) Fix json_tuple and add unit tests for the Python functions added in SPARK-10621

2015-11-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-11980: Summary: Fix json_tuple and add unit tests for the Python functions added in SPARK-10621 (was: Add unit

[jira] [Commented] (SPARK-12030) Incorrect results when aggregate joined data

2015-11-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031206#comment-15031206 ] Xiao Li commented on SPARK-12030: - I can reproduced a similar issue in a Sort. I think the impact could

[jira] [Comment Edited] (SPARK-12030) Incorrect results when aggregate joined data

2015-11-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031060#comment-15031060 ] Xiao Li edited comment on SPARK-12030 at 11/29/15 6:11 PM: --- [~maver1ck] Yeah,

[jira] [Updated] (SPARK-12091) [PySpark] Removal of the JAVA-specific deserialized storage levels

2015-12-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12091: Description: Since the data is always serialized on the Python side, the JAVA-specific deserialized levels

[jira] [Updated] (SPARK-12091) [PySpark] Removal of the JAVA-specific deserialized storage levels

2015-12-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12091: Priority: Major (was: Minor) > [PySpark] Removal of the JAVA-specific deserialized storage levels >

[jira] [Updated] (SPARK-12091) [PySpark] Removal of the JAVA-specific deserialized storage levels

2015-12-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12091: Summary: [PySpark] Removal of the JAVA-specific deserialized storage levels (was: [PySpark] Inconsistent

[jira] [Updated] (SPARK-12164) [SQL] Display the binary/encoded values

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12164: Description: So far, we are using comma-separated decimal format to output the encoded contents. This way

[jira] [Updated] (SPARK-12164) [SQL] Display the binary/encoded values

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12164: Description: So far, we are using comma-separated decimal format to output the encoded contents. This way

[jira] [Updated] (SPARK-12164) [SQL] Display the binary/encoded values

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12164: Description: So far, we are using comma-separated decimal format to output the encoded contents. This way

[jira] [Updated] (SPARK-12164) [SQL] Display the binary/encoded values

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12164: Description: So far, we are using comma-separated decimal format to output the encoded contents. This way

[jira] [Updated] (SPARK-12164) [SQL] Display the binary/encoded values

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12164: Description: So far, we are using comma-separated decimal format to output the encoded contents. This way

[jira] [Updated] (SPARK-12158) [R] [SQL] Fix 'sample' functions that break R unit test cases

2015-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12158: Component/s: (was: R) SparkR > [R] [SQL] Fix 'sample' functions that break R unit

[jira] [Commented] (SPARK-12233) Cannot specify a data frame column during join

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048232#comment-15048232 ] Xiao Li commented on SPARK-12233: - I am unable to reproduce your error. Try to run my example, can you

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048256#comment-15048256 ] Xiao Li commented on SPARK-12218: - That is what I did in my environment. {code} val df1 = Seq(1, 2,

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047928#comment-15047928 ] Xiao Li commented on SPARK-12218: - Could you provide the plan by explain(true)? [~imachabeli] Thanks! >

[jira] [Comment Edited] (SPARK-12225) Support adding or replacing multiple columns at once in DataFrame API

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048059#comment-15048059 ] Xiao Li edited comment on SPARK-12225 at 12/9/15 5:21 AM: -- This is related to

[jira] [Commented] (SPARK-6929) Alias for more complex expression causes attribute not been able to resolve

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048098#comment-15048098 ] Xiao Li commented on SPARK-6929: [~srowen] Could you close this issue? This has been resolved, I think.

[jira] [Commented] (SPARK-12225) Support adding or replacing multiple columns at once in DataFrame API

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047934#comment-15047934 ] Xiao Li commented on SPARK-12225: - Ok, thank you! > Support adding or replacing multiple columns at

[jira] [Comment Edited] (SPARK-12233) Cannot specify a data frame column during join

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048096#comment-15048096 ] Xiao Li edited comment on SPARK-12233 at 12/9/15 6:00 AM: -- This is another self

[jira] [Commented] (SPARK-12225) Support adding or replacing multiple columns at once in DataFrame API

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047922#comment-15047922 ] Xiao Li commented on SPARK-12225: - [~sunrui] Will you deliver the feature? Otherwise, I can work on it.

[jira] [Commented] (SPARK-12225) Support adding or replacing multiple columns at once in DataFrame API

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048059#comment-15048059 ] Xiao Li commented on SPARK-12225: - This is related to the changes on the external APIs. Need to collect

[jira] [Commented] (SPARK-12233) Cannot specify a data frame column during join

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048087#comment-15048087 ] Xiao Li commented on SPARK-12233: - Please post the error message you got. Thanks! > Cannot specify a

[jira] [Commented] (SPARK-12233) Cannot specify a data frame column during join

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048096#comment-15048096 ] Xiao Li commented on SPARK-12233: - This is another self join issue. I will try to see if it is a

[jira] [Commented] (SPARK-12233) Cannot specify a data frame column during join

2015-12-08 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048186#comment-15048186 ] Xiao Li commented on SPARK-12233: - {code} val df1 = Seq(1, 2, 3).map(i => (i, i.toString,

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048974#comment-15048974 ] Xiao Li commented on SPARK-12218: - Thanks for the info! I will reproduce it soon. Will keep you posted. :

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049029#comment-15049029 ] Xiao Li commented on SPARK-12218: - Hi, [~imachabeli] Sorry, in the latest 1.6 build, I am unable to

[jira] [Commented] (SPARK-12225) Support adding or replacing multiple columns at once in DataFrame API

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049049#comment-15049049 ] Xiao Li commented on SPARK-12225: - [~rxin] We can call withColumn() multiple times for achieving the

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049201#comment-15049201 ] Xiao Li commented on SPARK-12218: - Great! That means, the problem has been resolved in Spark 1.6, which

[jira] [Created] (SPARK-12158) [R] [SQL] Fix 'sample' functions that break R unit test cases

2015-12-05 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12158: --- Summary: [R] [SQL] Fix 'sample' functions that break R unit test cases Key: SPARK-12158 URL: https://issues.apache.org/jira/browse/SPARK-12158 Project: Spark Issue

[jira] [Commented] (SPARK-12150) numPartitions argument to sqlContext.range() should be optional

2015-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042156#comment-15042156 ] Xiao Li commented on SPARK-12150: - Yeah, you are right. Will deliver a PR soon. Thanks! > numPartitions

[jira] [Commented] (SPARK-12138) Escape \u in the generated comments.

2015-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042492#comment-15042492 ] Xiao Li commented on SPARK-12138: - If nobody takes it, I can make a try. [~yhuai] Could you explain how

[jira] [Commented] (SPARK-12138) Escape \u in the generated comments.

2015-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042541#comment-15042541 ] Xiao Li commented on SPARK-12138: - Sure. Will try it tonight or tomorrow. Thanks! : ) > Escape \u in the

[jira] [Commented] (SPARK-12030) Incorrect results when aggregate joined data

2015-12-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035164#comment-15035164 ] Xiao Li commented on SPARK-12030: - I did verify the fix using my test cases. It works! I posted a

[jira] [Created] (SPARK-12091) [PySpark] Inconsistent default storage level of persist/cache in Python API

2015-12-01 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12091: --- Summary: [PySpark] Inconsistent default storage level of persist/cache in Python API Key: SPARK-12091 URL: https://issues.apache.org/jira/browse/SPARK-12091 Project: Spark

[jira] [Commented] (SPARK-8360) Streaming DataFrames

2015-12-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035379#comment-15035379 ] Xiao Li commented on SPARK-8360: "You need permission to access this published document." I got this

[jira] [Created] (SPARK-12188) [SQL] Code refactoring and comment correction in Dataset APIs

2015-12-07 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12188: --- Summary: [SQL] Code refactoring and comment correction in Dataset APIs Key: SPARK-12188 URL: https://issues.apache.org/jira/browse/SPARK-12188 Project: Spark Issue

[jira] [Created] (SPARK-12195) Adding BigDecimal, Date and Timestamp into Encoder

2015-12-07 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12195: --- Summary: Adding BigDecimal, Date and Timestamp into Encoder Key: SPARK-12195 URL: https://issues.apache.org/jira/browse/SPARK-12195 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12218) Boolean logic in sql does not work "not (A and B)" is not the same as "(not A) or (not B)"

2015-12-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052763#comment-15052763 ] Xiao Li commented on SPARK-12218: - Agree! I will do a search to find out what happened in the push down

[jira] [Created] (SPARK-12256) [SQL] Code refactoring: naming boolean variables

2015-12-09 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12256: --- Summary: [SQL] Code refactoring: naming boolean variables Key: SPARK-12256 URL: https://issues.apache.org/jira/browse/SPARK-12256 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-12259) Kryo/javaSerialization encoder are not composable

2015-12-09 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12259: --- Summary: Kryo/javaSerialization encoder are not composable Key: SPARK-12259 URL: https://issues.apache.org/jira/browse/SPARK-12259 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050246#comment-15050246 ] Xiao Li edited comment on SPARK-12258 at 12/10/15 7:37 AM: --- [~cloud_fan] It

[jira] [Issue Comment Deleted] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12258: Comment: was deleted (was: [~cloud_fan] It sounds like it is related to the PR

[jira] [Commented] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-09 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050246#comment-15050246 ] Xiao Li commented on SPARK-12258: - [~cloud_fan] It sounds like it is related to the PR

[jira] [Commented] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050398#comment-15050398 ] Xiao Li commented on SPARK-12258: - A PR has been submitted. Thanks > Hive Timestamp UDF is binded with

  1   2   3   4   5   6   7   8   9   10   >