[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134942#comment-14134942 ] Apache Spark commented on SPARK-2096: - User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/2405 Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter Fix For: 1.2.0 For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117381#comment-14117381 ] Apache Spark commented on SPARK-2096: - User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/2230 Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105221#comment-14105221 ] Hanwei Jin commented on SPARK-2096: --- I think i almost have solved the issue. I have passed the test case in JsonSuite (Complex field and type inferring (Ignored)) which is ignored, by a little modified. modified test part : checkAnswer( sql(select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable), (Seq(true, false, null), Seq(str1, null, null)) :: Nil ) However, another question is repeated nested structure is a problem, like arrayOfStruct.field1.arrayOfStruct.field1 or arrayOfStruct[0].field1.arrayOfStruct[0].field1 I plan to ignore this problem and try to add select arrayOfStruct.field1, arrayOfStruct.field2 from jsonTable where arrayOfStruct.field1==true Besides, my friend anyweil (Wei Li) solved the problem of arrayOfStruct.field1 and its Filter part( means where parsing). I am fresh here but will continue working on spark :) Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105329#comment-14105329 ] Hanwei Jin commented on SPARK-2096: --- I checked the problem where arrayOfStruct.field1==true this problem will lead to modify every kind of comparisonExpression. And I think it makes no sense to add this function. So I discard it. Over. Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105353#comment-14105353 ] Apache Spark commented on SPARK-2096: - User 'chuxi' has created a pull request for this issue: https://github.com/apache/spark/pull/2082 Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098020#comment-14098020 ] Michael Yannakopoulos commented on SPARK-2096: -- Hi Yin, If nobody is working in this issue, I can help! Thanks, Michael Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098028#comment-14098028 ] Yin Huai commented on SPARK-2096: - [~miccagiann] Yeah, sure. Thank you. We have fixed it in Hive parser (https://issues.apache.org/jira/browse/SPARK-2483). We just need to fix it in the SqlParser in Catalyst. Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai Priority: Minor Labels: starter For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2096) Correctly parse dot notations for accessing an array of structs
[ https://issues.apache.org/jira/browse/SPARK-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026607#comment-14026607 ] Yin Huai commented on SPARK-2096: - Also, we need to check the behavior of Hive parser. Correctly parse dot notations for accessing an array of structs --- Key: SPARK-2096 URL: https://issues.apache.org/jira/browse/SPARK-2096 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.0.0 Reporter: Yin Huai For example, arrayOfStruct is an array of structs and every element of this array has a field called field1. arrayOfStruct[0].field1 means to access the value of field1 for the first element of arrayOfStruct, but the SQL parser (in sql-core) treats field1 as an alias. Also, arrayOfStruct.field1 means to access all values of field1 in this array of structs and the returns those values as an array. But, the SQL parser cannot resolve it. -- This message was sent by Atlassian JIRA (v6.2#6252)
