[jira] [Commented] (DRILL-7741) Columns are missing when using convert_from function

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/DRILL-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110112#comment-17110112
 ] 

ASF GitHub Bot commented on DRILL-7741:
---------------------------------------

vvysotskyi opened a new pull request #2081:
URL: https://github.com/apache/drill/pull/2081


   # [DRILL-7741](https://issues.apache.org/jira/browse/DRILL-7741): Columns 
are missing when using convert_from function
   
   ## Description
   See Jira's description for details.
   
   ## Documentation
   NA
   
   ## Testing
   Added a unit test and checked manually on larger data.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Columns are missing when using convert_from function
> ----------------------------------------------------
>
>                 Key: DRILL-7741
>                 URL: https://issues.apache.org/jira/browse/DRILL-7741
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.13.0
>            Reporter: Vova Vysotskyi
>            Assignee: Vova Vysotskyi
>            Priority: Major
>             Fix For: 1.18.0
>
>
> The query returns wrong results when {{convert_from}} function and additional 
> columns may appear in new batches. Here is a simple test case for this issue:
> {code:java}
>   @Test
>   public void testConvertFromJson() throws Exception {
>     String fileName = "table.tsv";
>     try (BufferedWriter writer = new BufferedWriter(new FileWriter(new 
> File(dirTestWatcher.getRootDir(), fileName)))) {
>       for (int i = 0; i < JSONRecordReader.DEFAULT_ROWS_PER_BATCH; i++) {
>         writer.write("{\"id\":\"1\"}\n");
>       }
>       writer.write("{\"id\":\"2\",\"v\":[\"abc\"]}");
>     }
>     String sql = "SELECT t.m.id AS id, t.m.v[0] v FROM \n" +
>         "(SELECT convert_from(columns[0], 'json') AS m FROM dfs.`%s`) t\n" +
>         "where t.m.id='2'";
>     testBuilder()
>         .sqlQuery(sql, fileName)
>         .unOrdered()
>         .baselineColumns("id", "v")
>         .baselineValues("2", "abc")
>         .go();
>   }
> {code}
> Currently, the {{"v"}} column is null since {{OK_NEW_SCHEMA}} wasn't returned 
> from the project when the complex writer was used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)