[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864245#comment-17864245 ] ASF GitHub Bot commented on DRILL-8501: --- cgivre merged PR #2921: URL: https://github.com/apache/drill/pull/2921 > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864237#comment-17864237 ] ASF GitHub Bot commented on DRILL-8501: --- jnturton commented on PR #2921: URL: https://github.com/apache/drill/pull/2921#issuecomment-2217905543 Oh that's great, thanks for the enhancements +1. > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863589#comment-17863589 ] ASF GitHub Bot commented on DRILL-8501: --- cgivre commented on PR #2921: URL: https://github.com/apache/drill/pull/2921#issuecomment-2212491446 @jnturton I added new versions of the UDF so that the user can specify in the function call whether they want `allTextMode` and the other option. > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863534#comment-17863534 ] ASF GitHub Bot commented on DRILL-8501: --- cgivre commented on PR #2921: URL: https://github.com/apache/drill/pull/2921#issuecomment-2212305704 > Before I approve, did you consider making these JSON parsing settings parameters of the function itself? It feels odd to me that `store.json.*` settings could influence UDFs too. I'm not sure why they aren't storage plugin config, rather than global config, in the first place... @jnturton I thought about doing exactly what you're describing. Here's the thing. We started some work a while ago to get rid of all the non-EVF2 readers in Drill. It turns out that there are a few places which still use the old non-EVF JSON reader. Specifically, this UDF, the Druid Storage Plugin and the MongoDB storage plugin. I started work on [Drill-8316](https://github.com/apache/drill/pull/2657) which addresses the Druid plugin and [Drill-8329](https://github.com/apache/drill/pull/2567) addresses converting the UDF. Neither one of these were a high priority so they're kind of sitting at the moment. I agree with your premise that the whole idea of having global settings for file formats (including parquet) is not the best idea. > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863481#comment-17863481 ] ASF GitHub Bot commented on DRILL-8501: --- jnturton commented on PR #2921: URL: https://github.com/apache/drill/pull/2921#issuecomment-2211746498 Before I approve, did you consider making these JSON parsing settings parameters of the function itself? It feels odd to me that `store.json.*` settings could influence UDFs too. I'm not sure why they aren't storage plugin config, rather than global config, in the first place... > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8501) Json Conversion UDF Not Respecting System JSON Options
[ https://issues.apache.org/jira/browse/DRILL-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863057#comment-17863057 ] ASF GitHub Bot commented on DRILL-8501: --- cgivre opened a new pull request, #2921: URL: https://github.com/apache/drill/pull/2921 # [DRILL-8501](https://issues.apache.org/jira/browse/DRILL-): Json Conversion UDF Not Respecting System JSON Options ## Description The `convert_fromJSON()` function was ignoring Drill system configuration variables for reading JSON. This PR adds support for `allTextMode` and `readNumbersAsDouble` to this function. Once merged, the `convert_fromJSON()` function will follow the system settings. I also split one of the unit test files because it had all the UDF tests mixed with NaN tests. ## Documentation No user facing changes. ## Testing Added unit tests. > Json Conversion UDF Not Respecting System JSON Options > -- > > Key: DRILL-8501 > URL: https://issues.apache.org/jira/browse/DRILL-8501 > Project: Apache Drill > Issue Type: Bug > Components: Storage - JSON >Affects Versions: 1.21.2 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.22.0 > > > The convert_fromJSON() UDF does not respect the system JSON options of > allTextMode and readAllNumbersAsDouble. > This PR fixes that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
