[jira] [Created] (DRILL-7364) Timeout reading from Kafka topic using Kafka plugin
Aditya Allamraju created DRILL-7364:
---
Summary: Timeout reading from Kafka topic using Kafka plugin
Key: DRILL-7364
URL: https://issues.apache.org/jira/browse/DRILL-7364
Project: Apache Drill
Issue Type: Bug
Components: Storage - Kafka
Affects Versions: 1.16.0, 1.15.0
Reporter: Aditya Allamraju
When we try to query Mapr-streams(similar to Apache Kafka) topic using Kafka
plugin, we see the below timeout being thrown.
{code:java}
0: jdbc:drill:drillbit=10.10.75.158:31010>
0: jdbc:drill:drillbit=10.10.75.158:31010> select count(*) from
`/sample-stream:fast-messages` where k<100;
Error: DATA_READ ERROR: Failed to fetch messages within 200 milliseconds.
Consider increasing the value of the property : store.kafka.poll.timeout
DATA_READ ERROR: Failed to fetch messages within 200 milliseconds. Consider
increasing the value of the property : store.kafka.poll.timeout
[Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 ]
Fragment 0:0
[Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 on
vm75-158.support.mapr.com:31010] (state=,code=0)
0: jdbc:drill:drillbit=10.10.75.158:31010> ALTER SYSTEM SET
`store.kafka.poll.timeout` = 5;
+---++
| ok | summary |
+---++
| true | store.kafka.poll.timeout updated. |
+---++
1 row selected (0.148 seconds)
0: jdbc:drill:drillbit=10.10.75.158:31010>
{code}
The other interesting behavior is that:
1) Even after increasing the timeout value to 50secs, the drill query failed
after the execution time crossed 50 secs(~51 secs).
This pattern continued to whatever value we increased. For ex, after increasing
the timeout to 100secs, query failed with above error after 101 secs of
execution time.
The user is using Drill 1.15.
I tried to reproduce this on my test cluster. But this was not consistently
reproducing in the test cluster. Whereas in the client's cluster, we were able
to reproduce this behavior consistently. We collected the logs. But they have
very little info on what's happening.
I believe it is now essential to know how(and why) the timeout parameter
"store.kafka.poll.timeout" is related to the Query execution time to understand
this bug.
I also have few more questions where i couldn't find much documentation.
_1) Is there a one-to-one mapping between number of partitions of a topic to
minor fragments of a query? For ex, If a given topic(t) has say 20 partitions,
then will the query most likely have 20 minor fragments, other parameters being
fairly sized._
_2) Does each drillbit in the cluster equivalent to a Kafka-Consumer and all
such drillbits of a cluster treated as part of a consumer group?_
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[
https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921605#comment-16921605
]
ASF GitHub Bot commented on DRILL-7362:
---
vvysotskyi commented on pull request #1849: DRILL-7362: COUNT(*) on JSON with
outer list results in JsonParse error
URL: https://github.com/apache/drill/pull/1849#discussion_r320351497
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonReader.java
##
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.easy.json.reader;
+
+import com.fasterxml.jackson.core.JsonToken;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter;
+
+import java.io.IOException;
+
+/**
+ * Basic reader implementation for json documents.
+ */
+public abstract class BaseJsonReader extends BaseJsonProcessor {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(BaseJsonReader.class);
+
+ /**
+ * Describes whether or not this reader can unwrap a single root array record
+ * and treat it like a set of distinct records.
+ */
+ private final boolean skipOuterList;
+
+ /**
+ * Whether the reader is currently in a situation where we are unwrapping an
+ * outer list.
+ */
+ private boolean inOuterList;
+
+ public BaseJsonReader(DrillBuf workBuf, boolean enableNanInf, boolean
enableEscapeAnyChar, boolean skipOuterList) {
+super(workBuf, enableNanInf, enableEscapeAnyChar);
+this.skipOuterList = skipOuterList;
+ }
+
+ @Override
+ public ReadState write(ComplexWriter writer) throws IOException {
+
+ReadState readState = null;
+try {
+ JsonToken t = lastSeenJsonToken;
+ if (t == null || t == JsonToken.END_OBJECT) {
+t = parser.nextToken();
+ }
+ while (!parser.hasCurrentToken() && !parser.isClosed()) {
+t = parser.nextToken();
+ }
+ lastSeenJsonToken = null;
+
+ if (parser.isClosed()) {
+return ReadState.END_OF_STREAM;
+ }
+
+ readState = writeToVector(writer, t);
+
+ switch (readState) {
+case END_OF_STREAM:
+ break;
+case WRITE_SUCCEED:
+ break;
+default:
+ throw getExceptionWithContext(UserException.dataReadError(),
null).message(
+"Failure while reading JSON. (Got an invalid read state %s )",
readState.toString())
+.build(logger);
+ }
+} catch (com.fasterxml.jackson.core.JsonParseException ex) {
+ if (ignoreJSONParseError()) {
+if (processJSONException() ==
JsonExceptionProcessingState.END_OF_STREAM) {
+ return ReadState.JSON_RECORD_PARSE_EOF_ERROR;
+} else {
+ return ReadState.JSON_RECORD_PARSE_ERROR;
+}
+ } else {
+throw ex;
+ }
+}
+return readState;
+ }
+
+
+ private ReadState writeToVector(ComplexWriter writer, JsonToken t)
+throws IOException {
+
+switch (t) {
+ case START_OBJECT:
+writeDocument(writer, t);
+break;
+ case START_ARRAY:
+if (inOuterList) {
+ throw createDocumentTopLevelException();
+}
+
+if (skipOuterList) {
+ t = parser.nextToken();
+ if (t == JsonToken.START_OBJECT) {
+inOuterList = true;
+writeDocument(writer, t);
+ } else {
+throw createDocumentTopLevelException();
+ }
+
+} else {
+ writeDocument(writer, t);
+}
+break;
+ case END_ARRAY:
+
+if (inOuterList) {
+ confirmLast();
+ return ReadState.END_OF_STREAM;
+} else {
+ throw getExceptionWithContext(UserException.dataReadError(),
null).message(
+"Failure while parsing JSON. Ran across unexpected %s.",
JsonToken.END_ARRAY).build(logger);
+}
+
+ case NOT_AVAILABLE:
+return ReadState.END_OF_STREAM;
+ default:
+throw getExceptionWithContext(UserException.dataReadError()
[jira] [Commented] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[
https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921603#comment-16921603
]
ASF GitHub Bot commented on DRILL-7362:
---
vvysotskyi commented on pull request #1849: DRILL-7362: COUNT(*) on JSON with
outer list results in JsonParse error
URL: https://github.com/apache/drill/pull/1849#discussion_r320344193
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonReader.java
##
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.easy.json.reader;
+
+import com.fasterxml.jackson.core.JsonToken;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter;
+
+import java.io.IOException;
+
+/**
+ * Basic reader implementation for json documents.
+ */
+public abstract class BaseJsonReader extends BaseJsonProcessor {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(BaseJsonReader.class);
+
+ /**
+ * Describes whether or not this reader can unwrap a single root array record
+ * and treat it like a set of distinct records.
+ */
+ private final boolean skipOuterList;
+
+ /**
+ * Whether the reader is currently in a situation where we are unwrapping an
+ * outer list.
+ */
+ private boolean inOuterList;
+
+ public BaseJsonReader(DrillBuf workBuf, boolean enableNanInf, boolean
enableEscapeAnyChar, boolean skipOuterList) {
+super(workBuf, enableNanInf, enableEscapeAnyChar);
+this.skipOuterList = skipOuterList;
+ }
+
+ @Override
+ public ReadState write(ComplexWriter writer) throws IOException {
+
+ReadState readState = null;
Review comment:
This variable may be moved inside try block to the place where it is
assigned.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> COUNT(*) on JSON with outer list results in JsonParse error
> ---
>
> Key: DRILL-7362
> URL: https://issues.apache.org/jira/browse/DRILL-7362
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.17.0
>
>
> Count from a JSON file with a outer array results in JsonParseException:
> Cannot read from the middle of a record. Current token was START_ARRAY.
> P.S. A simple select from such file works normally.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[
https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921606#comment-16921606
]
ASF GitHub Bot commented on DRILL-7362:
---
vvysotskyi commented on pull request #1849: DRILL-7362: COUNT(*) on JSON with
outer list results in JsonParse error
URL: https://github.com/apache/drill/pull/1849#discussion_r320269096
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/reader/BaseJsonReader.java
##
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.easy.json.reader;
+
+import com.fasterxml.jackson.core.JsonToken;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter;
+
+import java.io.IOException;
+
+/**
+ * Basic reader implementation for json documents.
+ */
+public abstract class BaseJsonReader extends BaseJsonProcessor {
+
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(BaseJsonReader.class);
Review comment:
Please use imports instead of fully qualifying classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> COUNT(*) on JSON with outer list results in JsonParse error
> ---
>
> Key: DRILL-7362
> URL: https://issues.apache.org/jira/browse/DRILL-7362
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.17.0
>
>
> Count from a JSON file with a outer array results in JsonParseException:
> Cannot read from the middle of a record. Current token was START_ARRAY.
> P.S. A simple select from such file works normally.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[
https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921604#comment-16921604
]
ASF GitHub Bot commented on DRILL-7362:
---
vvysotskyi commented on pull request #1849: DRILL-7362: COUNT(*) on JSON with
outer list results in JsonParse error
URL: https://github.com/apache/drill/pull/1849#discussion_r320397567
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/json/TestJsonRecordReader.java
##
@@ -261,4 +261,18 @@ public void
testNotCountingQueryNotSkippingInvalidJSONRecords()
}
throw new Exception("testNotCountingQueryNotSkippingInvalidJSONRecords");
}
+
+ @Test
+ @Category(UnlikelyTest.class)
+ // See DRILL-7362
+ /* Test for CountingJSONReader */
+ public void testContainingArrayCount() throws Exception {
+testBuilder()
+ .sqlQuery("select count(*) as cnt from cp.`store/json/listdoc.json`")
+ .unOrdered()
+ .baselineColumns("cnt")
+ .baselineValues(2L)
+.build()
Review comment:
Please fix formatting and use `.go()` instead of `.build().run();`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> COUNT(*) on JSON with outer list results in JsonParse error
> ---
>
> Key: DRILL-7362
> URL: https://issues.apache.org/jira/browse/DRILL-7362
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Assignee: Oleg Zinoviev
>Priority: Major
> Fix For: 1.17.0
>
>
> Count from a JSON file with a outer array results in JsonParseException:
> Cannot read from the middle of a record. Current token was START_ARRAY.
> P.S. A simple select from such file works normally.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (DRILL-7299) Infinite exception loop in Sqlline after kill process
[
https://issues.apache.org/jira/browse/DRILL-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-7299:
---
Assignee: Volodymyr Vysotskyi
> Infinite exception loop in Sqlline after kill process
> -
>
> Key: DRILL-7299
> URL: https://issues.apache.org/jira/browse/DRILL-7299
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Paul Rogers
>Assignee: Volodymyr Vysotskyi
>Priority: Major
>
> Tried killing Sqlline using the "kill" command. Ended up in an infinte loop
> that repeated printed the following to the console:
> {noformat}
> java.lang.IllegalStateException
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:464)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:445)
> at sqlline.SqlLine.begin(SqlLine.java:537)
> at sqlline.SqlLine.start(SqlLine.java:266)
> at sqlline.SqlLine.main(SqlLine.java:205)
> java.lang.IllegalStateException
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:464)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:445)
> at sqlline.SqlLine.begin(SqlLine.java:537)
> at sqlline.SqlLine.start(SqlLine.java:266)
> at sqlline.SqlLine.main(SqlLine.java:205)
> ...
> {noformat}
> Using "kill -9" properly killed the process.
> Expected a simple "kill" ({{SIGTERM}}) to have done the job.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7299) Infinite exception loop in Sqlline after kill process
[
https://issues.apache.org/jira/browse/DRILL-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7299:
Fix Version/s: 1.17.0
> Infinite exception loop in Sqlline after kill process
> -
>
> Key: DRILL-7299
> URL: https://issues.apache.org/jira/browse/DRILL-7299
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Paul Rogers
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Tried killing Sqlline using the "kill" command. Ended up in an infinte loop
> that repeated printed the following to the console:
> {noformat}
> java.lang.IllegalStateException
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:464)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:445)
> at sqlline.SqlLine.begin(SqlLine.java:537)
> at sqlline.SqlLine.start(SqlLine.java:266)
> at sqlline.SqlLine.main(SqlLine.java:205)
> java.lang.IllegalStateException
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:464)
> at
> org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:445)
> at sqlline.SqlLine.begin(SqlLine.java:537)
> at sqlline.SqlLine.start(SqlLine.java:266)
> at sqlline.SqlLine.main(SqlLine.java:205)
> ...
> {noformat}
> Using "kill -9" properly killed the process.
> Expected a simple "kill" ({{SIGTERM}}) to have done the job.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7096) Develop vector for canonical Map
[ https://issues.apache.org/jira/browse/DRILL-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921490#comment-16921490 ] ASF GitHub Bot commented on DRILL-7096: --- KazydubB commented on pull request #1829: DRILL-7096: Develop vector for canonical Map URL: https://github.com/apache/drill/pull/1829 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop vector for canonical Map > - > > Key: DRILL-7096 > URL: https://issues.apache.org/jira/browse/DRILL-7096 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Igor Guzenko >Assignee: Bohdan Kazydub >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Canonical Map datatype can be represented using combination of three > value vectors: > keysVector - vector for storing keys of each map > valuesVector - vector for storing values of each map > offsetsVector - vector for storing of start indexes of next each map > So it's not very hard to create such Map vector, but there is a major issue > with such map representation. It's hard to search maps values by key in such > vector, need to investigate some advanced techniques to make such search > efficient. Or find other more suitable options to represent map datatype in > world of vectors. > After question about maps, Apache Arrow developers responded that for Java > they don't have real Map vector, for now they just have logical Map type > definition where they define Map like: List< Struct value:value_type> >. So implementation of value vector would be useful for > Arrow too. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7096) Develop vector for canonical Map
[ https://issues.apache.org/jira/browse/DRILL-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7096: --- Labels: ready-to-commit (was: ) > Develop vector for canonical Map > - > > Key: DRILL-7096 > URL: https://issues.apache.org/jira/browse/DRILL-7096 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Igor Guzenko >Assignee: Bohdan Kazydub >Priority: Major > Labels: ready-to-commit > Fix For: 1.17.0 > > > Canonical Map datatype can be represented using combination of three > value vectors: > keysVector - vector for storing keys of each map > valuesVector - vector for storing values of each map > offsetsVector - vector for storing of start indexes of next each map > So it's not very hard to create such Map vector, but there is a major issue > with such map representation. It's hard to search maps values by key in such > vector, need to investigate some advanced techniques to make such search > efficient. Or find other more suitable options to represent map datatype in > world of vectors. > After question about maps, Apache Arrow developers responded that for Java > they don't have real Map vector, for now they just have logical Map type > definition where they define Map like: List< Struct value:value_type> >. So implementation of value vector would be useful for > Arrow too. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7168) Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
[
https://issues.apache.org/jira/browse/DRILL-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921346#comment-16921346
]
ASF GitHub Bot commented on DRILL-7168:
---
arina-ielchiieva commented on pull request #1850: DRILL-7168: Implement ALTER
SCHEMA ADD / REMOVE commands
URL: https://github.com/apache/drill/pull/1850
Jira - [DRILL-7168](https://issues.apache.org/jira/browse/DRILL-7168).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
> --
>
> Key: DRILL-7168
> URL: https://issues.apache.org/jira/browse/DRILL-7168
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> By [~Paul.Rogers]:
> {quote}
> Sooner or later users are going to ask for a command to update just the
> properties, or just add or remove a column, without having to spell out the
> entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
> {quote}
> Syntax:
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> add [or replace]
> [columns (col1 int, col2 varchar)]
> [properties ('prop1'='val1', 'prop2'='val2')]
> {code}
> Add command will fail if column or property with the same name exists, unless
> OR REPLACE keywords are indicated.
> Add command will fail, if schema file does not exit.
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> remove
> [columns (`col1`, `col2`)]
> [properties ('prop1', 'prop2')]
> {code}
> Remove command won't fail if column or property does not exist but will fail
> if schema file is absent.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7362) COUNT(*) on JSON with outer list results in JsonParse error
[ https://issues.apache.org/jira/browse/DRILL-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921305#comment-16921305 ] ASF GitHub Bot commented on DRILL-7362: --- oleg-zinovev commented on issue #1849: DRILL-7362: COUNT(*) on JSON with outer list results in JsonParse error URL: https://github.com/apache/drill/pull/1849#issuecomment-527395687 @vvysotskyi Can you review, please? I tried to fix your comments. Also, thank you for the information on task management in jira - it looks like the last time I didn’t read the guidelines to the end (Also thanks to the @arina-ielchiieva for fixing jira task for me) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > COUNT(*) on JSON with outer list results in JsonParse error > --- > > Key: DRILL-7362 > URL: https://issues.apache.org/jira/browse/DRILL-7362 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Assignee: Oleg Zinoviev >Priority: Major > Fix For: 1.17.0 > > > Count from a JSON file with a outer array results in JsonParseException: > Cannot read from the middle of a record. Current token was START_ARRAY. > P.S. A simple select from such file works normally. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7096) Develop vector for canonical Map
[ https://issues.apache.org/jira/browse/DRILL-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921251#comment-16921251 ] ASF GitHub Bot commented on DRILL-7096: --- KazydubB commented on issue #1829: DRILL-7096: Develop vector for canonical Map URL: https://github.com/apache/drill/pull/1829#issuecomment-527358737 @ihuzenko I've updated the PR. @vvysotskyi yes, you're right. Reverted generics. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Develop vector for canonical Map > - > > Key: DRILL-7096 > URL: https://issues.apache.org/jira/browse/DRILL-7096 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Igor Guzenko >Assignee: Bohdan Kazydub >Priority: Major > Fix For: 1.17.0 > > > Canonical Map datatype can be represented using combination of three > value vectors: > keysVector - vector for storing keys of each map > valuesVector - vector for storing values of each map > offsetsVector - vector for storing of start indexes of next each map > So it's not very hard to create such Map vector, but there is a major issue > with such map representation. It's hard to search maps values by key in such > vector, need to investigate some advanced techniques to make such search > efficient. Or find other more suitable options to represent map datatype in > world of vectors. > After question about maps, Apache Arrow developers responded that for Java > they don't have real Map vector, for now they just have logical Map type > definition where they define Map like: List< Struct value:value_type> >. So implementation of value vector would be useful for > Arrow too. -- This message was sent by Atlassian Jira (v8.3.2#803003)
