[jira] [Commented] (DRILL-8269) Classified metrics for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571185#comment-17571185 ] ASF GitHub Bot commented on DRILL-8269: --- cgivre merged PR #2611: URL: https://github.com/apache/drill/pull/2611 > Classified metrics for easier maintenance > - > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > 1. Add classified prefix for metrics parameters like thread, heap and etc, > for easier Classified metrics parameters for easier maintenance. > 2. Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8272) Skip MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8272. --- > Skip MAP column without children when creating parquet tables > - > > Key: DRILL-8272 > URL: https://issues.apache.org/jira/browse/DRILL-8272 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571164#comment-17571164 ] ASF GitHub Bot commented on DRILL-8268: --- jnturton commented on code in PR #2610: URL: https://github.com/apache/drill/pull/2610#discussion_r929457387 ## exec/vector/pom.xml: ## @@ -25,7 +25,7 @@ org.apache.drill.exec 2.0.0-SNAPSHOT - vector + drill-vector Review Comment: @luocooong it is only the artifactId that is inconsistent here. The name of `Drill : Exec : Vectors` already looks good to me. So I'll revert this. I agree that the renaming could not be backported like the rest of the changes here are meant to be. > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571158#comment-17571158 ] ASF GitHub Bot commented on DRILL-8268: --- jnturton commented on code in PR #2610: URL: https://github.com/apache/drill/pull/2610#discussion_r929457387 ## exec/vector/pom.xml: ## @@ -25,7 +25,7 @@ org.apache.drill.exec 2.0.0-SNAPSHOT - vector + drill-vector Review Comment: @luocooong it is only the artifactId that is inconsistent here. The name of `Drill : Exec : Vectors` already looks good to me. So I'll revert this. > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8272) Skip MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571155#comment-17571155 ] ASF GitHub Bot commented on DRILL-8272: --- jnturton merged PR #2613: URL: https://github.com/apache/drill/pull/2613 > Skip MAP column without children when creating parquet tables > - > > Key: DRILL-8272 > URL: https://issues.apache.org/jira/browse/DRILL-8272 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8272) Skip MAP column without children when creating parquet tables
[
https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571150#comment-17571150
]
ASF GitHub Bot commented on DRILL-8272:
---
jnturton commented on code in PR #2613:
URL: https://github.com/apache/drill/pull/2613#discussion_r929453717
##
exec/java-exec/src/main/codegen/templates/ParquetOutputRecordWriter.java:
##
@@ -94,23 +94,165 @@ public void setUp(MessageType schema, RecordConsumer
consumer) {
protected abstract PrimitiveType getPrimitiveType(MaterializedField field);
+ public abstract class BaseFieldConverter extends FieldConverter {
+
+public BaseFieldConverter(int fieldId, String fieldName, FieldReader
reader) {
+ super(fieldId, fieldName, reader);
+}
+
+public abstract void read();
+
+public abstract void read(int i);
+
+public abstract void consume();
+
+@Override
+public void writeField() throws IOException {
+ read();
+ consume();
+}
+ }
+
+ public class NullableFieldConverter extends FieldConverter {
+private BaseFieldConverter delegate;
+
+public NullableFieldConverter(int fieldId, String fieldName, FieldReader
reader, BaseFieldConverter delegate) {
+ super(fieldId, fieldName, reader);
+ this.delegate = delegate;
+}
+
+@Override
+public void writeField() throws IOException {
+ if (!reader.isSet()) {
+return;
+ }
+ consumer.startField(fieldName, fieldId);
+ delegate.writeField();
+ consumer.endField(fieldName, fieldId);
+}
+
+public void setPosition(int index) {
+ delegate.setPosition(index);
+}
+
+public void startField() throws IOException {
+ delegate.startField();
+}
+
+public void endField() throws IOException {
+ delegate.endField();
+}
+ }
+
+ public class RequiredFieldConverter extends FieldConverter {
+private BaseFieldConverter delegate;
+
+public RequiredFieldConverter(int fieldId, String fieldName, FieldReader
reader, BaseFieldConverter delegate) {
+ super(fieldId, fieldName, reader);
+ this.delegate = delegate;
+}
+
+@Override
+public void writeField() throws IOException {
+ consumer.startField(fieldName, fieldId);
+ delegate.writeField();
+ consumer.endField(fieldName, fieldId);
+}
+
+public void setPosition(int index) {
+ delegate.setPosition(index);
+}
+
+public void startField() throws IOException {
+ delegate.startField();
+}
+
+public void endField() throws IOException {
+ delegate.endField();
+}
+ }
+
+ public class RepeatedFieldConverter extends FieldConverter {
+
+private BaseFieldConverter delegate;
+
+public RepeatedFieldConverter(int fieldId, String fieldName, FieldReader
reader, BaseFieldConverter delegate) {
+ super(fieldId, fieldName, reader);
+ this.delegate = delegate;
+}
+
+@Override
+public void writeField() throws IOException {
+ // empty lists are represented by simply not starting a field, rather
than starting one and putting in 0 elements
+ if (reader.size() == 0) {
+return;
+ }
+ consumer.startField(fieldName, fieldId);
+ for (int i = 0; i < reader.size(); i++) {
+delegate.read(i);
+delegate.consume();
+ }
+ consumer.endField(fieldName, fieldId);
+}
+
+@Override
+public void writeListField() {
+ if (reader.size() == 0) {
+return;
+ }
+ consumer.startField(LIST, ZERO_IDX);
+ for (int i = 0; i < reader.size(); i++) {
+consumer.startGroup();
+consumer.startField(ELEMENT, ZERO_IDX);
+
+delegate.read(i);
+delegate.consume();
+
+consumer.endField(ELEMENT, ZERO_IDX);
+consumer.endGroup();
+ }
+ consumer.endField(LIST, ZERO_IDX);
+}
+
+public void setPosition(int index) {
+ delegate.setPosition(index);
+}
+
+public void startField() throws IOException {
+ delegate.startField();
+}
+
+public void endField() throws IOException {
+ delegate.endField();
+}
+ }
+
<#list vv.types as type>
<#list type.minor as minor>
<#list vv.modes as mode>
@Override
public FieldConverter getNew${mode.prefix}${minor.class}Converter(int
fieldId, String fieldName, FieldReader reader) {
-return new ${mode.prefix}${minor.class}ParquetConverter(fieldId,
fieldName, reader);
+BaseFieldConverter converter = new ${minor.class}ParquetConverter(fieldId,
fieldName, reader);
+ <#if mode.prefix == "Nullable">
+return new NullableFieldConverter(fieldId, fieldName, reader, converter);
+ <#elseif mode.prefix == "Repeated">
+return new RepeatedFieldConverter(fieldId, fieldName, reader, converter);
+ <#else>
+return new RequiredFieldConverter(fieldId, fieldName, reader, converter);
+
}
- public class ${mode.prefix}${minor.class}ParquetConverter
[jira] [Updated] (DRILL-8272) Skip MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8272: Fix Version/s: 1.20.2 > Skip MAP column without children when creating parquet tables > - > > Key: DRILL-8272 > URL: https://issues.apache.org/jira/browse/DRILL-8272 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: 1.20.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8272) Skip MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton updated DRILL-8272: Affects Version/s: 1.20.1 > Skip MAP column without children when creating parquet tables > - > > Key: DRILL-8272 > URL: https://issues.apache.org/jira/browse/DRILL-8272 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DRILL-8261) Make the XML format plugin's attributes MAP an implicit column
[ https://issues.apache.org/jira/browse/DRILL-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Turton closed DRILL-8261. --- Resolution: Not A Problem > Make the XML format plugin's attributes MAP an implicit column > -- > > Key: DRILL-8261 > URL: https://issues.apache.org/jira/browse/DRILL-8261 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - XML >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 2.0.0 > > > The attributes MAP column generated by the XML format plugin is currently > explicit and present in wildcard selects. However, attributes are frequently > not present at all in XML data, or are better queried using explicit > projections of the individual attributes of interest to scalars. The > motivating example here is an ETL-style query that transforms XML to Parquet > using a CTAS with a wildcard column spec. This query will fail for XML that > has no attributes because the Parquet writer cannot write a Parquet schema > containing the empty struct produced by the attributes map. It is therefore > proposed that the attributes MAP becomes an implicit column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8261) Make the XML format plugin's attributes MAP an implicit column
[ https://issues.apache.org/jira/browse/DRILL-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571149#comment-17571149 ] James Turton commented on DRILL-8261: - Empty structs start to be skipped by the Parquet writer in the linked ticket, making this ticket obsolete. > Make the XML format plugin's attributes MAP an implicit column > -- > > Key: DRILL-8261 > URL: https://issues.apache.org/jira/browse/DRILL-8261 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - XML >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Minor > Fix For: 2.0.0 > > > The attributes MAP column generated by the XML format plugin is currently > explicit and present in wildcard selects. However, attributes are frequently > not present at all in XML data, or are better queried using explicit > projections of the individual attributes of interest to scalars. The > motivating example here is an ETL-style query that transforms XML to Parquet > using a CTAS with a wildcard column spec. This query will fail for XML that > has no attributes because the Parquet writer cannot write a Parquet schema > containing the empty struct produced by the attributes map. It is therefore > proposed that the attributes MAP becomes an implicit column. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571148#comment-17571148 ] ASF GitHub Bot commented on DRILL-8268: --- luocooong commented on code in PR #2610: URL: https://github.com/apache/drill/pull/2610#discussion_r929442941 ## exec/vector/pom.xml: ## @@ -25,7 +25,7 @@ org.apache.drill.exec 2.0.0-SNAPSHOT - vector + drill-vector Review Comment: Good point. It is recommended to update the name of the module instead of the ID, and consistency will do better. Because we also need to treat this pull request that allows backport. > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571141#comment-17571141 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre merged PR #2585: URL: https://github.com/apache/drill/pull/2585 > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8272) Skip MAP column without children when creating parquet tables
[ https://issues.apache.org/jira/browse/DRILL-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571112#comment-17571112 ] ASF GitHub Bot commented on DRILL-8272: --- vvysotskyi opened a new pull request, #2613: URL: https://github.com/apache/drill/pull/2613 # [DRILL-8272](https://issues.apache.org/jira/browse/DRILL-8272): Skip MAP column without children when creating parquet tables ## Description TBA ## Documentation NA ## Testing Unit tests pass > Skip MAP column without children when creating parquet tables > - > > Key: DRILL-8272 > URL: https://issues.apache.org/jira/browse/DRILL-8272 > Project: Apache Drill > Issue Type: Bug >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8272) Skip MAP column without children when creating parquet tables
Vova Vysotskyi created DRILL-8272: - Summary: Skip MAP column without children when creating parquet tables Key: DRILL-8272 URL: https://issues.apache.org/jira/browse/DRILL-8272 Project: Apache Drill Issue Type: Bug Reporter: Vova Vysotskyi Assignee: Vova Vysotskyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571045#comment-17571045 ] ASF GitHub Bot commented on DRILL-8235: --- vdiravka commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929183640 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: @cgivre In any case, if the issue persists, there is a [way](https://stackoverflow.com/questions/987142/make-gitignore-ignore-everything-except-a-few-files) how to exclude only one file from the pattern: `!contrib/native/client/patches/zookeeper-3.4.6-x64.patch` > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571012#comment-17571012 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929119471 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: Done! > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571010#comment-17571010 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929116137 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: Sounds good. Updating README. > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571004#comment-17571004 ] ASF GitHub Bot commented on DRILL-8235: --- jnturton commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929111001 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: I think the .gitignore is working properly already because all of our previous commits have left zookeeper-3.4.6-x64.patch in peace. How about we do it this way? If this PR wants to delete the patch then it should simultaneously update readme.win.txt loosely saying "ZooKeeper 3.4.6 is ancient, ZooKeeper 3.5.7 is current and does not need any patch to build on Windows so the patch remains in the Git history only". Alternatively, we revert the deletion here and carry the cruft a bit longer. +1 one from me otherwise. > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570998#comment-17570998 ] ASF GitHub Bot commented on DRILL-8235: --- jnturton commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929029434 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: If this PR really wants to be the one to delete this file from ~~version control~~ master then I think it should include some comment in the description like "this patch was added for X and is no longer relevant". I can see that it's a C code patch to ZooKeeper with a title of `[PATCH] Allow zookeeper to build in x64` which makes me 99% sure it's completely obsolete. @vdiravka, @vvysotskyi any advice for us here? > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570982#comment-17570982 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929065365 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: @vdiravka Forgive my lack of knowledge here, but the current `.gitignore` excludes patch files. Is there a way to ignore all patch files, but leave that one? How would you suggest we fix this? @jnturton Aside from this patch file, are we good to go, pending CI? > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570977#comment-17570977 ] ASF GitHub Bot commented on DRILL-8235: --- vdiravka commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929060432 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: There is [readme.win.txt](https://github.com/apache/drill/blob/master/contrib/native/client/readme.win.txt#L135) about it: > The 3.4.6 release of Zookeeper does not build correctly on 64 bit windows. To fix that for the 64 bit build, apply patch zookeeper-3.4.6-x64.patch > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8271) Make Storage and Format Config Case Insensitive
[ https://issues.apache.org/jira/browse/DRILL-8271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570970#comment-17570970 ] ASF GitHub Bot commented on DRILL-8271: --- cgivre opened a new pull request, #2612: URL: https://github.com/apache/drill/pull/2612 # [DRILL-8271](https://issues.apache.org/jira/browse/DRILL-8271): Make Storage and Format Config Case Insensitive ## Description Allows format and storage configs to be case insensitive. ## Documentation N/A ## Testing Ran existing unit tests. > Make Storage and Format Config Case Insensitive > --- > > Key: DRILL-8271 > URL: https://issues.apache.org/jira/browse/DRILL-8271 > Project: Apache Drill > Issue Type: Task >Reporter: Charles Givre >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8271) Make Storage and Format Config Case Insensitive
Charles Givre created DRILL-8271: Summary: Make Storage and Format Config Case Insensitive Key: DRILL-8271 URL: https://issues.apache.org/jira/browse/DRILL-8271 Project: Apache Drill Issue Type: Task Reporter: Charles Givre -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570965#comment-17570965 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929038841 ## logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java: ## @@ -33,6 +34,7 @@ @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type") @JsonInclude(JsonInclude.Include.NON_DEFAULT) +@JsonFormat(with = JsonFormat.Feature.ACCEPT_CASE_INSENSITIVE_PROPERTIES) Review Comment: Done! > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (DRILL-8270) Delete absolete zookeeper patch (tech debt)
[ https://issues.apache.org/jira/browse/DRILL-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Givre reassigned DRILL-8270: Assignee: Charles Givre > Delete absolete zookeeper patch (tech debt) > --- > > Key: DRILL-8270 > URL: https://issues.apache.org/jira/browse/DRILL-8270 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 2.0.0 > > > Patch files are in the `.gitignore` and yet a .patch file > ([contrib/native/client/patches/zookeeper-3.4.6-x64.patch|https://github.com/apache/drill/pull/2585/files/06625708f0419442d823d0025afa6e043fffcc4e#diff-0b6d0330fc567658b83263c83e902ec72dc0e95bb0ad0830736dc5cae8449168]) > somehow has been included in the Drill build. This PR removes it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8270) Delete absolete zookeeper patch (tech debt)
[ https://issues.apache.org/jira/browse/DRILL-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Givre updated DRILL-8270: - Affects Version/s: 1.20.1 > Delete absolete zookeeper patch (tech debt) > --- > > Key: DRILL-8270 > URL: https://issues.apache.org/jira/browse/DRILL-8270 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.20.1 >Reporter: Charles Givre >Priority: Minor > > Patch files are in the `.gitignore` and yet a .patch file > ([contrib/native/client/patches/zookeeper-3.4.6-x64.patch|https://github.com/apache/drill/pull/2585/files/06625708f0419442d823d0025afa6e043fffcc4e#diff-0b6d0330fc567658b83263c83e902ec72dc0e95bb0ad0830736dc5cae8449168]) > somehow has been included in the Drill build. This PR removes it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8270) Delete absolete zookeeper patch (tech debt)
[ https://issues.apache.org/jira/browse/DRILL-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Givre updated DRILL-8270: - Description: Patch files are in the `.gitignore` and yet a .patch file ([contrib/native/client/patches/zookeeper-3.4.6-x64.patch|https://github.com/apache/drill/pull/2585/files/06625708f0419442d823d0025afa6e043fffcc4e#diff-0b6d0330fc567658b83263c83e902ec72dc0e95bb0ad0830736dc5cae8449168]) somehow has been included in the Drill build. This PR removes it. > Delete absolete zookeeper patch (tech debt) > --- > > Key: DRILL-8270 > URL: https://issues.apache.org/jira/browse/DRILL-8270 > Project: Apache Drill > Issue Type: Task >Reporter: Charles Givre >Priority: Minor > > Patch files are in the `.gitignore` and yet a .patch file > ([contrib/native/client/patches/zookeeper-3.4.6-x64.patch|https://github.com/apache/drill/pull/2585/files/06625708f0419442d823d0025afa6e043fffcc4e#diff-0b6d0330fc567658b83263c83e902ec72dc0e95bb0ad0830736dc5cae8449168]) > somehow has been included in the Drill build. This PR removes it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8270) Delete absolete zookeeper patch (tech debt)
[ https://issues.apache.org/jira/browse/DRILL-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Givre updated DRILL-8270: - Fix Version/s: 2.0.0 > Delete absolete zookeeper patch (tech debt) > --- > > Key: DRILL-8270 > URL: https://issues.apache.org/jira/browse/DRILL-8270 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.20.1 >Reporter: Charles Givre >Priority: Minor > Fix For: 2.0.0 > > > Patch files are in the `.gitignore` and yet a .patch file > ([contrib/native/client/patches/zookeeper-3.4.6-x64.patch|https://github.com/apache/drill/pull/2585/files/06625708f0419442d823d0025afa6e043fffcc4e#diff-0b6d0330fc567658b83263c83e902ec72dc0e95bb0ad0830736dc5cae8449168]) > somehow has been included in the Drill build. This PR removes it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8270) Delete absolete zookeeper patch (tech debt)
Charles Givre created DRILL-8270: Summary: Delete absolete zookeeper patch (tech debt) Key: DRILL-8270 URL: https://issues.apache.org/jira/browse/DRILL-8270 Project: Apache Drill Issue Type: Task Reporter: Charles Givre -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570961#comment-17570961 ] ASF GitHub Bot commented on DRILL-8235: --- cgivre commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929030265 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: @jnturton I'll create a separate JIRA and add it to this ticket. > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570960#comment-17570960 ] ASF GitHub Bot commented on DRILL-8235: --- jnturton commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929029434 ## contrib/native/client/patches/zookeeper-3.4.6-x64.patch: ## @@ -1,163 +0,0 @@ -From 64697ddd8a90f29d1693658f04e975e435e3c869 Mon Sep 17 00:00:00 2001 Review Comment: If this PR really wants to be the one to delete this file from version control then I think it should include some comment in the description like "this patch was added for X and is no longer relevant". I can see that it's a C code patch to ZooKeeper with a title of `[PATCH] Allow zookeeper to build in x64` which makes me 99% sure it's completely obsolete. @vdiravka, @vvysotskyi any advice for us here? > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8235) Add Storage Plugin for Google Sheets
[ https://issues.apache.org/jira/browse/DRILL-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570955#comment-17570955 ] ASF GitHub Bot commented on DRILL-8235: --- jnturton commented on code in PR #2585: URL: https://github.com/apache/drill/pull/2585#discussion_r929020952 ## logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java: ## @@ -33,6 +34,7 @@ @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type") @JsonInclude(JsonInclude.Include.NON_DEFAULT) +@JsonFormat(with = JsonFormat.Feature.ACCEPT_CASE_INSENSITIVE_PROPERTIES) Review Comment: Yes, thank you, let's make case insensitive properties its own PR. Please revert it here. > Add Storage Plugin for Google Sheets > > > Key: DRILL-8235 > URL: https://issues.apache.org/jira/browse/DRILL-8235 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Affects Versions: 1.20.1 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 2.0.0 > > > Google Sheets is a very commonly used data source among business users. > Presto and other query engines do include integrations with Google Sheets and > so it would be useful for Drill to add this functionality. > The proposed plugin supports both reading and writing to Google Sheets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570898#comment-17570898 ] ASF GitHub Bot commented on DRILL-8268: --- jnturton commented on code in PR #2610: URL: https://github.com/apache/drill/pull/2610#discussion_r928870530 ## exec/vector/pom.xml: ## @@ -25,7 +25,7 @@ org.apache.drill.exec 2.0.0-SNAPSHOT - vector + drill-vector Review Comment: @luocooong it was seeing the effort already made in [DRILL-7847](https://issues.apache.org/jira/browse/DRILL-7847) that caused me to try to complete the job. I do like completeness and consistency but the name of the vector module also does not keep me awake at night so I won't argue much here. Do you recommend that I revert this part of the PR? > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8269) Classified metrics for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570883#comment-17570883 ] ASF GitHub Bot commented on DRILL-8269: --- kingswanwho opened a new pull request, #2611: URL: https://github.com/apache/drill/pull/2611 # [DRILL-8269](https://issues.apache.org/jira/browse/DRILL-8269): Classified metrics for easier maintenance ## Description 1. Add classified prefix for metrics indicators like thread, heap, jvm and etc, so that user can easily make sense of each indicators. 2. Change the key reference of usage summary on Drill Web UI Main Page and Metrics page due to the key of each indicators has been changed. The usage can be shown correctly.  3. Drill Web UI Metrics page usage part cannot fully show the numbers if the bar isn't long enough.  After Optimized UI,  ## Documentation I will also update the corresponding Drill Doc later. https://drill.apache.org/docs/monitoring-metrics/ ## Testing Manually test and UT. > Classified metrics for easier maintenance > - > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > 1. Add classified prefix for metrics parameters like thread, heap and etc, > for easier Classified metrics parameters for easier maintenance. > 2. Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Summary: Classified metrics for easier maintenance (was: Classified metrics parameters for easier maintenance) > Classified metrics for easier maintenance > - > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > 1. Add classified prefix for metrics parameters like thread, heap and etc, > for easier Classified metrics parameters for easier maintenance. > 2. Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570870#comment-17570870 ] ASF GitHub Bot commented on DRILL-8268: --- martin-g commented on PR #2610: URL: https://github.com/apache/drill/pull/2610#issuecomment-1193986421 @luocooong See https://stackoverflow.com/a/61267496/497381 ` java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;` is not available in JDK 1.8. It is available since Java 11. In 1.8 the return type is `Ljava/nio/Buffer;`. I am not sure why Github CI / Main Build (8)` passes. Maybe it does not cover this code path ?! > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[
https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570863#comment-17570863
]
ASF GitHub Bot commented on DRILL-8268:
---
luocooong commented on code in PR #2610:
URL: https://github.com/apache/drill/pull/2610#discussion_r928797183
##
exec/vector/pom.xml:
##
@@ -25,7 +25,7 @@
org.apache.drill.exec
2.0.0-SNAPSHOT
- vector
+ drill-vector
Review Comment:
Change is not the best thing without more new things.
##
pom.xml:
##
@@ -3965,6 +3977,46 @@
+
Review Comment:
The dependence on Hadoop was tricky. Ensure that the dependency tree does
not have a new dependency.
##
exec/java-exec/src/main/resources/drill-module.conf:
##
@@ -190,7 +190,9 @@ drill.exec: {
maximum: 9223372036854775807
}
},
-memory.heap.failure.threshold: 0.85,
+# Default to failing queries only at 100% heap usage, i.e. the heap usage
+# limiting logic in the REST API is disabled by default.
+memory.heap.failure.threshold: 1.0,
Review Comment:
That's okay.
In fact, this option is only used for REST modules and it has little effect
before Streaming RESTful feature joins because we cannot expect users to send
long or short queries.
> Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
> ---
>
> Key: DRILL-8268
> URL: https://issues.apache.org/jira/browse/DRILL-8268
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.20.1
>Reporter: James Turton
>Assignee: James Turton
>Priority: Major
> Fix For: 1.20.2
>
>
> # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2
> profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2.
> # We remove the netty-all metapackage which entered the dependency tree with
> the change introducing the Netty bom bringing many uneeded libs with it.
> # The heap memory usage limiting logic in the REST server becomes disabled
> by default since REST query results are streamed these days. This change aims
> to let the Java GC now do its job without interference and if that results in
> OOM under a constant load then there is good evidence for a heap leak which
> must be tracked down and completely resolved anyway, not mitigated or "swept
> under a rug".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics parameters for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Description: 1. Add classified prefix for metrics parameters like thread, heap and etc, for easier Classified metrics parameters for easier maintenance. 2. Optimized UI. was: Add classified prefix for metrics parameters like thread, heap and etc, for easier Classified metrics parameters for easier maintenance. Optimized UI. > Classified metrics parameters for easier maintenance > > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > 1. Add classified prefix for metrics parameters like thread, heap and etc, > for easier Classified metrics parameters for easier maintenance. > 2. Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics parameters for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Description: Add classified prefix for metrics parameters like thread, heap and etc, for easier Classified metrics parameters for easier maintenance. Optimized UI. was: Add classified prefix for metrics parameters like thread, heap and etc, for easier Classified metrics parameters for easier maintainence. Optimized UI. > Classified metrics parameters for easier maintenance > > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > Add classified prefix for metrics parameters like thread, heap and etc, for > easier Classified metrics parameters for easier maintenance. > Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics parameters for easier maintenance
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Summary: Classified metrics parameters for easier maintenance (was: Classified metrics parameters for easier maintainence) > Classified metrics parameters for easier maintenance > > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > Add classified prefix for metrics parameters like thread, heap and etc, for > easier Classified metrics parameters for easier maintainence. > Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics parameters for easier maintainence
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Description: Add classified prefix for metrics parameters like thread, heap and etc, for easier Classified metrics parameters for easier maintainence. Optimized UI. was: Add classified prefix for metrics parameters like thread, heap and etc, for easier Optimized UI. > Classified metrics parameters for easier maintainence > - > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > Add classified prefix for metrics parameters like thread, heap and etc, for > easier Classified metrics parameters for easier maintainence. > Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DRILL-8269) Classified metrics parameters for easier maintainence
[ https://issues.apache.org/jira/browse/DRILL-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingchuan Hu updated DRILL-8269: Description: Add classified prefix for metrics parameters like thread, heap and etc, for easier Optimized UI. was:Add classified prefix for metrics parameters > Classified metrics parameters for easier maintainence > - > > Key: DRILL-8269 > URL: https://issues.apache.org/jira/browse/DRILL-8269 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jingchuan Hu >Priority: Major > > Add classified prefix for metrics parameters like thread, heap and etc, for > easier > Optimized UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DRILL-8269) Classified metrics parameters for easier maintainence
Jingchuan Hu created DRILL-8269: --- Summary: Classified metrics parameters for easier maintainence Key: DRILL-8269 URL: https://issues.apache.org/jira/browse/DRILL-8269 Project: Apache Drill Issue Type: Improvement Reporter: Jingchuan Hu Add classified prefix for metrics parameters -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8268) Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default
[ https://issues.apache.org/jira/browse/DRILL-8268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570821#comment-17570821 ] ASF GitHub Bot commented on DRILL-8268: --- jnturton commented on PR #2610: URL: https://github.com/apache/drill/pull/2610#issuecomment-1193860317 @luocooong @paul-rogers a bit of data related to the heap memory usage limiting from some experiments run by a helpful user. I did test the recommendation given by you and it worked well. Below are steps of testing > drill.exec.http.memory.heap.failure.threshold = 2 > > Before making changes in the drill-override.conf > 1. Opened 4 sessions of drill web UI query > 2. Fired same query in all 4 sessions. This query will scan through 8000 parquet compress files > > Results - all 4 sessions gave heap error. > Observations > 1. The heap memory displayed on "metrics" tab varies between 30-80% while query is running > 2. The heap memory is constant at 87.89% after queries are failed > > After making the changes in the drill-overide.conf as recommended by you > 1. Opened 4 sessions of drill web UI query > 2. Fired same query in all 4 sessions. This query will scan through 8000 parquet compress files > Results - All 4 sessions returned the results. It took average 6+ minutes to return the results > Observations > 1. Could not observe the heap memory displayed on "metrics" as display was just in running state and then gave error "page cannot be displayed" during query run > 2. The heap memory is varying between 69.71% after queries are successfully completed > > I ran the same queries again and again and below is the heap memory after every run > 2nd run - Between 61-62% > 3rd run - . Between 82-83% > 4th Run - Between 69-70% > 5th Run - Between 65-66% > 6th run - Between 69-70% > 7th run - Between 70-71% > 8th run - increased one session, so now total are 5 web sessions - 1 session gave "Page cannot be displayed error. 4 returned the results - Memory 79-80% > 9th run (5 sessions) - - 1 session gave "Page cannot be displayed error. 4 returned the results - Memory 66-67% > > Fix Hadoop 2 and Netty lib exclusions, REST mem limiter disabled by default > --- > > Key: DRILL-8268 > URL: https://issues.apache.org/jira/browse/DRILL-8268 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.20.1 >Reporter: James Turton >Assignee: James Turton >Priority: Major > Fix For: 1.20.2 > > > # New exclusions of reload4j, slf4j-reload4j are required in the Hadoop 2 > profile, probably due to the upgrade of Hadoop from 2.10.1 to 2.10.2. > # We remove the netty-all metapackage which entered the dependency tree with > the change introducing the Netty bom bringing many uneeded libs with it. > # The heap memory usage limiting logic in the REST server becomes disabled > by default since REST query results are streamed these days. This change aims > to let the Java GC now do its job without interference and if that results in > OOM under a constant load then there is good evidence for a heap leak which > must be tracked down and completely resolved anyway, not mitigated or "swept > under a rug". -- This message was sent by Atlassian Jira (v8.20.10#820010)
