[GitHub] [druid] irwvanroosmalen closed issue #10596: New metrics are 0 on Druid parallel reingestion
irwvanroosmalen closed issue #10596: URL: https://github.com/apache/druid/issues/10596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #10363: fix injection failure of StorageLocationSelectorStrategy objects
suneet-s commented on pull request #10363: URL: https://github.com/apache/druid/pull/10363#issuecomment-731479320 @FrankChen021 - I got pulled in to some other issues and haven't had the time to look at this yet. I will look at this over the weekend and get back to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (111b431 -> 4537016)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 111b431 Introduce query/timeout/count metric (#10567) add 4537016 Update google client libraries (#10536) No new revisions were added by this update. Summary of changes: licenses.yaml | 10 +- pom.xml | 11 ++- 2 files changed, 11 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #10536: Update google client libraries
himanshug merged pull request #10536: URL: https://github.com/apache/druid/pull/10536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on pull request #10541: AWS Web Identity / IRSA Support
himanshug commented on pull request #10541: URL: https://github.com/apache/druid/pull/10541#issuecomment-731457432 +1 after the build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #10555: Fix : Druid throws java.util.concurrent.RejectedExecutionException when ingest task is stopping.
himanshug commented on a change in pull request #10555:
URL: https://github.com/apache/druid/pull/10555#discussion_r528021227
##
File path:
core/src/main/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutors.java
##
@@ -53,7 +53,11 @@ public static void scheduleWithFixedDelay(
public Signal call()
{
runnable.run(); // (Exceptions are handled for us)
-return Signal.REPEAT;
+if (exec.isShutdown()) {
+ return Signal.STOP;
Review comment:
nit: I would make that a `log.warn(..)` because debug would hardly be
enabled and this log is not gonna pollute the logs, but looks good otherwise.
thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (3cafd53 -> 111b431)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 3cafd53 fix issue causing incorrect config in Docker (#10595) add 111b431 Introduce query/timeout/count metric (#10567) No new revisions were added by this update. Summary of changes: docs/operations/metrics.md | 9 +- .../src/main/resources/defaultMetrics.json | 1 + .../main/resources/defaultMetricDimensions.json| 1 + .../druid/server/AsyncQueryForwardingServlet.java | 8 ++ .../org/apache/druid/server/QueryResource.java | 9 +- .../server/metrics/QueryCountStatsMonitor.java | 7 +- .../server/metrics/QueryCountStatsProvider.java| 14 .../org/apache/druid/server/QueryResourceTest.java | 2 + .../server/metrics/QueryCountStatsMonitorTest.java | 96 ++ 9 files changed, 141 insertions(+), 6 deletions(-) create mode 100644 server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #10567: Introduce query/timeout/count metric
himanshug merged pull request #10567: URL: https://github.com/apache/druid/pull/10567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on pull request #10495: Added Request log updates for status change on cooridnator / overlord…
himanshug commented on pull request #10495: URL: https://github.com/apache/druid/pull/10495#issuecomment-731452856 > It will open flood gate and finding things will be needle in a haystack current patch prints all of non-GET requests , so by setting "org.apache.druid.jetty.RequestLog" logger to debug in coordinator/overlord would lead to print all GET requests additionally... I didn't think coordinator/overlord would have so many of those . Have you tried and found an overwhelming number of GETs coming to coordinator/overlord (maybe I am forgetting something that periodically is sending requests to them). all the other problems you mentioned are tooling things, e.g. you would anyways have to manage different configuration for different environments. anyways, current patch appears to introduce work for all processes (broker/mm/historical) where the predicates will ALWAYS be false .. so there needs to be a way to make this optional so that users can opt-in to it rather than always getting it. you could probably introduce a flag in existing code where you can optionally disable the printing of GET requests and that would solve the flood gate problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on pull request #10562: Add TravisCI job that builds and tests on ARM64 CPU architecture
himanshug commented on pull request #10562: URL: https://github.com/apache/druid/pull/10562#issuecomment-731449599 yeah, that is technically true and more testing is always better :) however, I think trade-off here is the build time which is probably ok considering the new build runs in parallel to others hopefully , however travis might put some limits on build resources used overall/concurrently and that would actually increase it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #10595: Fix issue causing incorrect config in Docker
himanshug merged pull request #10595: URL: https://github.com/apache/druid/pull/10595 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (2201ffa -> 3cafd53)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 2201ffa druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup (#10591) add 3cafd53 fix issue causing incorrect config in Docker (#10595) No new revisions were added by this update. Summary of changes: distribution/docker/druid.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug merged pull request #10591: druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup
himanshug merged pull request #10591: URL: https://github.com/apache/druid/pull/10591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (ba915b7 -> 2201ffa)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from ba915b7 Security overview documentation (#10339) add 2201ffa druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup (#10591) No new revisions were added by this update. Summary of changes: distribution/docker/druid.sh | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #10597: Web console: improve how code is imported, use API instance
vogievetsky commented on a change in pull request #10597:
URL: https://github.com/apache/druid/pull/10597#discussion_r527855571
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -121,8 +120,8 @@ import {
upgradeSpec,
} from '../../druid-models';
import { getLink } from '../../links';
-import { AppToaster } from '../../singletons/toaster';
-import { UrlBaser } from '../../singletons/url-baser';
+import { Api, AppToaster } from '../../singletons';
+import { UrlBaser } from '../../singletons';
Review comment:
oh yeah, search replace victim
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jgoz commented on a change in pull request #10597: Web console: improve how code is imported, use API instance
jgoz commented on a change in pull request #10597:
URL: https://github.com/apache/druid/pull/10597#discussion_r527812621
##
File path: web-console/src/components/show-log/show-log.tsx
##
@@ -17,13 +17,12 @@
*/
import { AnchorButton, Button, ButtonGroup, Intent, Switch } from
'@blueprintjs/core';
-import axios from 'axios';
import copy from 'copy-to-clipboard';
import React from 'react';
import { Loader } from '../../components';
-import { AppToaster } from '../../singletons/toaster';
-import { UrlBaser } from '../../singletons/url-baser';
+import { Api, AppToaster } from '../../singletons';
+import { UrlBaser } from '../../singletons';
Review comment:
Minor: these imports can be merged
##
File path: web-console/src/entry.ts
##
@@ -69,16 +67,23 @@ if (typeof consoleConfig.title === 'string') {
window.document.title = consoleConfig.title;
}
+const apiConfig: AxiosRequestConfig = {
+ headers: {},
+};
+
if (consoleConfig.baseURL) {
- axios.defaults.baseURL = consoleConfig.baseURL;
+ apiConfig.baseURL = consoleConfig.baseURL;
UrlBaser.baseUrl = consoleConfig.baseURL;
}
if (consoleConfig.customHeaderName && consoleConfig.customHeaderValue) {
- axios.defaults.headers.common[consoleConfig.customHeaderName] =
consoleConfig.customHeaderValue;
+ apiConfig.headers.common[consoleConfig.customHeaderName] =
consoleConfig.customHeaderValue;
}
if (consoleConfig.customHeaders) {
- Object.assign(axios.defaults.headers, consoleConfig.customHeaders);
+ Object.assign(apiConfig.headers, consoleConfig.customHeaders);
}
+
+Api.initialize(apiConfig);
Review comment:
nice
##
File path: web-console/src/components/show-json/show-json.tsx
##
@@ -17,13 +17,12 @@
*/
import { Button, ButtonGroup, Intent, TextArea } from '@blueprintjs/core';
-import axios from 'axios';
import copy from 'copy-to-clipboard';
import React from 'react';
import { useQueryManager } from '../../hooks';
-import { AppToaster } from '../../singletons/toaster';
-import { UrlBaser } from '../../singletons/url-baser';
+import { Api, AppToaster } from '../../singletons';
+import { UrlBaser } from '../../singletons';
Review comment:
Minor: these imports can be merged
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -121,8 +120,8 @@ import {
upgradeSpec,
} from '../../druid-models';
import { getLink } from '../../links';
-import { AppToaster } from '../../singletons/toaster';
-import { UrlBaser } from '../../singletons/url-baser';
+import { Api, AppToaster } from '../../singletons';
+import { UrlBaser } from '../../singletons';
Review comment:
Minor: merged
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] senthilkv commented on pull request #10495: Added Request log updates for status change on cooridnator / overlord…
senthilkv commented on pull request #10495: URL: https://github.com/apache/druid/pull/10495#issuecomment-731269656 Hi @himanshug , Yes we have considered that. Couple of reason not doing is 1. It will open flood gate and finding things will be needle in a haystack. 2. When you have multiple installations it becomes cumbersome to manage these different configs. Regarding sending the logs 2 file I agree we can update the log4j. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] a2l007 commented on pull request #10551: Remove hard limitation that druid(after 0.15.0) only can consume Kafka version 0.11.x or better
a2l007 commented on pull request #10551: URL: https://github.com/apache/druid/pull/10551#issuecomment-731267750 Presently, the read_committed isolation level supports transactional producers as well as preserves the behavior for non-transactional producers. If this property is made user-configurable, it is possible that a cluster operator may unintentionally disable it for higher versions of Kafka that support transactional topics. Are there any issues that may arise if we disable this property for higher versions of Kafka? If so, I think we should have some sort of logging that makes the operator aware of it. Also with non-transactional producers in lower versions of Kafka, it _may_ be reasonable to expect that offsets are sequential. Since we no longer have the offset gap check in Druid, the docs may need to make it clear that Druid doesn't perform this check. @surekhasaharan Since you had introduced the transactional topic support in Druid, I wanted to check if you had any comments about this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #10363: fix injection failure of StorageLocationSelectorStrategy objects
FrankChen021 commented on pull request #10363: URL: https://github.com/apache/druid/pull/10363#issuecomment-731087817 Hi @suneet-s , Is this PR ready to be merged ? or is there anything I need to do ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 opened a new pull request #10598: Improve doc and exception message for invalid user configurations
FrankChen021 opened a new pull request #10598: URL: https://github.com/apache/druid/pull/10598 This PR is an optimization according to the feedback #10579 Both `druid.processing.buffer.sizeBytes` and `druid.indexer.runner.maxZnodeBytes` have an upper size limit (2GiB) according to current implementation, but these limits are not stated in the doc. So, this PR 1. improves the doc 2. improves the exception message for invalid configurations so that user could resolve exceptions by themselves according to the more meaningful message This PR has: - [X] been self-reviewed. - [] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [X] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
