[GitHub] [druid] vogievetsky opened a new pull request #10888: Web console: remove namespace prop that does not exist from JDBC lookup
vogievetsky opened a new pull request #10888: URL: https://github.com/apache/druid/pull/10888 Turns out there is no `namespace` property in JDBC lookups. https://github.com/apache/druid/blob/753bce324bdf8c7c5b2b602f89c720749bfa6e22/extensions-core/lookups-cached-global/src/main/java/org/apache/druid/query/lookup/namespace/JdbcExtractionNamespace.java#L41 Removing it to make the UI more accurate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website] vogievetsky merged pull request #116: Autobuild
vogievetsky merged pull request #116: URL: https://github.com/apache/druid-website/pull/116 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website] branch asf-site updated: Autobuild (#116)
This is an automated email from the ASF dual-hosted git repository. vogievetsky pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/druid-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 57da8ed Autobuild (#116) 57da8ed is described below commit 57da8ed989fbfb6679eab69e75bdca1bcd9647d7 Author: Vadim Ogievetsky AuthorDate: Fri Feb 12 22:44:06 2021 -0800 Autobuild (#116) * Autobuild * restore files --- druid-powered.html | 6 ++ index.html | 16 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/druid-powered.html b/druid-powered.html index 5d22cfd..373a8f8 100644 --- a/druid-powered.html +++ b/druid-powered.html @@ -749,6 +749,12 @@ issue related to Druid setup and tuning. Sift Science provides an online trust platform that online businesses use to prevent fraud and abuse. We use Druid as a tool to gain real-time insights about our data and machine learning models. +SigNoz + +https://signoz.io;>SigNoz is an open source observability platform. SigNoz uses distributed tracing to gain visibility into your systems and powers data using Kafka (to handle high ingestion rate and backpressure) and Apache Druid (Apache Druid is a high performance real-time analytics database), both proven in industry to handle scale. + +Druid powers aggregations after slicing and dicing of high-dimensional trace data. + Sina Weibo Weibo Advertising Platform deploys Druid as realtime data tool for online advertising analytics and business intelligence(BI). Druid processes TBs of realtime data per day with latency in one minute. diff --git a/index.html b/index.html index 75fa4aa..1571f87 100644 --- a/index.html +++ b/index.html @@ -221,15 +221,15 @@ -Jan +Feb -21 +23 - https://www.meetup.com/druidio/events/275628814/;> -The Superstars of Apache Druid + https://www.meetup.com/druidio/events/276051901;> +February Druid Office Hours Bay Area Apache Druid Meetup by Imply @@ -238,15 +238,15 @@ -Jan +Feb -27 +25 - https://www.meetup.com/Apache-Druid-London/events/275627138;> -The January Apache Druid Drop In! + https://www.meetup.com/Apache-Druid-London/events/276052050/;> +The February Druid Drop-in! London Apache Druid Meetup by Imply - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website] vogievetsky opened a new pull request #116: Autobuild
vogievetsky opened a new pull request #116: URL: https://github.com/apache/druid-website/pull/116 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] vogievetsky merged pull request #207: fix event indentation
vogievetsky merged pull request #207: URL: https://github.com/apache/druid-website-src/pull/207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated: fix event indentation
This is an automated email from the ASF dual-hosted git repository. vogievetsky pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new fddecc8 fix event indentation new f890aca Merge pull request #207 from implydata/fix-indent fddecc8 is described below commit fddecc87e3d72f10de0b7ce40c9647b68aa6efc2 Author: Vadim Ogievetsky AuthorDate: Fri Feb 12 22:38:39 2021 -0800 fix event indentation --- _data/events.yml | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_data/events.yml b/_data/events.yml index 1c60253..888bfb5 100644 --- a/_data/events.yml +++ b/_data/events.yml @@ -1,10 +1,10 @@ - date: 2021-02-23 -name: February Druid Office Hours -info: Bay Area Apache Druid Meetup by Imply -link: https://www.meetup.com/druidio/events/276051901 + name: February Druid Office Hours + info: Bay Area Apache Druid Meetup by Imply + link: https://www.meetup.com/druidio/events/276051901 - date: 2021-02-25 -name: The February Druid Drop-in! -info: London Apache Druid Meetup by Imply -link: https://www.meetup.com/Apache-Druid-London/events/276052050/ + name: The February Druid Drop-in! + info: London Apache Druid Meetup by Imply + link: https://www.meetup.com/Apache-Druid-London/events/276052050/ - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] vogievetsky opened a new pull request #207: fix event indentation
vogievetsky opened a new pull request #207: URL: https://github.com/apache/druid-website-src/pull/207 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] vogievetsky merged pull request #206: Update events.yml
vogievetsky merged pull request #206: URL: https://github.com/apache/druid-website-src/pull/206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated: Update events.yml
This is an automated email from the ASF dual-hosted git repository. vogievetsky pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new f005638 Update events.yml new bbbf0ea Merge pull request #206 from rachelpedreschi/patch-11 f005638 is described below commit f005638c29e19300f23a76410d3c8246d9fde2a4 Author: Rachel Pedreschi <53063158+rachelpedres...@users.noreply.github.com> AuthorDate: Mon Feb 8 14:52:48 2021 -0800 Update events.yml --- _data/events.yml | 17 - 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/_data/events.yml b/_data/events.yml index 097cf04..1c60253 100644 --- a/_data/events.yml +++ b/_data/events.yml @@ -1,11 +1,10 @@ +- date: 2021-02-23 +name: February Druid Office Hours +info: Bay Area Apache Druid Meetup by Imply +link: https://www.meetup.com/druidio/events/276051901 -- date: 2021-01-21 - name: The Superstars of Apache Druid - info: Bay Area Apache Druid Meetup by Imply - link: https://www.meetup.com/druidio/events/275628814/ - -- date: 2021-01-27 - name: The January Apache Druid Drop In! - info: London Apache Druid Meetup by Imply - link: https://www.meetup.com/Apache-Druid-London/events/275627138 +- date: 2021-02-25 +name: The February Druid Drop-in! +info: London Apache Druid Meetup by Imply +link: https://www.meetup.com/Apache-Druid-London/events/276052050/ - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jp707049 opened a new issue #10887: [Just Curious] About Zookeeper Dependency
jp707049 opened a new issue #10887: URL: https://github.com/apache/druid/issues/10887 Is there a roadmap for druid to remove zookeeper dependency completely. If so, what will zookeeper be replaced by? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson opened a new pull request #10886: Remove stale 'namespace' config for JDBC lookups from doc and web-console
jihoonson opened a new pull request #10886: URL: https://github.com/apache/druid/pull/10886 ### Description The `namespace` config was removed in #2926. Also fixed a wrong instruction for the JDBC connector location. This PR has: - [x] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #10885: license.yaml fixes for code introduced related to AWS RDS token based password provider in PR #9518
jihoonson commented on a change in pull request #10885: URL: https://github.com/apache/druid/pull/10885#discussion_r575581693 ## File path: licenses.yaml ## @@ -157,6 +157,15 @@ libraries: --- +name: Code adopted from org.apache.commons.dbcp2.BasicDataSource +license_category: source +module: server +license_name: Apache License version 2.0 +source_paths: + - server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java Review comment: Can you please add a notice here too? The notice for commons-dbcp can be found in https://github.com/apache/commons-dbcp/blob/master/NOTICE.txt. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug opened a new pull request #10885: license.yaml fixes for code introduced related to AWS RDS token based password provider in PR #9518
himanshug opened a new pull request #10885: URL: https://github.com/apache/druid/pull/10885 see https://github.com/apache/druid/pull/9518#discussion_r575571188 and https://github.com/apache/druid/pull/9518#discussion_r575571831 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9518: AWS RDS token based password provider
jihoonson commented on a change in pull request #9518:
URL: https://github.com/apache/druid/pull/9518#discussion_r575573699
##
File path:
server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.metadata;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.dbcp2.BasicDataSource;
+import org.apache.commons.dbcp2.ConnectionFactory;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.sql.Driver;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Properties;
+
+/**
+ * This class exists so that {@link MetadataStorageConnectorConfig} is asked
for password every time a brand new
+ * connection is established with DB. {@link PasswordProvider} impls such as
those based on AWS tokens refresh the
+ * underlying token periodically since each token is valid for a certain
period of time only.
+ * So, This class overrides (and largely copies due to lack of extensibility),
the methods from base class in order to keep
+ * track of connection properties and call {@link
MetadataStorageConnectorConfig#getPassword()} everytime a new
+ * connection is setup.
+ */
+public class BasicDataSourceExt extends BasicDataSource
+{
+ private static final Logger LOGGER = new Logger(BasicDataSourceExt.class);
+
+ private Properties connectionProperties;
+ private final MetadataStorageConnectorConfig connectorConfig;
+
+ public BasicDataSourceExt(MetadataStorageConnectorConfig connectorConfig)
+ {
+this.connectorConfig = connectorConfig;
+this.connectionProperties = new Properties();
+ }
+
+ @Override
+ public void addConnectionProperty(String name, String value)
+ {
+connectionProperties.put(name, value);
+super.addConnectionProperty(name, value);
+ }
+
+ @Override
+ public void removeConnectionProperty(String name)
+ {
+connectionProperties.remove(name);
+super.removeConnectionProperty(name);
+ }
+
+ @Override
+ public void setConnectionProperties(String connectionProperties)
+ {
+if (connectionProperties == null) {
+ throw new NullPointerException("connectionProperties is null");
+}
+
+String[] entries = connectionProperties.split(";");
+Properties properties = new Properties();
+for (String entry : entries) {
+ if (entry.length() > 0) {
+int index = entry.indexOf('=');
+if (index > 0) {
+ String name = entry.substring(0, index);
+ String value = entry.substring(index + 1);
+ properties.setProperty(name, value);
+} else {
+ // no value is empty string which is how java.util.Properties works
+ properties.setProperty(entry, "");
+}
+ }
+}
+this.connectionProperties = properties;
+super.setConnectionProperties(connectionProperties);
+ }
+
+ @VisibleForTesting
+ public Properties getConnectionProperties()
+ {
+return connectionProperties;
+ }
+
+ @Override
+ protected ConnectionFactory createConnectionFactory() throws SQLException
Review comment:
Thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9518: AWS RDS token based password provider
himanshug commented on a change in pull request #9518:
URL: https://github.com/apache/druid/pull/9518#discussion_r575573350
##
File path: licenses.yaml
##
@@ -147,6 +147,16 @@ source_paths:
---
+name: AWS RDS SDK for Java
+license_category: source
+module: extensions/druid-aws-rds-extensions
+license_name: Apache License version 2.0
+version: 1.11.199
+libraries:
+ - com.amazonaws: aws-java-sdk-rds
Review comment:
source code is not copied here , I guess this should be "binary" , yes.
.. will update.
##
File path:
server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.metadata;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.dbcp2.BasicDataSource;
+import org.apache.commons.dbcp2.ConnectionFactory;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.sql.Driver;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Properties;
+
+/**
+ * This class exists so that {@link MetadataStorageConnectorConfig} is asked
for password every time a brand new
+ * connection is established with DB. {@link PasswordProvider} impls such as
those based on AWS tokens refresh the
+ * underlying token periodically since each token is valid for a certain
period of time only.
+ * So, This class overrides (and largely copies due to lack of extensibility),
the methods from base class in order to keep
+ * track of connection properties and call {@link
MetadataStorageConnectorConfig#getPassword()} everytime a new
+ * connection is setup.
+ */
+public class BasicDataSourceExt extends BasicDataSource
+{
+ private static final Logger LOGGER = new Logger(BasicDataSourceExt.class);
+
+ private Properties connectionProperties;
+ private final MetadataStorageConnectorConfig connectorConfig;
+
+ public BasicDataSourceExt(MetadataStorageConnectorConfig connectorConfig)
+ {
+this.connectorConfig = connectorConfig;
+this.connectionProperties = new Properties();
+ }
+
+ @Override
+ public void addConnectionProperty(String name, String value)
+ {
+connectionProperties.put(name, value);
+super.addConnectionProperty(name, value);
+ }
+
+ @Override
+ public void removeConnectionProperty(String name)
+ {
+connectionProperties.remove(name);
+super.removeConnectionProperty(name);
+ }
+
+ @Override
+ public void setConnectionProperties(String connectionProperties)
+ {
+if (connectionProperties == null) {
+ throw new NullPointerException("connectionProperties is null");
+}
+
+String[] entries = connectionProperties.split(";");
+Properties properties = new Properties();
+for (String entry : entries) {
+ if (entry.length() > 0) {
+int index = entry.indexOf('=');
+if (index > 0) {
+ String name = entry.substring(0, index);
+ String value = entry.substring(index + 1);
+ properties.setProperty(name, value);
+} else {
+ // no value is empty string which is how java.util.Properties works
+ properties.setProperty(entry, "");
+}
+ }
+}
+this.connectionProperties = properties;
+super.setConnectionProperties(connectionProperties);
+ }
+
+ @VisibleForTesting
+ public Properties getConnectionProperties()
+ {
+return connectionProperties;
+ }
+
+ @Override
+ protected ConnectionFactory createConnectionFactory() throws SQLException
Review comment:
yeah, most code in this file is copied from
`org.apache.commons.dbcp2.BasicDataSource` , will send a PR shortly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional
[druid] branch 0.21.0 updated: Revert "Add support for Blacklisting some domains for HTTPInputSource (#10535)" (#10871) (#10872)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch 0.21.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/0.21.0 by this push:
new bfc066c Revert "Add support for Blacklisting some domains for
HTTPInputSource (#10535)" (#10871) (#10872)
bfc066c is described below
commit bfc066cdf577c0a8e44c8b0186931a749128e5cf
Author: Jihoon Son
AuthorDate: Fri Feb 12 15:27:30 2021 -0800
Revert "Add support for Blacklisting some domains for HTTPInputSource
(#10535)" (#10871) (#10872)
This reverts commit 6b14bdb3a53d6aec45e485e6849956a69720ba3f.
---
.../druid/data/input/impl/HttpInputSource.java | 15 +--
.../data/input/impl/HttpInputSourceConfig.java | 112 -
.../druid/data/input/impl/HttpInputSourceTest.java | 74 +-
docs/configuration/index.md| 9 --
.../druid/metadata/input/InputSourceModule.java| 3 -
.../metadata/input/InputSourceModuleTest.java | 76 +-
website/.spelling | 2 -
7 files changed, 4 insertions(+), 287 deletions(-)
diff --git
a/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSource.java
b/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSource.java
index 36d6b97..21480fd 100644
--- a/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSource.java
+++ b/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSource.java
@@ -19,7 +19,6 @@
package org.apache.druid.data.input.impl;
-import com.fasterxml.jackson.annotation.JacksonInject;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.base.Preconditions;
@@ -29,7 +28,6 @@ import org.apache.druid.data.input.InputRowSchema;
import org.apache.druid.data.input.InputSourceReader;
import org.apache.druid.data.input.InputSplit;
import org.apache.druid.data.input.SplitHintSpec;
-import org.apache.druid.java.util.common.StringUtils;
import org.apache.druid.metadata.PasswordProvider;
import javax.annotation.Nullable;
@@ -48,25 +46,17 @@ public class HttpInputSource extends AbstractInputSource
implements SplittableIn
@Nullable
private final PasswordProvider httpAuthenticationPasswordProvider;
- private final HttpInputSourceConfig config;
-
@JsonCreator
public HttpInputSource(
@JsonProperty("uris") List uris,
@JsonProperty("httpAuthenticationUsername") @Nullable String
httpAuthenticationUsername,
- @JsonProperty("httpAuthenticationPassword") @Nullable PasswordProvider
httpAuthenticationPasswordProvider,
- @JacksonInject HttpInputSourceConfig config
+ @JsonProperty("httpAuthenticationPassword") @Nullable PasswordProvider
httpAuthenticationPasswordProvider
)
{
Preconditions.checkArgument(uris != null && !uris.isEmpty(), "Empty URIs");
-uris.forEach(uri -> Preconditions.checkArgument(
-config.isURIAllowed(uri),
-StringUtils.format("Access to [%s] DENIED!", uri)
-));
this.uris = uris;
this.httpAuthenticationUsername = httpAuthenticationUsername;
this.httpAuthenticationPasswordProvider =
httpAuthenticationPasswordProvider;
-this.config = config;
}
@JsonProperty
@@ -107,8 +97,7 @@ public class HttpInputSource extends AbstractInputSource
implements SplittableIn
return new HttpInputSource(
Collections.singletonList(split.get()),
httpAuthenticationUsername,
-httpAuthenticationPasswordProvider,
-config
+httpAuthenticationPasswordProvider
);
}
diff --git
a/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSourceConfig.java
b/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSourceConfig.java
deleted file mode 100644
index a84fb31..000
---
a/core/src/main/java/org/apache/druid/data/input/impl/HttpInputSourceConfig.java
+++ /dev/null
@@ -1,112 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.druid.data.input.impl;
-
-import com.fasterxml.jackson.annotation.JsonCreator;
-import
[GitHub] [druid] jihoonson merged pull request #10872: [Backport] Revert "Add support for Blacklisting some domains for HTTPInputSource (#10535)"
jihoonson merged pull request #10872: URL: https://github.com/apache/druid/pull/10872 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9518: AWS RDS token based password provider
jihoonson commented on a change in pull request #9518:
URL: https://github.com/apache/druid/pull/9518#discussion_r575571831
##
File path:
server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java
##
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.metadata;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.dbcp2.BasicDataSource;
+import org.apache.commons.dbcp2.ConnectionFactory;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.sql.Driver;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.Properties;
+
+/**
+ * This class exists so that {@link MetadataStorageConnectorConfig} is asked
for password every time a brand new
+ * connection is established with DB. {@link PasswordProvider} impls such as
those based on AWS tokens refresh the
+ * underlying token periodically since each token is valid for a certain
period of time only.
+ * So, This class overrides (and largely copies due to lack of extensibility),
the methods from base class in order to keep
+ * track of connection properties and call {@link
MetadataStorageConnectorConfig#getPassword()} everytime a new
+ * connection is setup.
+ */
+public class BasicDataSourceExt extends BasicDataSource
+{
+ private static final Logger LOGGER = new Logger(BasicDataSourceExt.class);
+
+ private Properties connectionProperties;
+ private final MetadataStorageConnectorConfig connectorConfig;
+
+ public BasicDataSourceExt(MetadataStorageConnectorConfig connectorConfig)
+ {
+this.connectorConfig = connectorConfig;
+this.connectionProperties = new Properties();
+ }
+
+ @Override
+ public void addConnectionProperty(String name, String value)
+ {
+connectionProperties.put(name, value);
+super.addConnectionProperty(name, value);
+ }
+
+ @Override
+ public void removeConnectionProperty(String name)
+ {
+connectionProperties.remove(name);
+super.removeConnectionProperty(name);
+ }
+
+ @Override
+ public void setConnectionProperties(String connectionProperties)
+ {
+if (connectionProperties == null) {
+ throw new NullPointerException("connectionProperties is null");
+}
+
+String[] entries = connectionProperties.split(";");
+Properties properties = new Properties();
+for (String entry : entries) {
+ if (entry.length() > 0) {
+int index = entry.indexOf('=');
+if (index > 0) {
+ String name = entry.substring(0, index);
+ String value = entry.substring(index + 1);
+ properties.setProperty(name, value);
+} else {
+ // no value is empty string which is how java.util.Properties works
+ properties.setProperty(entry, "");
+}
+ }
+}
+this.connectionProperties = properties;
+super.setConnectionProperties(connectionProperties);
+ }
+
+ @VisibleForTesting
+ public Properties getConnectionProperties()
+ {
+return connectionProperties;
+ }
+
+ @Override
+ protected ConnectionFactory createConnectionFactory() throws SQLException
Review comment:
@himanshug Did you adopt this source code from
`org.apache.commons.dbcp2.BasicDataSource`? If so, it should be added in the
licenses.yaml file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9518: AWS RDS token based password provider
jihoonson commented on a change in pull request #9518: URL: https://github.com/apache/druid/pull/9518#discussion_r575571188 ## File path: licenses.yaml ## @@ -147,6 +147,16 @@ source_paths: --- +name: AWS RDS SDK for Java +license_category: source +module: extensions/druid-aws-rds-extensions +license_name: Apache License version 2.0 +version: 1.11.199 +libraries: + - com.amazonaws: aws-java-sdk-rds Review comment: Hi @himanshug, did you adopt source code from the AWS SDK? If so, `source_paths` field should be added. Otherwise, `license_category` should be `binary`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #10830: Add configurations for allowed protocols for HTTP and HDFS inputSources/firehoses
jihoonson commented on a change in pull request #10830:
URL: https://github.com/apache/druid/pull/10830#discussion_r575509818
##
File path: docs/ingestion/native-batch.md
##
@@ -1064,7 +1064,7 @@ Sample specs:
"type": "index_parallel",
"inputSource": {
"type": "hdfs",
-"paths": "hdfs://foo/bar/", "hdfs://bar/foo"
+"paths": "hdfs:/foo/bar/", "hdfs:/bar/foo"
Review comment:
That sounds even better. I will update it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] medb commented on issue #7467: Historical node issue with segments being loaded when running locally.
medb commented on issue #7467: URL: https://github.com/apache/druid/issues/7467#issuecomment-778389384 @rae89 how did you fix this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (6541178 -> c8d2654)
This is an automated email from the ASF dual-hosted git repository. suneet pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 6541178 Support segmentGranularity for auto-compaction (#10843) add c8d2654 Use native git for git-commit-id-plugin to speed up build (#10881) No new revisions were added by this update. Summary of changes: pom.xml | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s merged pull request #10881: Use native git for git-commit-id-plugin to speed up build
suneet-s merged pull request #10881: URL: https://github.com/apache/druid/pull/10881 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] 2bethere commented on pull request #10881: Use native git for git-commit-id-plugin to speed up build
2bethere commented on pull request #10881: URL: https://github.com/apache/druid/pull/10881#issuecomment-778327887 I used https://github.com/intuit/maven-build-scanner Original blog: https://medium.com/@alex_collins/10x-faster-maven-builds-at-intuit-5b7bb60c65e6 The # on the pie chart is cumulative ms during the build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #10881: Use native git for git-commit-id-plugin to speed up build
suneet-s commented on pull request #10881: URL: https://github.com/apache/druid/pull/10881#issuecomment-778321526 @2bethere how did you generate those pie charts? And what do the measurements mean? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on pull request #10882: Fix example names of security artifacts in docs
suneet-s commented on pull request #10882: URL: https://github.com/apache/druid/pull/10882#issuecomment-778319677 @jihoonson should this be labeled with 0.21.0 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] a2l007 commented on a change in pull request #10830: Add configurations for allowed protocols for HTTP and HDFS inputSources/firehoses
a2l007 commented on a change in pull request #10830:
URL: https://github.com/apache/druid/pull/10830#discussion_r575262001
##
File path: docs/ingestion/native-batch.md
##
@@ -1064,7 +1064,7 @@ Sample specs:
"type": "index_parallel",
"inputSource": {
"type": "hdfs",
-"paths": "hdfs://foo/bar/", "hdfs://bar/foo"
+"paths": "hdfs:/foo/bar/", "hdfs:/bar/foo"
Review comment:
I think `hdfs:/foo/bar` may only work if the namenode is configured in
hadoop. For general cases, the URI format is `scheme://authority/path`
(https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#Overview)
Do you think it would be clearer to change this to
`"hdfs://namenodehost/foo/bar/"` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] Hemant0601 edited a comment on issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
Hemant0601 edited a comment on issue #9299: URL: https://github.com/apache/druid/issues/9299#issuecomment-778193005 Is there any solution for this ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] Hemant0601 commented on issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
Hemant0601 commented on issue #9299: URL: https://github.com/apache/druid/issues/9299#issuecomment-778193005 Is there any soluton for this ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated: Support segmentGranularity for auto-compaction (#10843)
This is an automated email from the ASF dual-hosted git repository. maytasm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid.git The following commit(s) were added to refs/heads/master by this push: new 6541178 Support segmentGranularity for auto-compaction (#10843) 6541178 is described below commit 6541178c21839530a42af4b4675a9bc680bffca6 Author: Maytas Monsereenusorn AuthorDate: Fri Feb 12 03:03:20 2021 -0800 Support segmentGranularity for auto-compaction (#10843) * Support segmentGranularity for auto-compaction * Support segmentGranularity for auto-compaction * Support segmentGranularity for auto-compaction * Support segmentGranularity for auto-compaction * resolve conflict * Support segmentGranularity for auto-compaction * Support segmentGranularity for auto-compaction * fix tests * fix more tests * fix checkstyle * add unit tests * fix checkstyle * fix checkstyle * fix checkstyle * add unit tests * add integration tests * fix checkstyle * fix checkstyle * fix failing tests * address comments * address comments * fix tests * fix tests * fix test * fix test * fix test * fix test * fix test * fix test * fix test * fix test --- .../NewestSegmentFirstPolicyBenchmark.java | 1 + .../org/apache/druid/timeline/CompactionState.java | 19 +- .../druid/timeline/partition/PartitionChunk.java | 2 +- .../org/apache/druid/timeline/DataSegmentTest.java | 4 +- .../common/task/AbstractBatchIndexTask.java| 6 +- .../druid/indexing/common/task/CompactionTask.java | 35 ++- .../druid/indexing/common/task/IndexTask.java | 3 +- .../parallel/ParallelIndexSupervisorTask.java | 3 +- .../task/ClientCompactionTaskQuerySerdeTest.java | 18 ++ .../common/task/CompactionTaskParallelRunTest.java | 84 +-- .../common/task/CompactionTaskRunTest.java | 81 +-- .../indexing/common/task/CompactionTaskTest.java | 41 .../coordinator/duty/ITAutoCompactionTest.java | 80 ++- .../druid/tests/indexer/ITCompactionTaskTest.java | 75 +-- ...edia_compaction_task_with_granularity_spec.json | 17 ++ ...a_compaction_task_with_segment_granularity.json | 15 ++ .../client/indexing/ClientCompactionTaskQuery.java | 14 +- .../ClientCompactionTaskQueryGranularitySpec.java | 95 .../client/indexing/HttpIndexingServiceClient.java | 4 +- .../client/indexing/IndexingServiceClient.java | 1 + .../granularity/ArbitraryGranularitySpec.java | 1 - .../indexing/granularity/BaseGranularitySpec.java | 22 +- .../indexing/granularity/GranularitySpec.java | 4 + .../granularity/UniformGranularitySpec.java| 5 - .../coordinator/DataSourceCompactionConfig.java| 34 +++ .../server/coordinator/duty/CompactSegments.java | 33 +++ .../duty/NewestSegmentFirstIterator.java | 133 +++- .../client/indexing/NoopIndexingServiceClient.java | 1 + .../granularity/ArbitraryGranularityTest.java | 27 +++ .../granularity/UniformGranularityTest.java| 33 ++- .../DataSourceCompactionConfigTest.java| 98 + .../coordinator/duty/CompactSegmentsTest.java | 241 - .../duty/NewestSegmentFirstIteratorTest.java | 10 + .../duty/NewestSegmentFirstPolicyTest.java | 216 -- .../druid/sql/calcite/schema/SystemSchemaTest.java | 3 +- 35 files changed, 1336 insertions(+), 123 deletions(-) diff --git a/benchmarks/src/test/java/org/apache/druid/server/coordinator/NewestSegmentFirstPolicyBenchmark.java b/benchmarks/src/test/java/org/apache/druid/server/coordinator/NewestSegmentFirstPolicyBenchmark.java index 91c1409..e744bf9 100644 --- a/benchmarks/src/test/java/org/apache/druid/server/coordinator/NewestSegmentFirstPolicyBenchmark.java +++ b/benchmarks/src/test/java/org/apache/druid/server/coordinator/NewestSegmentFirstPolicyBenchmark.java @@ -99,6 +99,7 @@ public class NewestSegmentFirstPolicyBenchmark null, null, null, + null, null ) ); diff --git a/core/src/main/java/org/apache/druid/timeline/CompactionState.java b/core/src/main/java/org/apache/druid/timeline/CompactionState.java index c30f427..8588717 100644 --- a/core/src/main/java/org/apache/druid/timeline/CompactionState.java +++ b/core/src/main/java/org/apache/druid/timeline/CompactionState.java @@ -43,15 +43,20 @@ public class CompactionState // org.apache.druid.segment.IndexSpec cannot be used here because it's in the 'processing' module which // has a dependency on the 'core' module where this class is. private final Map indexSpec; + //
[GitHub] [druid] maytasm merged pull request #10843: Support segmentGranularity for auto-compaction
maytasm merged pull request #10843: URL: https://github.com/apache/druid/pull/10843 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekagarwal87 closed pull request #10803: [Draft] travis test
abhishekagarwal87 closed pull request #10803: URL: https://github.com/apache/druid/pull/10803 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekagarwal87 closed pull request #10849: [WIP] Import jdk11 tests and checks from separate config
abhishekagarwal87 closed pull request #10849: URL: https://github.com/apache/druid/pull/10849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] abhishekagarwal87 opened a new pull request #10884: Reload segment usage when starting the process
abhishekagarwal87 opened a new pull request #10884: URL: https://github.com/apache/druid/pull/10884 ### Description When a historical process is restarted, we lose the accounting being kept by the `StorageLocation` class. This implies that `StorageLocation` could reserve space for a segment even if the cache is completely full. Since after restart, `StorageLocation` thinks that current usage is zero. This PR has: - [ ] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. # Key changed/added classes in this PR * `SegmentLoaderLocalCacheManager` * `StorageLocation` * `SegmentLoadDropHandlerCacheTest` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
