[GitHub] [druid] viongpanzi opened a new pull request #9722: optimize FileWriteOutBytes to avoid high sys cpu
viongpanzi opened a new pull request #9722: optimize FileWriteOutBytes to avoid high sys cpu URL: https://github.com/apache/druid/pull/9722 Fixes #9721. ### Description see #9721 The benchmark report shows that the performance is improved by about 44% with this pr. Machine info: CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz Disk: HDD Command: java -Djava.io.tmpdir=/data00/tmp_dir -jar benchmarks.jar ``` IndexMergeWithTmpFileBenchmark before optimization: Benchmark (numSegments) (rollup) (rowsPerSegment) (schema) Mode Cnt Score Error Units IndexMergeWithTmpFileBenchmark.mergeV9 5 true 75000 basic avgt 25 8161891.327 ± 32636.767 us/op IndexMergeWithTmpFileBenchmark.mergeV9 5 false 75000 basic avgt 25 8041137.131 ± 41477.861 us/op after optimization: Benchmark (numSegments) (rollup) (rowsPerSegment) (schema) Mode Cnt Score Error Units IndexMergeWithTmpFileBenchmark.mergeV9 5 true 75000 basic avgt 25 4536098.486 ± 13668.764 us/op IndexMergeWithTmpFileBenchmark.mergeV9 5 false 75000 basic avgt 25 4321243.165 ± 30293.772 us/op ``` This PR has: - [] been self-reviewed. - [] been tested in a test Druid cluster. # Key changed/added classes in this PR * `FileWriteOutBytes` * `IndexMergeWithTmpFileBenchmark` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] viongpanzi opened a new issue #9721: Frequent calls to FileWriteOutBytes.size() will result in high sys CPU
viongpanzi opened a new issue #9721: Frequent calls to FileWriteOutBytes.size()
will result in high sys CPU
URL: https://github.com/apache/druid/issues/9721
### Affected Version
0.13.0+
### Description
In order to compare the disk performance between local disk and cloud disk,
we replace the ```OffHeapMemorySegmentWriteOutMediumFactory``` with
```TmpFileSegmentWriteOutMediumFactory``` when instancing **INDEX_MERGER_V9**
in IndexMergeBenchmark(rename to IndexMergeWithTmpFileBenchmark). However,
during benchmark running, we found that the sys cpu is too high:
With the help of flame graph, we found that every time we call the size()
method will trigger flush() method which call write system call.
To avoid calling flush method, we introduce a new variable
```writeOutBytes``` to record the number of bytes written:
```
final class FileWriteOutBytes extends WriteOutBytes
{
...
private long writeOutBytes;
...
FileWriteOutBytes(File file, FileChannel ch)
{
this.file = file;
this.ch = ch;
this.writeOutBytes = 0L;
}
...
@Override
public void write(int b) throws IOException
{
flushIfNeeded(1);
buffer.put((byte) b);
writeOutBytes++;
}
@Override
public void writeInt(int v) throws IOException
{
flushIfNeeded(Integer.BYTES);
buffer.putInt(v);
writeOutBytes += Integer.BYTES;
}
@Override
public int write(ByteBuffer src) throws IOException
{
...
buffer.put(src);
writeOutBytes += len;
return len;
}
@Override
public long size() throws IOException
{
return writeOutBytes;
}
```
Then we rerun the benchmark and sys cpu returned to normal. The benchmark
report shows that the performance is improved by about 44%.
* Machine info:
CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
Disk: HDD
Command: java -Djava.io.tmpdir=/data00/tmp_dir -jar benchmarks.jar
IndexMergeWithTmpFileBenchmark
before optimization:
Benchmark (numSegments) (rollup)
(rowsPerSegment) (schema) Mode CntScore Error Units
IndexMergeWithTmpFileBenchmark.mergeV9 5 true
75000 basic avgt 25 8161891.327 ± 32636.767 us/op
IndexMergeWithTmpFileBenchmark.mergeV9 5 false
75000 basic avgt 25 8041137.131 ± 41477.861 us/op
after optimization:
Benchmark (numSegments) (rollup)
(rowsPerSegment) (schema) Mode CntScore Error Units
IndexMergeWithTmpFileBenchmark.mergeV9 5 true
75000 basic avgt 25 4536098.486 ± 13668.764 us/op
IndexMergeWithTmpFileBenchmark.mergeV9 5 false
75000 basic avgt 25 4321243.165 ± 30293.772 us/op
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #9231: enable running indexer in docker container
stale[bot] commented on issue #9231: enable running indexer in docker container URL: https://github.com/apache/druid/pull/9231#issuecomment-615561531 This pull request/issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] closed pull request #9231: enable running indexer in docker container
stale[bot] closed pull request #9231: enable running indexer in docker container URL: https://github.com/apache/druid/pull/9231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #8127: [WIP]growable sketch aggregator support
stale[bot] commented on issue #8127: [WIP]growable sketch aggregator support URL: https://github.com/apache/druid/pull/8127#issuecomment-615561517 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] fjy merged pull request #125: Adding Fanatics Company usage
fjy merged pull request #125: Adding Fanatics Company usage URL: https://github.com/apache/druid-website-src/pull/125 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] 01/01: Merge pull request #120 from apache/update-powered-by
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git commit 5b57ba6318c48eaf609d1d0654d20fe7ab4c9f0d Merge: a9623d6 73a6b50 Author: Fangjin Yang AuthorDate: Fri Apr 17 16:46:38 2020 -0700 Merge pull request #120 from apache/update-powered-by update wikimedia foundation powered by druid-powered.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated (a9623d6 -> 5b57ba6)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git. from a9623d6 Merge pull request #125 from im2kul/patch-1 add b108e29 update wikimedia foundation powered by add 73a6b50 add blank line new 5b57ba6 Merge pull request #120 from apache/update-powered-by The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: druid-powered.md | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] fjy merged pull request #120: update wikimedia foundation powered by
fjy merged pull request #120: update wikimedia foundation powered by URL: https://github.com/apache/druid-website-src/pull/120 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated: Adding Fanatics Company usage
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new 2f80b15 Adding Fanatics Company usage new a9623d6 Merge pull request #125 from im2kul/patch-1 2f80b15 is described below commit 2f80b15f6c3b48c7914a0190d90ae870b9407b00 Author: im2kul AuthorDate: Fri Apr 17 16:34:47 2020 -0700 Adding Fanatics Company usage --- druid-powered.md | 4 1 file changed, 4 insertions(+) diff --git a/druid-powered.md b/druid-powered.md index eec24d8..8833ca2 100644 --- a/druid-powered.md +++ b/druid-powered.md @@ -138,6 +138,10 @@ eBay uses Druid to aggregate multiple data streams for real-time user behavior a One of our key goals at [FacilityConneX](https://www.facilityconnex.com/) is to offer real-time insights that help our customers optimize their equipment or processes, reduce cost, or prevent incidents, to ultimately improve our customers business. This *real-time* requirement has always been a major technical challenge to scale within our SaaS environment. FacilityConneX has looked to **Druid** to help solve many of these challenging performance and growth issues. +## Fanatics + +Druid powers our applications related metrics and helps us slice and dice analytics on both historical and realtime-time metrics. It significantly reduces latency of analytic queries and help people to get insights more interactively. Through druid data we can now do anomaly detection as well + ## FullContact * [Enrich API Brings Higher Match Rates with Multi-Field Enrichment Capabilities](https://www.fullcontact.com/blog/enrich-api-brings-higher-match-rates-with-multi-field-enrichment-capabilities/) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] im2kul opened a new pull request #125: Adding Fanatics Company usage
im2kul opened a new pull request #125: Adding Fanatics Company usage URL: https://github.com/apache/druid-website-src/pull/125 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9698: fixes for inline subqueries when multi-value dimension is present
jihoonson commented on a change in pull request #9698: fixes for inline
subqueries when multi-value dimension is present
URL: https://github.com/apache/druid/pull/9698#discussion_r410507374
##
File path:
processing/src/main/java/org/apache/druid/query/topn/TopNQueryEngine.java
##
@@ -132,7 +132,7 @@ private TopNMapFn getMapFn(
topNAlgorithm = new TimeExtractionTopNAlgorithm(adapter, query);
} else if (selector.isHasExtractionFn()) {
topNAlgorithm = new HeapBasedTopNAlgorithm(adapter, query);
-} else if (columnCapabilities != null && !(columnCapabilities.getType() ==
ValueType.STRING
+} else if (columnCapabilities == null || !(columnCapabilities.getType() ==
ValueType.STRING
Review comment:
nit: maybe nice to comment when `columnCapabilities` can be null?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vaibhav-imply opened a new issue #9720: java.lang.ClassNotFoundException: org.apache.druid.security.basic.BasicSecuritySSLSocketFactory
vaibhav-imply opened a new issue #9720: java.lang.ClassNotFoundException: org.apache.druid.security.basic.BasicSecuritySSLSocketFactory URL: https://github.com/apache/druid/issues/9720 When a druid is configured to authenticate against SSL enabled LDAP server Coordinator service throws `ClassNotFoundException` **Exception trace from coordinator log:** ``` Caused by: java.lang.ClassNotFoundException: org.apache.druid.security.basic.BasicSecuritySSLSocketFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_242] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) ~[?:1.8.0_242] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_242] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ~[?:1.8.0_242] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_242] at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_242] at com.sun.jndi.ldap.VersionHelper12.loadClass(VersionHelper12.java:72) ~[?:1.8.0_242] at com.sun.jndi.ldap.Connection.createSocket(Connection.java:291) ~[?:1.8.0_242] at com.sun.jndi.ldap.Connection.(Connection.java:213) ~[?:1.8.0_242] ... 46 more ``` **Steps to reproduce this issue:** 1) Configure druid to use SSL enabled LDAP authentication 2) Start the master services. You can observe the exception in the coordinator log. The issue can be quickly reproduced in micor start mode as well. **Few Interesting facts:** 1) Druid basic-security-extension is loaded and Verified that only one version of the druid-basic-security extension jar present into the druid security extension directory. 2)The coordinator log shows that the druid-basic-security extension jar being loaded. 3)Verified the druid-basic security extension jar and it has the BasicSecuritySSLSocketFactory.class 4) To extra confirm if BasicSecuritySSLSocketFactory class is loaded by coordinator JVM or not, Enabled verbose class loading and this confirms the BasicSecuritySSLSocketFactory class being loaded by the class loader, but still, coordinator throws `ClassNotFoundException` `[Loaded org.apache.druid.security.basic.BasicSecuritySSLSocketFactory from file:/druid-basic-security/druid-basic-security-0.17.1.jar:0.17.1]` **Workaround:** – I placed druid-security-extension jar into the druid lib directory and restarted the server and this resolves the ClassNotFoundException. It seems some jar dependency issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on issue #8987: Adding support for autoscaling in GCE
frnidito commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-615162819 @clintropolis — checks passed :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] qidaijie2018 opened a new issue #9719: SQL query with Like can't add anything else
qidaijie2018 opened a new issue #9719: SQL query with Like can't add anything else URL: https://github.com/apache/druid/issues/9719 Druid Version:0.16 SQL-1:SELECT "TABLE_NAME" FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME NOT LIKE '%err%' . This is OK! SQL-2:SELECT "TABLE_NAME" FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'TABLE' AND TABLE_NAME NOT LIKE '%err%'. This return "Query returned no data" Why SQL only have a like condition is OK. Can't add anything else? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r410080579
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
Review comment:
That seems reasonable :+1:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on a change in pull request #8987: Adding support for autoscaling in GCE
frnidito commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r410074690
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
Review comment:
My take on this is that the mig / non-mig should be managed through the
module itself and its configuration instead of having a different modules for
each flavour of management.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on a change in pull request #8987: Adding support for autoscaling in GCE
frnidito commented on a change in pull request #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#discussion_r410072948 ## File path: extensions-contrib/gce-extensions/pom.xml ## @@ -0,0 +1,126 @@ + + +http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"; xmlns="http://maven.apache.org/POM/4.0.0"; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> + +org.apache.druid +druid +0.18.0-SNAPSHOT Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r410069896
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
Review comment:
On further thought I think just leaving it as `gce` is fine, feel free to
ignore this comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#discussion_r410062740 ## File path: extensions-contrib/gce-extensions/pom.xml ## @@ -0,0 +1,126 @@ + + +http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"; xmlns="http://maven.apache.org/POM/4.0.0"; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> + +org.apache.druid +druid +0.18.0-SNAPSHOT Review comment: This needs to be `0.19.0-SNAPSHOT` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r410065652
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
Review comment:
One last thought, not really suggesting a change just thinking ahead, but do
you imagine there could ever be any other types of Google compute based
autoscalers that _do not_ use managed instance groups? I wonder if we should
name the type for this `gce-mig` or something like that to leave space for
variation. On the other hand, i guess the only other raw GCE based autoscaler
implementation I can imagine would work like the AWS autoscaler which just
deals directly with raw instances (i built one of these in the past as a
private extension), so maybe it isn't worth distinguishing. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #8987: Adding support for autoscaling in GCE
clintropolis commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-615108423 > ping? :) Ah, very sorry i've been swamped, but am going to have one last look and finish up review right now. Thanks for being persistent :+1: > do you believe that this change may make it into the next release of druid? Unfortunately not, we only backport bug fixes after the branch has been made, my bad I couldn't finish up review and get this in before the cut 😞. I'll make sure we get it in to the next release though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito edited a comment on issue #8987: Adding support for autoscaling in GCE
frnidito edited a comment on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-615085566 ping? :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on issue #8987: Adding support for autoscaling in GCE
frnidito commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-615085566 ping? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
