[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-459101276 I moved the ec2 autoscaler into an extension on its own. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-459101170 https://travis-ci.org/apache/incubator-druid/jobs/486550791 Injection bites a big "no can do" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458795692 @gianm I just realized, this makes it where injecting some of the dependencies is nearly impossible. I'll think some more on it when I get a chance and see if a good solution comes up This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458791905 haha, its a bunch more work that it sounds. A lot of things in the IndexingService are AWS based This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458763917 "Dependency of dependency" is a problem in a few areas now. I can't think of a better solution for the time being, though. At least with a "standard" module that is pulled into extensions can allow us to manage dependencies a bit better, even if the libs are duplicated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458763966 I'll update my PR to see how that looks This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458658214 The *most* useful override I foresee is `HttpRequestInitializer`, where a company may want to have their own custom way of fetching GCP service account credentials, and need a module to fetch such things. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core functionality URL: https://github.com/apache/incubator-druid/issues/6910#issuecomment-458657058 I do have experience with those things. You have to include `com.google.cloud.bigdataoss:gcs-connector` as another default hadoop artifact in addition to the hadoop client you are using for dataproc. Be aware that recent versions of `com.google.cloud.bigdataoss:gcs-connector` have terrible guava dependency problems. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core
functionality
URL:
https://github.com/apache/incubator-druid/issues/6910#issuecomment-457410832
For a module which uses KMS, it can be injected by something like the
following:
```java
class CloudKmsProvider implements Provider
{
private final HttpRequestInitializer requestInitializer;
private final HttpTransport httpTransport;
private final JsonFactory jsonFactory;
@Inject
public CloudKmsProvider(
final HttpRequestInitializer requestInitializer,
final HttpTransport httpTransport,
final JsonFactory jsonFactory
)
{
this.requestInitializer = requestInitializer;
this.httpTransport = httpTransport;
this.jsonFactory = jsonFactory;
}
@Override
public CloudKMS get()
{
return new CloudKMS(httpTransport, jsonFactory, requestInitializer);
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] drcrallen commented on issue #6910: Bring GCP functionality to core functionality
drcrallen commented on issue #6910: Bring GCP functionality to core
functionality
URL:
https://github.com/apache/incubator-druid/issues/6910#issuecomment-457410561
Sure, the pain points are that I have extensions that do things like
password providers using KMS, and things like the watermark extension. These
use GCP clients which require a lot of duplicates of libraries somewhere in
the class loaders. This makes Hadoop style indexing have problems because all
of the libs get jumbled together when the hadoop jobs are run.
Here is a rough example of the module I'm hoping can be in stock druid:
```java
public class GcpModule implements DruidModule
{
@Override
public List getJacksonModules()
{
return ImmutableList.of();
}
@Override
public void configure(Binder binder)
{
// NOOP
}
@Provides
@LazySingleton
public HttpRequestInitializer getHttpRequestInitializer(
HttpTransport transport,
JsonFactory factory
)
{
try {
return GoogleCredential
.getApplicationDefault(transport, factory)
.createScoped(Collections.singleton("https://www.googleapis.com/auth/cloud-platform";));
}
catch (IOException e) {
throw new RuntimeException("Unable to build authentication", e);
}
}
@Provides
@LazySingleton
public HttpTransport getHttpTransport()
{
return new NetHttpTransport.Builder()
.build();
}
@Provides
@LazySingleton
public JsonFactory getJsonFactory()
{
return JacksonFactory.getDefaultInstance();
}
}
```
This would allow the GCP api client, which has transitive guava and jackson
dependencies, to be controlled at the master pom level rather than having their
lib versions buried down in the extension. That makes guava and jackson version
conflicts easier to manage, and allows extensions in house to not have to load
the same jars multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
