[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650723821 @asdf2014 It's done. Thank you for your instruction. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650705280 Hi @jihoonson , CI excludes all other deep storage IT testcases through following property. I didn't found where this property is defined in project. Is it configured in CI ? The test group of aliyun OSS extension is `aliyun-oss-deep-storage`, which needs to be excluded from CI too.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650678797 @jihoonson OK, I'll fix the doc. And for the IT test case, it seems like `BUCKET` and `PATH` are not configured. In my environment, I put these two parameters and other OSS related configuration parameters in a config file and use `-Ddruid.test.config.configFile` to reference that file. I don't know how to configure the IT test cases in CI environment to use OSS. Do IT test cases need real s3 or google cloud storage or Azure or OSS service ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650667153 @jon-wei I agree with you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650663610 Hi @jihoonson , using s3 extension with OSS, the exception is as below I think OSS was compatible at the time the doc was created, but as the s3 SDK evolves, OSS doesn't keep the pace with it. 2020-06-28T00:14:03,975 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task. java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021.oss-cn-hangzhou-internal.aliyuncs.com at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:825) ~[druid-indexing-service-0.18.1.jar:0.18.1] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:277) [druid-indexing-service-0.18.1.jar:0.18.1] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:164) [druid-indexing-service-0.18.1.jar:0.18.1] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:421) [druid-indexing-service-0.18.1.jar:0.18.1] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:393) [druid-indexing-service-0.18.1.jar:0.18.1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252] Caused by: java.lang.RuntimeException: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:822) ~[druid-server-0.18.1.jar:0.18.1] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:657) ~[druid-server-0.18.1.jar:0.18.1] at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?] ... 3 more Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: A header you provided implies functionality that is not implemented. (Service: Amazon S3; Status Code: 400; Error Code: NotImplemented; Request ID: 5EF7E0CBAF285D3538BCFD5C; S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com), S3 Extended Request ID: frankchen021-public.oss-cn-hangzhou-internal.aliyuncs.com at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:103) ~[?:?] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:791) ~[druid-server-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.18.1.jar:0.18.1] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:787) ~[druid-server-0.18.1.jar:0.18.1] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:657) ~[druid-server-0.18.1.jar:0.18.1] at com.google.common.util.concurr
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-650562183 Hi @jihoonson , s3 extension does not work with aliyun OSS although OSS provides compatibility with s3. I tested to utilize OSS with s3 extension in the past, but the `putObject` returned HTTP 400 error, saying there were some unsupported HEADERS in the request generated by s3 SDK. I didn't explore that problem very deeply, I guess the latest s3 SDK adds some new HEADERS which are not compatible in OSS. the APIs between OSS and s3 are very similar, even their names. but I don't think it's a good idea to use one extension to support both OSS and s3, or use some abstraction method to extract the same code between s3 and OSS. Actually, this kind of abstraction has been done by deep storage framework, so every implementation interacts with its underlying storage API. As s3 and OSS are provided by two companies, it's better to provide two extensions respectively so that they can evolve independently. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-649602917 Hi @jon-wei , I've updated the code, and re-tested all core functions. IT testcases runs OK, and real time ingestion task, compact task, kill task runs well in our test cluster. Please check it again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-647895605 Hi @jon-wei @fjy Thank you for your opinions. I'm busy these days, and I will fix the code this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-642413836 > > CI reports insufficient unit test line/function/branch coverage. Is it mandatory to meet that requirement ? > > Given the nature of the extension, I feel like adding an integration test that connects to Aliyun would be more useful than full unit test coverage, and I would consider that enough in terms of test coverage. > > We have some existing tests for S3/GCS/Azure at that you could use as a reference: > ITS3ToS3ParallelIndexTest > ITAzureToAzureParallelIndexTest > ITGcsToGcsParallelIndexTest > > I'll keep you posted with more comments as I review, thanks! I will add some IT test cases This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] FrankChen021 commented on pull request #9898: support Aliyun OSS service as deep storage
FrankChen021 commented on pull request #9898: URL: https://github.com/apache/druid/pull/9898#issuecomment-634001052 @clintropolis CI reports insufficient unit test line/function/branch coverage. Is it mandatory to meet that requirement ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
