Hello again!
Next problem I am facing is configuring Minio S3 with Accumulo. I am referring
to this document:
https://accumulo.apache.org/blog/2019/09/10/accumulo-S3-notes.html
I have already invoked the command “accumulo init” with and without the option
“–upload-accumulo-props” and using accumulo.properties as below:
instance.volumes=
hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo
instance.zookeeper.host=accumulo-zookeeper
general.volume.chooser=org.apache.accumulo.core.spi.fs.PreferredVolumeChooser
general.custom.volume.preferred.logger=hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo
general.custom.volume.preferred.default=
hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo
Next, when I run the command “accumulo init –add-volumes” with
accumulo.properties is as below:
instance.volumes=s3a://minio-s3/accumulo,hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo
instance.zookeeper.host=accumulo-zookeeper
general.volume.chooser=org.apache.accumulo.core.spi.fs.PreferredVolumeChooser
general.custom.volume.preferred.logger=hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo
general.custom.volume.preferred.default=s3a://minio-s3/accumulo
I see error as below:
ERROR StatusLogger An exception occurred processing Appender MonitorLog
java.lang.RuntimeException: Can't tell if Accumulo is initialized; can't read
instance id at s3a://minio-s3/accumulo/instance_id
at
org.apache.accumulo.server.fs.VolumeManager.getInstanceIDFromHdfs(VolumeManager.java:229)
at
org.apache.accumulo.server.ServerInfo.<init>(ServerInfo.java:102)
at
org.apache.accumulo.server.ServerContext.<init>(ServerContext.java:106)
at
org.apache.accumulo.monitor.util.logging.AccumuloMonitorAppender.lambda$new$1(AccumuloMonitorAppender.java:93)
at
org.apache.accumulo.monitor.util.logging.AccumuloMonitorAppender.append(AccumuloMonitorAppender.java:111)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:674)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:643)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:612)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:98)
at
org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:488)
at
org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:156)
at
org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:51)
at
org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.file.AccessDeniedException:
s3a://minio-s3/accumulo/instance_id: listStatus on
s3a://minio-s3/accumulo/instance_id:
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you
provided does not exist in our records. (Service: Amazon S3; Status Code: 403;
Error Code: InvalidAccessKeyId; Request ID: 71HC1ZM3D43W0H67; S3 Extended
Request ID:
OsRVgg057cm+M7EP+P069hY97mA6na8rkhnNVunVRTUmttCDc5Sm5aKqodS+oogU5/UupgsEy1A=;
Proxy: null), S3 Extended Request ID:
OsRVgg057cm+M7EP+P069hY97mA6na8rkhnNVunVRTUmttCDc5Sm5aKqodS+oogU5/UupgsEy1A=:InvalidAccessKeyId
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:255)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:119)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262)
at
org.apache.accumulo.server.fs.VolumeManager.getInstanceIDFromHdfs(VolumeManager.java:211)
... 23 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access
Key Id you provided does not exist in our records. (Service: Amazon S3; Status
Code: 403; Error Code: InvalidAccessKeyId; Request ID: 71HC1ZM3D43W0H67; S3
Extended Request ID:
OsRVgg057cm+M7EP+P069hY97mA6na8rkhnNVunVRTUmttCDc5Sm5aKqodS+oogU5/UupgsEy1A=;
Proxy: null), S3 Extended Request ID:
OsRVgg057cm+M7EP+P069hY97mA6na8rkhnNVunVRTUmttCDc5Sm5aKqodS+oogU5/UupgsEy1A=
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397)
at
com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:971)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$11(S3AFileSystem.java:2595)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at
org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
at
org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2586)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2153)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
When I invoke commands from HDFS, I see no problems though:
* hdfs dfs -fs s3a://minio-s3 -ls /
2023-01-20 16:38:51,319 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser]: Sanitizing
XML document destined for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
2023-01-20 16:38:51,321 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager]:
Connection [id: 0][route: {}->http://accumulo-minio:9000] can be kept alive
for 60.0 seconds
2023-01-20 16:38:51,321 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.thirdparty.apache.http.impl.conn.DefaultManagedHttpClientConnection]:
http-outgoing-0: set socket timeout to 0
2023-01-20 16:38:51,321 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager]:
Connection released: [id: 0][route: {}->http://accumulo-minio:9000][total
available: 1; route allocated: 1 of 128; total allocated: 1 of 128]
2023-01-20 16:38:51,321 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser]: Parsing XML
response document with handler: class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
2023-01-20 16:38:51,328 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser]: Examining
listing for bucket: minio-s3
2023-01-20 16:38:51,329 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.request]: Received successful response: 200, AWS Request ID:
173C11CC6FEF29A0
2023-01-20 16:38:51,329 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.requestId]: x-amzn-RequestId: not available
2023-01-20 16:38:51,329 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.requestId]: AWS Request ID: 173C11CC6FEF29A0
2023-01-20 16:38:51,338 DEBUG [s3a-transfer-minio-s3-unbounded-pool2-t1]
[com.amazonaws.latency]: ServiceName=[Amazon S3], StatusCode=[200],
ServiceEndpoint=[http://accumulo-minio:9000],
RequestType=[ListObjectsV2Request], AWSRequestID=[173C11CC6FEF29A0],
HttpClientPoolPendingCount=0, RetryCapacityConsumed=0,
HttpClientPoolAvailableCount=0, RequestCount=1, HttpClientPoolLeasedCount=0,
ResponseProcessingTime=[71.198], ClientExecuteTime=[297.496],
HttpClientSendRequestTime=[7.255], HttpRequestTime=[119.87],
ApiCallLatency=[279.779], RequestSigningTime=[56.006],
CredentialsRequestTime=[5.091, 0.015], HttpClientReceiveResponseTime=[12.849]
