[
https://issues.apache.org/jira/browse/HDDS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-8543:
-----------------------------------
Description:
I have a bucket with erasure coding enabled
{code}
$ ozone sh bucket info impala/test-warehouse
{
"metadata" : { },
"volumeName" : "impala",
"name" : "test-warehouse",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 16966237353,
"usedNamespace" : 28550,
"creationTime" : "2023-04-11T16:59:38.827Z",
"modificationTime" : "2023-04-11T17:22:22.328Z",
"encryptionKeyName" : "testkey",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
"owner" : "michael",
"replicationConfig" : {
"data" : 3,
"parity" : 2,
"ecChunkSize" : 1048576,
"codec" : "RS",
"replicationType" : "EC",
"requiredNodes" : 5
},
"link" : false
}
{code}
I’m going to create a key at foo/bar in that bucket. It doesn’t exist yet.
{code}
$ ozone sh key info impala/test-warehouse/foo
KEY_NOT_FOUND Key:foo not found
{code}
I create the key in that bucket
{code}
$ ozone sh key cp impala/test-warehouse
alltypes_avro_snap/year=2009/month=1/000000_0 foo/bar
23/04/11 11:13:41 WARN
erasurecode.ErasureCodeNative: ISA-L support is not available in your
platform... using builtin-java codec where applicable
23/04/11 11:13:41 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
23/04/11 11:13:41 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
23/04/11 11:13:41 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics
system started
{code}
The key reports EC replication
{code}
$ ozone sh key info impala/test-warehouse/foo/bar
{
"volumeName" : "impala",
"bucketName" : "test-warehouse",
"name" : "foo/bar",
"dataSize" : 6937,
"creationTime" : "2023-04-11T18:13:41.081Z",
"modificationTime" : "2023-04-11T18:13:41.660Z",
"replicationConfig" : {
"data" : 3,
"parity" : 2,
"ecChunkSize" : 1048576,
"codec" : "RS",
"replicationType" : "EC",
"requiredNodes" : 5
},
...
}
{code}
However the prefix reports RATIS
{code}
$ ozone sh key info impala/test-warehouse/foo
{
"volumeName" : "impala",
"bucketName" : "test-warehouse",
"name" : "foo/",
"dataSize" : 0,
"creationTime" : "2023-04-11T18:13:41.081Z",
"modificationTime" : "2023-04-11T18:13:41.081Z",
"replicationConfig" : {
"replicationFactor" : "ONE",
"requiredNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"ozoneKeyLocations" : [ ]
}
{code}
It used to report EC for the prefix (maybe with LEGACY layout), not RATIS.
was:
I have a bucket with erasure coding enabled
{code}
$ ozone sh bucket info impala/test-warehouse
{
"metadata" : { },
"volumeName" : "impala",
"name" : "test-warehouse",
"storageType" : "DISK",
"versioning" : false,
"usedBytes" : 16966237353,
"usedNamespace" : 28550,
"creationTime" : "2023-04-11T16:59:38.827Z",
"modificationTime" : "2023-04-11T17:22:22.328Z",
"encryptionKeyName" : "testkey",
"quotaInBytes" : -1,
"quotaInNamespace" : -1,
"bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
"owner" : "michael",
"replicationConfig" : {
"data" : 3,
"parity" : 2,
"ecChunkSize" : 1048576,
"codec" : "RS",
"replicationType" : "EC",
"requiredNodes" : 5
},
"link" : false
}
{code}
I’m going to create a key at foo/bar in that bucket. It doesn’t exist yet.
{code}
$ ozone sh key info impala/test-warehouse/foo
KEY_NOT_FOUND Key:foo not found
{code}
I create the key in that bucket
{code}
$ ozone sh key cp impala/test-warehouse
alltypes_avro_snap/year=2009/month=1/000000_0 foo/bar
23/04/11 11:13:41 WARN
erasurecode.ErasureCodeNative: ISA-L support is not available in your
platform... using builtin-java codec where applicable
23/04/11 11:13:41 WARN impl.MetricsConfig: Cannot locate configuration: tried
hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
23/04/11 11:13:41 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period
at 10 second(s).
23/04/11 11:13:41 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics
system started
{code}
The key reports EC replication
{code}
$ ozone sh key info impala/test-warehouse/foo/bar
{
"volumeName" : "impala",
"bucketName" : "test-warehouse",
"name" : "foo/bar",
"dataSize" : 6937,
"creationTime" : "2023-04-11T18:13:41.081Z",
"modificationTime" : "2023-04-11T18:13:41.660Z",
"replicationConfig" : {
"data" : 3,
"parity" : 2,
"ecChunkSize" : 1048576,
"codec" : "RS",
"replicationType" : "EC",
"requiredNodes" : 5
},
"metadata" : { },
"ozoneKeyLocations" : [ {
"containerID" : 5,
"localID" : 111677748019216002,
"length" : 6937,
"offset" : 0,
"keyOffset" : 0
} ],
"fileEncryptionInfo" : {
"cipherSuite" : "AES_CTR_NOPADDING",
"iv" : "7y6R8DxVj9q0+1yWbl4ZRw==",
"keyName" : "testkey",
"ezKeyVersionName" : "testkey@0",
"encryptedDataEncryptionKey" : "yn/4CXq5dP4Dnwif/oxf0w==",
"cryptoProtocolVersion" : "ENCRYPTION_ZONES"
}
}
{code}
However the prefix reports RATIS
{code}
$ ozone sh key info impala/test-warehouse/foo
{
"volumeName" : "impala",
"bucketName" : "test-warehouse",
"name" : "foo/",
"dataSize" : 0,
"creationTime" : "2023-04-11T18:13:41.081Z",
"modificationTime" : "2023-04-11T18:13:41.081Z",
"replicationConfig" : {
"replicationFactor" : "ONE",
"requiredNodes" : 1,
"replicationType" : "RATIS"
},
"metadata" : { },
"ozoneKeyLocations" : [ ]
}
{code}
It used to report EC for the prefix (maybe with LEGACY layout), not RATIS.
> New key prefix in EC bucket with FSO reports RATIS replication
> --------------------------------------------------------------
>
> Key: HDDS-8543
> URL: https://issues.apache.org/jira/browse/HDDS-8543
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC
> Affects Versions: 1.3.0
> Reporter: Michael Smith
> Assignee: Attila Doroszlai
> Priority: Major
>
> I have a bucket with erasure coding enabled
> {code}
> $ ozone sh bucket info impala/test-warehouse
>
> {
> "metadata" : { },
> "volumeName" : "impala",
> "name" : "test-warehouse",
> "storageType" : "DISK",
> "versioning" : false,
> "usedBytes" : 16966237353,
> "usedNamespace" : 28550,
> "creationTime" : "2023-04-11T16:59:38.827Z",
> "modificationTime" : "2023-04-11T17:22:22.328Z",
> "encryptionKeyName" : "testkey",
> "quotaInBytes" : -1,
> "quotaInNamespace" : -1,
> "bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
> "owner" : "michael",
> "replicationConfig" : {
> "data" : 3,
> "parity" : 2,
> "ecChunkSize" : 1048576,
> "codec" : "RS",
> "replicationType" : "EC",
> "requiredNodes" : 5
> },
> "link" : false
> }
> {code}
> I’m going to create a key at foo/bar in that bucket. It doesn’t exist yet.
> {code}
> $ ozone sh key info impala/test-warehouse/foo
> KEY_NOT_FOUND Key:foo not found
> {code}
> I create the key in that bucket
> {code}
> $ ozone sh key cp impala/test-warehouse
> alltypes_avro_snap/year=2009/month=1/000000_0 foo/bar
> 23/04/11 11:13:41 WARN
> erasurecode.ErasureCodeNative: ISA-L support is not available in your
> platform... using builtin-java codec where applicable
> 23/04/11 11:13:41 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
> 23/04/11 11:13:41 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 23/04/11 11:13:41 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics
> system started
> {code}
> The key reports EC replication
> {code}
> $ ozone sh key info impala/test-warehouse/foo/bar
> {
> "volumeName" : "impala",
> "bucketName" : "test-warehouse",
> "name" : "foo/bar",
> "dataSize" : 6937,
> "creationTime" : "2023-04-11T18:13:41.081Z",
> "modificationTime" : "2023-04-11T18:13:41.660Z",
> "replicationConfig" : {
> "data" : 3,
> "parity" : 2,
> "ecChunkSize" : 1048576,
> "codec" : "RS",
> "replicationType" : "EC",
> "requiredNodes" : 5
> },
> ...
> }
> {code}
> However the prefix reports RATIS
> {code}
> $ ozone sh key info impala/test-warehouse/foo
> {
> "volumeName" : "impala",
> "bucketName" : "test-warehouse",
> "name" : "foo/",
> "dataSize" : 0,
> "creationTime" : "2023-04-11T18:13:41.081Z",
> "modificationTime" : "2023-04-11T18:13:41.081Z",
> "replicationConfig" : {
> "replicationFactor" : "ONE",
> "requiredNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "ozoneKeyLocations" : [ ]
> }
> {code}
> It used to report EC for the prefix (maybe with LEGACY layout), not RATIS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
