Re: [PR] Lift the storage limit for tag and attribute management [iotdb]
JackieTien97 merged PR #12447: URL: https://github.com/apache/iotdb/pull/12447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Lift the storage limit for tag and attribute management [iotdb]
JackieTien97 commented on code in PR #12447:
URL: https://github.com/apache/iotdb/pull/12447#discussion_r1584515815
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagManager.java:
##
@@ -212,20 +209,18 @@ public void removeIndex(String tagKey, String tagValue,
IMeasurementMNode mea
// init memory size
int memorySize = 0;
Review Comment:
make it as long
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagManager.java:
##
@@ -181,18 +180,16 @@ public void addIndex(String tagKey, String tagValue,
IMeasurementMNode measur
int memorySize = 0;
Review Comment:
make it as long
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagLogFile.java:
##
@@ -129,14 +129,12 @@ public static ByteBuffer parseByteBuffer(FileChannel
fileChannel, long position)
ByteBuffer byteBuffers = ByteBuffer.allocate(blockNum * MAX_LENGTH);
byteBuffers.put(byteBuffer);
byteBuffers.position(4); // Skip blockNum
-List blockOffset = new ArrayList<>();
-blockOffset.add(position);
for (int i = 1; i < blockNum; i++) {
- blockOffset.add(ReadWriteIOUtils.readLong(byteBuffers));
+ Long nextPosition = ReadWriteIOUtils.readLong(byteBuffers);
Review Comment:
```suggestion
long nextPosition = ReadWriteIOUtils.readLong(byteBuffers);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Lift the storage limit for tag and attribute management [iotdb]
JackieTien97 commented on code in PR #12447:
URL: https://github.com/apache/iotdb/pull/12447#discussion_r1584008968
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagLogFile.java:
##
@@ -95,22 +102,85 @@ public synchronized void copyTo(File targetFile) throws
IOException {
* @return tags map, attributes map
* @throws IOException error occurred when reading disk
*/
- public Pair, Map> read(int size, long
position)
- throws IOException {
+ public Pair, Map> read(long position)
throws IOException {
if (position < 0) {
return new Pair<>(Collections.emptyMap(), Collections.emptyMap());
}
-ByteBuffer byteBuffer = ByteBuffer.allocate(size);
+ByteBuffer byteBuffer = parseByteBuffer(fileChannel, position);
+return new Pair<>(ReadWriteIOUtils.readMap(byteBuffer),
ReadWriteIOUtils.readMap(byteBuffer));
+ }
+
+ public Map readTag(long position) throws IOException {
+ByteBuffer byteBuffer = parseByteBuffer(fileChannel, position);
+return ReadWriteIOUtils.readMap(byteBuffer);
+ }
+
+ public static ByteBuffer parseByteBuffer(FileChannel fileChannel, long
position)
+ throws IOException {
+// Read the first block
+ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_LENGTH);
fileChannel.read(byteBuffer, position);
byteBuffer.flip();
-return new Pair<>(ReadWriteIOUtils.readMap(byteBuffer),
ReadWriteIOUtils.readMap(byteBuffer));
+if (byteBuffer.limit() > 0) { // This indicates that there is data at this
position
+ int firstInt = ReadWriteIOUtils.readInt(byteBuffer); // first int
+ byteBuffer.position(0);
+ if (firstInt < -1) { // This position is blockNum, the original data
occupies multiple blocks
+int blockNum = -firstInt;
+ByteBuffer byteBuffers = ByteBuffer.allocate(blockNum * MAX_LENGTH);
+byteBuffers.put(byteBuffer);
+byteBuffers.position(4); // Skip blockNum
+List blockOffset = new ArrayList<>();
+blockOffset.add(position);
+for (int i = 1; i < blockNum; i++) {
+ blockOffset.add(ReadWriteIOUtils.readLong(byteBuffers));
+ // read one offset, then use filechannel's read to read it
+ byteBuffers.position(MAX_LENGTH * i);
+ byteBuffers.limit(MAX_LENGTH * (i + 1));
+ fileChannel.read(byteBuffers, blockOffset.get(i));
+ byteBuffers.position(4 + i * Long.BYTES);
+}
+byteBuffers.limit(byteBuffers.capacity());
+return byteBuffers;
+ }
+}
+return byteBuffer;
}
- public Map readTag(int size, long position) throws
IOException {
-ByteBuffer byteBuffer = ByteBuffer.allocate(size);
+ private List ParseOffsetList(long position) throws IOException {
+List blockOffset = new ArrayList<>();
+blockOffset.add(position);
+// Read the first block
+ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_LENGTH);
fileChannel.read(byteBuffer, position);
byteBuffer.flip();
-return ReadWriteIOUtils.readMap(byteBuffer);
+if (byteBuffer.limit() > 0) { // This indicates that there is data at this
position
+ int firstInt = ReadWriteIOUtils.readInt(byteBuffer); // first int
+ byteBuffer.position(0);
+ if (firstInt < -1) { // This position is blockNum, the original data
occupies multiple blocks
+int blockNum = -firstInt;
+int blockOffsetStoreLen =
+(((blockNum - 1) * Long.BYTES + 4) / MAX_LENGTH + 1)
+* MAX_LENGTH; // blockOffset storage length
+ByteBuffer blockBuffer = ByteBuffer.allocate(blockOffsetStoreLen);
+blockBuffer.put(byteBuffer);
+blockBuffer.position(4); // Skip blockNum
+
+for (int i = 1; i < blockNum; i++) {
+ blockOffset.add(ReadWriteIOUtils.readLong(blockBuffer));
+ // Every time you read an offset, use filechannel's read to read it
+ if (MAX_LENGTH * (i + 1)
+ <= blockOffsetStoreLen) { // Compared with directly reading
bytebuffer, some reading
+// operations are reduced, only the content of offset is
+// read
+blockBuffer.position(MAX_LENGTH * i);
+blockBuffer.limit(MAX_LENGTH * (i + 1));
+fileChannel.read(blockBuffer, blockOffset.get(i));
+blockBuffer.position(4 + i * Long.BYTES);
+ }
+}
Review Comment:
You've already read all the first block content into memory in previous
`fileChannel.read(byteBuffer, position);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Lift the storage limit for tag and attribute management [iotdb]
JackieTien97 commented on code in PR #12447:
URL: https://github.com/apache/iotdb/pull/12447#discussion_r1584008968
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagLogFile.java:
##
@@ -95,22 +102,85 @@ public synchronized void copyTo(File targetFile) throws
IOException {
* @return tags map, attributes map
* @throws IOException error occurred when reading disk
*/
- public Pair, Map> read(int size, long
position)
- throws IOException {
+ public Pair, Map> read(long position)
throws IOException {
if (position < 0) {
return new Pair<>(Collections.emptyMap(), Collections.emptyMap());
}
-ByteBuffer byteBuffer = ByteBuffer.allocate(size);
+ByteBuffer byteBuffer = parseByteBuffer(fileChannel, position);
+return new Pair<>(ReadWriteIOUtils.readMap(byteBuffer),
ReadWriteIOUtils.readMap(byteBuffer));
+ }
+
+ public Map readTag(long position) throws IOException {
+ByteBuffer byteBuffer = parseByteBuffer(fileChannel, position);
+return ReadWriteIOUtils.readMap(byteBuffer);
+ }
+
+ public static ByteBuffer parseByteBuffer(FileChannel fileChannel, long
position)
+ throws IOException {
+// Read the first block
+ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_LENGTH);
fileChannel.read(byteBuffer, position);
byteBuffer.flip();
-return new Pair<>(ReadWriteIOUtils.readMap(byteBuffer),
ReadWriteIOUtils.readMap(byteBuffer));
+if (byteBuffer.limit() > 0) { // This indicates that there is data at this
position
+ int firstInt = ReadWriteIOUtils.readInt(byteBuffer); // first int
+ byteBuffer.position(0);
+ if (firstInt < -1) { // This position is blockNum, the original data
occupies multiple blocks
+int blockNum = -firstInt;
+ByteBuffer byteBuffers = ByteBuffer.allocate(blockNum * MAX_LENGTH);
+byteBuffers.put(byteBuffer);
+byteBuffers.position(4); // Skip blockNum
+List blockOffset = new ArrayList<>();
+blockOffset.add(position);
+for (int i = 1; i < blockNum; i++) {
+ blockOffset.add(ReadWriteIOUtils.readLong(byteBuffers));
+ // read one offset, then use filechannel's read to read it
+ byteBuffers.position(MAX_LENGTH * i);
+ byteBuffers.limit(MAX_LENGTH * (i + 1));
+ fileChannel.read(byteBuffers, blockOffset.get(i));
+ byteBuffers.position(4 + i * Long.BYTES);
+}
+byteBuffers.limit(byteBuffers.capacity());
+return byteBuffers;
+ }
+}
+return byteBuffer;
}
- public Map readTag(int size, long position) throws
IOException {
-ByteBuffer byteBuffer = ByteBuffer.allocate(size);
+ private List ParseOffsetList(long position) throws IOException {
+List blockOffset = new ArrayList<>();
+blockOffset.add(position);
+// Read the first block
+ByteBuffer byteBuffer = ByteBuffer.allocate(MAX_LENGTH);
fileChannel.read(byteBuffer, position);
byteBuffer.flip();
-return ReadWriteIOUtils.readMap(byteBuffer);
+if (byteBuffer.limit() > 0) { // This indicates that there is data at this
position
+ int firstInt = ReadWriteIOUtils.readInt(byteBuffer); // first int
+ byteBuffer.position(0);
+ if (firstInt < -1) { // This position is blockNum, the original data
occupies multiple blocks
+int blockNum = -firstInt;
+int blockOffsetStoreLen =
+(((blockNum - 1) * Long.BYTES + 4) / MAX_LENGTH + 1)
+* MAX_LENGTH; // blockOffset storage length
+ByteBuffer blockBuffer = ByteBuffer.allocate(blockOffsetStoreLen);
+blockBuffer.put(byteBuffer);
+blockBuffer.position(4); // Skip blockNum
+
+for (int i = 1; i < blockNum; i++) {
+ blockOffset.add(ReadWriteIOUtils.readLong(blockBuffer));
+ // Every time you read an offset, use filechannel's read to read it
+ if (MAX_LENGTH * (i + 1)
+ <= blockOffsetStoreLen) { // Compared with directly reading
bytebuffer, some reading
+// operations are reduced, only the content of offset is
+// read
+blockBuffer.position(MAX_LENGTH * i);
+blockBuffer.limit(MAX_LENGTH * (i + 1));
+fileChannel.read(blockBuffer, blockOffset.get(i));
+blockBuffer.position(4 + i * Long.BYTES);
+ }
+}
Review Comment:
You've already read all the first block content into memory in previous
`fileChannel.read(byteBuffer, position);`
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/tag/TagLogFile.java:
##
@@ -95,22 +102,85 @@ public synchronized void copyTo(File targetFile) throws
IOException {
* @return tags map, attributes map
* @throws IOException error occurred when reading disk
*/
- public Pair, Map> read(int size, long
position)
- throws IOException {
+ public Pair, Map> read(long position)
throws
[PR] Lift the storage limit for tag and attribute management [iotdb]
linxt20 opened a new pull request, #12447: URL: https://github.com/apache/iotdb/pull/12447 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
