Il Dom 30 Lug 2023, 09:19 horizonzy <horizo...@apache.org> ha scritto:
> When reading data from an EntryLog file, we create a BufferedReadChannel.
> In the size() method of BufferedReadChannel, FileChannelImpl.size() is
> called, which is a time-consuming operation.
>
> If the EntryLog file has been archived, it is unnecessary to call
> FileChannelImpl.size() every time we read from it. We only need to call it
> once, cache the result, and reuse it.
>
> This improvement can benefit two cases.
> 1. DefaultEntryLogger#scanEntryLog, it can help speed up compaction.
> 2. BufferedReadChannel#read, It can speed up reading entries from the
> EntryLog file.
>
I agree with this proposal.
I have left one little comment in the PR
Thanks
Enrico
>
> Performance Test.
> The entry log f file holds 1GB data.
> -rw-r--r--@ 1 horizon staff 1.0G 7 26 17:47 f.log
>
> 1. Test DefaultEntryLogger.scanEntryLog (10x performance improvement)
> ```
> public static void main(String[] args) throws IOException {
> ServerConfiguration serverConfiguration = new
> ServerConfiguration();
> serverConfiguration.setLedgerDirNames(new
> String[]{"/Users/horizon/Downloads/bk1/bk-data"});
> DefaultEntryLogger defaultEntryLogger = new
> DefaultEntryLogger(serverConfiguration);
> long l = System.currentTimeMillis();
> AtomicInteger entryCount = new AtomicInteger();
> defaultEntryLogger.scanEntryLog(15, new EntryLogScanner() {
> @Override
> public boolean accept(long ledgerId) {
> return true;
> }
>
> @Override
> public void process(long ledgerId, long offset, ByteBuf entry)
> throws IOException {
> entryCount.incrementAndGet();
> }
> });
> long spend = System.currentTimeMillis() - l;
> System.out.println("Scan entry end, entry count: " +
> entryCount.get());
> System.out.println("Spend: " + spend);
> }
> ```
>
> Before:
> Scan entry end, entry count: 17886769
> Spend: 37273
>
>
> After:
> Scan entry end, entry count: 17886769
> Spend: 3363
>
> 2. Test DefaultEntryLogger.readFromLogChannel (30x performance improvement)
> ```
> public static void main(String[] args) throws IOException {
> ServerConfiguration serverConfiguration = new
> ServerConfiguration();
> serverConfiguration.setLedgerDirNames(new
> String[]{"/Users/horizon/Downloads/bk1/bk-data"});
> DefaultEntryLogger defaultEntryLogger = new
> DefaultEntryLogger(serverConfiguration);
> long l = System.currentTimeMillis();
> BufferedReadChannel channel =
> defaultEntryLogger.getChannelForLogId(15);
> int pos = 0;
> //Read total 1gb data from the BufferedReadChannel.
> long gb = 1024 * 1024 * 1024;
> ByteBuf byteBuf = ByteBufAllocator.DEFAULT.heapBuffer(10);
> while (pos < gb) {
> defaultEntryLogger.readFromLogChannel(15, channel, byteBuf, 0);
> pos += byteBuf.readableBytes();
> byteBuf.clear();
> }
> long spend = System.currentTimeMillis() - l;
> System.out.println("Spend: " + spend);
> }
> ```
> Before:
> Spend: 69810
>
> After:
> Spend: 2338
>
