gortiz commented on PR #19021:
URL: https://github.com/apache/pinot/pull/19021#issuecomment-6824396781
I measured the cost of creating and closing a buffer with this factory
against the Unsafe one, because closing a shared arena makes the JVM coordinate
with all its threads. Netty 4.2 hits the same cost with its per-buffer
`Arena.ofShared()` cleaner on JDK 25.
The probe creates a buffer (`allocateDirect` or `mapFile`), reads one byte
and closes it, in a loop for 2 s after 1 s of warmup. Next to it, 4 threads
spin and N threads sleep, to simulate a server with many threads. "Workers" is
the number of threads that create and close buffers at the same time. Numbers
are µs per create+close on JDK 25.0.3 (macOS, Apple M4 Pro, 14 cores). JDK 27
gives the same picture; its worst case drops from 980 µs to 768 µs.
| Op, size | Threads | Workers | Unsafe | Foreign |
|---|---|---|---|---|
| alloc 1 MiB | none | 1 | 8.8 | 23.6 |
| alloc 1 MiB | 4 busy + 1000 parked | 1 | 9.4 | 202 |
| alloc 1 MiB | 4 busy + 1000 parked | 4 | 17.5 | 980 |
| alloc 64 MiB | 4 busy + 1000 parked | 4 | 8 547 | 11 252 |
| alloc 2.5 GiB | none / 4 busy + 1000 parked | 1 | ~200–250 ms | ~250 ms |
| map 1 MiB | 4 busy + 1000 parked | 4 | 34 | 941 |
| map 2.5 GiB | none | 1 | 760 | 45 |
| map 2.5 GiB | 4 busy + 1000 parked | 1 | 770 | 265 |
What I take from it:
- The close cost grows with the number of threads in the JVM, and concurrent
closes serialize. With 1000 threads, a small buffer pays about 200 µs alone and
about 1 ms when 4 threads close buffers at the same time.
- For large buffers the zero-fill dominates, so the arena cost disappears in
the noise. At 2.5 GiB both factories take about the same time to allocate.
- `mapFile` of a 2.5 GiB file is faster with this factory than with the
Unsafe one, even with 1000 threads.
- With the default `SmallWithFallbackPinotBufferFactory`, only buffers
larger than 2 GiB reach this factory. Those buffers live long, so this PR is
neutral or better on that path. Using arenas for every buffer size would be a
different decision: code that creates and closes small buffers often would pay
the close cost each time. In that case we would need a pool, or malloc/free
through the `Linker` plus `reinterpret` to the global scope (Unsafe speed, but
no use-after-free protection).
<details><summary>Probe source (run with the Pinot JVM flags, one JVM per
configuration)</summary>
```java
import java.io.File;
import java.io.RandomAccessFile;
import java.nio.ByteOrder;
import org.apache.pinot.segment.memory.foreign.ForeignPinotBufferFactory;
import org.apache.pinot.segment.spi.memory.PinotBufferFactory;
import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
import org.apache.pinot.segment.spi.memory.unsafe.UnsafePinotBufferFactory;
/// Args: factory(unsafe|foreign) op(alloc|map) busy parked sizeKiB
[threadsDoingOp]
/// Measures create+close of one PinotDataBuffer while `busy` threads spin
and `parked` threads sleep.
public class ArenaBench {
static volatile long sink;
public static void main(String[] a) throws Exception {
String fac = a[0];
String op = a[1];
int busy = Integer.parseInt(a[2]);
int parked = Integer.parseInt(a[3]);
long size = Long.parseLong(a[4]) << 10;
int workers = a.length > 5 ? Integer.parseInt(a[5]) : 1;
PinotBufferFactory f = fac.equals("unsafe") ? new
UnsafePinotBufferFactory() : new ForeignPinotBufferFactory();
File file = null;
if (op.equals("map")) {
file = File.createTempFile("arenabench", ".bin");
file.deleteOnExit();
try (RandomAccessFile raf = new RandomAccessFile(file, "rw")) {
raf.setLength(size);
}
}
for (int i = 0; i < busy + parked; i++) {
final boolean b = i < busy;
Thread t = new Thread(() -> {
if (b) {
long x = 0;
while (true) {
x += System.nanoTime();
if ((x & 0xFFFFF) == 0) {
sink = x;
}
}
} else {
try {
Thread.sleep(Long.MAX_VALUE);
} catch (InterruptedException e) {
}
}
});
t.setDaemon(true);
t.start();
}
final File ff = file;
Runnable once = () -> {
try {
PinotDataBuffer buf = op.equals("alloc") ? f.allocateDirect(size,
ByteOrder.nativeOrder())
: f.mapFile(ff, true, 0, size, ByteOrder.nativeOrder());
sink += buf.getByte(size - 1);
buf.close();
} catch (Exception e) {
throw new RuntimeException(e);
}
};
long[] counts = new long[workers];
Thread[] ws = new Thread[workers];
long warmEnd = System.nanoTime() + 1_000_000_000L;
long measEnd = warmEnd + 2_000_000_000L;
for (int w = 0; w < workers; w++) {
final int idx = w;
ws[w] = new Thread(() -> {
while (System.nanoTime() < warmEnd) {
once.run();
}
long n = 0;
while (System.nanoTime() < measEnd) {
once.run();
n++;
}
counts[idx] = n;
});
ws[w].start();
}
for (Thread t : ws) {
t.join();
}
long n = 0;
for (long c : counts) {
n += c;
}
// per-op latency seen by each worker = 2 s * workers / total ops
System.out.printf("%-7s %-5s busy=%-2d parked=%-4d workers=%-2d
size=%8dKiB : %9.1f us/op (n=%d)%n", fac, op, busy,
parked, workers, size >> 10, 2e6 * workers / n, n);
System.exit(0);
}
}
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]