GitHub user Yuukadesu created a discussion: Add rateLimiter and memoryManager
to do memory control and rate limit.
### Purpose
Add two functions separately, RareLimiter to control speed and MemoryManager to
control memory size, ultimately preventing excessive CPU usage and memory
overload.
### Programs
**SpRateLimiter**
Write a limiter using the speed limiting algorithm encapsulated in Guava, which
provides configurable parameters, configurable initialization parameters, and
core limit methods
```
private static final double DEFAULT_PERMITS_PER_SECOND = 100.0;
private static final long DEFAULT_WARMUP_PERIOD = 1000L;
private static final TimeUnit DEFAULT_TIME_UNIT = TimeUnit.MILLISECONDS;
void limit()
```
**SpMemoryManager**
Set the memory size (such as 10GB) during initialization, check the memory in a
loop during allocation, and if it is insufficient, block and wait for memory
allocation; After the event allocated to memory is released, the memory is
released.
```
SpMemoryManager() {
this.freeMemory = 10L * 1024 * 1024 * 1024; // Initialize with 10 GB of
free memory
log.info("SpMemoryManager initialized with {} bytes of free memory",
freeMemory);
}
void allocate()
void free()
```
**Tracking point**
Finally, embed points in the process and onEvent methods of classes such as
StandaloneEventSinkRuntime to enable StreamPipes to perform memory control and
rate limiting.
```
SpRateLimiter.INSTANCE.limit();
SpMemoryManager.INSTANCE.allocate(event.length);
SpMemoryManager.INSTANCE.free(event.length);
```
GitHub link: https://github.com/apache/streampipes/discussions/3749
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
