Gzure commented on PR #3290: URL: https://github.com/apache/brpc/pull/3290#issuecomment-4965173714
> ```shell > ./ubring_performance_client -use_ubring=true -echo_attachment=true -attachment_size=6291456 > I0712 23:42:12.017195 3264863 0 /workspace/cgm/brpc/src/brpc/server.cpp:1243 StartInternal] Server[DummyServerOf(./ubring_performance_client)] is serving on port=8001. > I0712 23:42:12.017340 3264863 0 /workspace/cgm/brpc/src/brpc/server.cpp:1246 StartInternal] Check out http://k8s-al-sh-gpu-rdma-h20-0032:8001 in web browser. > [Threads: 1, Depth: 1, Attachment: 6291456B, UBRING: yes, Echo: yes] > I0712 23:42:12.026167 3264863 0 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_mgr.cpp:72 ShmMgrInit] shm mgr init success, shm type=1 > E0712 23:42:12.026774 3264871 4294968065 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_ipc.cpp:39 ReserveIpcShm] IPC reserve shm=UBRING_127.0.0.1:62484_C length=4194304 failed, ret(28). > E0712 23:42:12.026797 3264871 4294968065 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_mgr.cpp:114 ShmLocalCalloc] Failed to alloc local shm. > E0712 23:42:12.026802 3264871 4294968065 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:926 ApplyAndMapLocalShm] Trx apply local shared memory failed, local shm name=UBRING_127.0.0.1:62484_C, rc=-100 > E0712 23:42:12.026807 3264871 4294968065 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:840 UbrAllocateLocalShm] Trx apply or map local shared memory failed, localName=127.0.0.1:62484 > W0712 23:42:12.026816 3264871 4294968065 /workspace/cgm/brpc/src/brpc/ubshm/ub_endpoint.cpp:348 ProcessHandshakeAtClient] Fallback to tcp:brpc::Socket{id=1 fd=8 addr=0.0.0.0:8002:62484} (0x5587f03b4f40) > Avg-Latency: 2486, 90th-Latency: 2630, 99th-Latency: 3961, 99.9th-Latency: 4696, Throughput: 2344.77MB/s, QPS: 0k, Server CPU-utilization: 191%, Client CPU-utilization: 179% > E0712 23:42:32.030796 3264873 489626272583 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:1035 UbrTrxCloseCheck] Trx close failed, client trx is null. > [Threads: 2, Depth: 1, Attachment: 6291456B, UBRING: yes, Echo: yes] > E0712 23:42:32.035919 3264871 493921240022 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_ipc.cpp:39 ReserveIpcShm] IPC reserve shm=UBRING_127.0.0.1:52664_C length=4194304 failed, ret(28). > E0712 23:42:32.035941 3264871 493921240022 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_mgr.cpp:114 ShmLocalCalloc] Failed to alloc local shm. > E0712 23:42:32.035947 3264871 493921240022 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:926 ApplyAndMapLocalShm] Trx apply local shared memory failed, local shm name=UBRING_127.0.0.1:52664_C, rc=-100 > E0712 23:42:32.035951 3264871 493921240022 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:840 UbrAllocateLocalShm] Trx apply or map local shared memory failed, localName=127.0.0.1:52664 > W0712 23:42:32.035958 3264871 493921240022 /workspace/cgm/brpc/src/brpc/ubshm/ub_endpoint.cpp:348 ProcessHandshakeAtClient] Fallback to tcp:brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52664} (0x5587f03b55c0) > W0712 23:42:32.036008 3264863 0 /workspace/cgm/brpc/example/ubring_performance/client.cpp:131 Init] RPC call failed, retrying... (3 left): [E11]Fail to connect brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52664} (0x5587f03b55c0): Resource temporarily unavailable > E0712 23:42:32.136118 3264873 4294970112 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:1035 UbrTrxCloseCheck] Trx close failed, client trx is null. > I0712 23:42:32.136156 3264873 4294970112 /workspace/cgm/brpc/src/brpc/socket.cpp:2570 CheckHealth] Checking Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) > I0712 23:42:32.136296 3264871 4294970112 /workspace/cgm/brpc/src/brpc/socket.cpp:2630 AfterRevived] Revived Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) (Connectable) > E0712 23:42:33.036319 3264871 528280980045 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_ipc.cpp:39 ReserveIpcShm] IPC reserve shm=UBRING_127.0.0.1:52674_C length=4194304 failed, ret(28). > E0712 23:42:33.036349 3264871 528280980045 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_mgr.cpp:114 ShmLocalCalloc] Failed to alloc local shm. > E0712 23:42:33.036353 3264871 528280980045 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:926 ApplyAndMapLocalShm] Trx apply local shared memory failed, local shm name=UBRING_127.0.0.1:52674_C, rc=-100 > E0712 23:42:33.036356 3264871 528280980045 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:840 UbrAllocateLocalShm] Trx apply or map local shared memory failed, localName=127.0.0.1:52674 > W0712 23:42:33.036364 3264871 528280980045 /workspace/cgm/brpc/src/brpc/ubshm/ub_endpoint.cpp:348 ProcessHandshakeAtClient] Fallback to tcp:brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52674} (0x5587f03b55c0) > W0712 23:42:33.036402 3264863 0 /workspace/cgm/brpc/example/ubring_performance/client.cpp:131 Init] RPC call failed, retrying... (2 left): [E115]Fail to connect brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52674} (0x5587f03b55c0): Operation now in progress > E0712 23:42:33.236540 3264873 4294970113 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:1035 UbrTrxCloseCheck] Trx close failed, client trx is null. > I0712 23:42:33.236592 3264873 4294970113 /workspace/cgm/brpc/src/brpc/socket.cpp:2570 CheckHealth] Checking Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) > I0712 23:42:33.236752 3264871 4294970113 /workspace/cgm/brpc/src/brpc/socket.cpp:2630 AfterRevived] Revived Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) (Connectable) > E0712 23:42:34.036694 3264871 519691043831 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_ipc.cpp:39 ReserveIpcShm] IPC reserve shm=UBRING_127.0.0.1:52688_C length=4194304 failed, ret(28). > E0712 23:42:34.036725 3264871 519691043831 /workspace/cgm/brpc/src/brpc/ubshm/shm/shm_mgr.cpp:114 ShmLocalCalloc] Failed to alloc local shm. > E0712 23:42:34.036730 3264871 519691043831 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:926 ApplyAndMapLocalShm] Trx apply local shared memory failed, local shm name=UBRING_127.0.0.1:52688_C, rc=-100 > E0712 23:42:34.036733 3264871 519691043831 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:840 UbrAllocateLocalShm] Trx apply or map local shared memory failed, localName=127.0.0.1:52688 > W0712 23:42:34.036741 3264871 519691043831 /workspace/cgm/brpc/src/brpc/ubshm/ub_endpoint.cpp:348 ProcessHandshakeAtClient] Fallback to tcp:brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52688} (0x5587f03b55c0) > W0712 23:42:34.036777 3264863 0 /workspace/cgm/brpc/example/ubring_performance/client.cpp:131 Init] RPC call failed, retrying... (1 left): [E115]Fail to connect brpc::Socket{id=3 fd=8 addr=0.0.0.0:8002:52688} (0x5587f03b55c0): Operation now in progress > E0712 23:42:34.436931 3264873 4294970114 /workspace/cgm/brpc/src/brpc/ubshm/ub_ring.cpp:1035 UbrTrxCloseCheck] Trx close failed, client trx is null. > I0712 23:42:34.436981 3264873 4294970114 /workspace/cgm/brpc/src/brpc/socket.cpp:2570 CheckHealth] Checking Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) > I0712 23:42:34.437151 3264871 4294970114 /workspace/cgm/brpc/src/brpc/socket.cpp:2630 AfterRevived] Revived Socket{id=3 addr=0.0.0.0:8002} (0x5587f03b55c0) (Connectable) > E0712 23:42:35.036864 3264863 0 /workspace/cgm/brpc/example/ubring_performance/client.cpp:135 Init] RPC call failed after multiple retries > ``` > > I failed to run the ubring example with the latest code; see the log above for details. @zchuango @chenBright Hello chenBright，The code reports ret(28) = errno 28 = ENOSPC (No space left on device). The ReserveIpcShm function internally calls posix_fallocate(fd, 0, shm->len) to reserve 4 MB space for the shm object. However, the remaining space of /dev/shm (tmpfs) is less than 4 MB, and posix_fallocate returns ENOSPC. This is directly related to the previous shm leak. In the previous test, a large number of UBRING_*_C files were left. These residual files occupied all the space of /dev/shm, causing the newly connected posix_fallocate to fail due to insufficient space. Please run the following commands on the machine where the error is reported (k8s-al-sh-gpu-rdma-h20-0032): df -h /dev/shm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
