vakaobr commented on issue #2588:
URL: https://github.com/apache/age/issues/2588#issuecomment-5947361256
### Additional diagnostics
Gathered proactively rather than waiting for a request. Same instance, same
workload, 48-hour window with 24 occurrences.
#### Build and configuration
```
PostgreSQL 17.10 (Debian 17.10-1.pgdg12+1) on x86_64-pc-linux-gnu,
compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
age 1.7.0
shared_preload_libraries = age
```
Settings that seemed most relevant:
```
jit on
jit_above_cost 100000
jit_inline_above_cost 500000
jit_optimize_above_cost 500000
shared_buffers 128MB
work_mem 4MB
maintenance_work_mem 64MB
max_connections 100
max_locks_per_transaction 64
max_parallel_workers_per_gather 2
dynamic_shared_memory_type posix
huge_pages try
```
**JIT is enabled.** We have not yet tested with `jit = off`, but given the
symptom is a pointer that stops referring to the label, it seemed worth
surfacing early in case it narrows things for you. We can run that experiment
and report back.
#### Graph scale at time of capture
```
vertices 531,452
edges 1,492,885
labels 4 (_ag_label_vertex, _ag_label_edge, base, DIRECTED)
```
#### Edge property sizes
The faulting operation inlines properties into `CREATE`, so payload size is
a plausible factor. Distribution over all edges, as `length(properties::text)`:
```
avg 350
p50 289
p99 1,332
max 73,715
```
So most payloads are small but the tail is three orders of magnitude larger.
#### The bad names, categorised
24 occurrences over 48 hours, classified by what the invalid relation name
appears to contain. Values are described rather than quoted, since the real
strings contain customer data, which is itself the point: the memory being read
is the payload.
```
8 text fragment with non-ASCII or raw bytes
5 chunk / task identifier fragment
4 property VALUE including our field separator
3 agtype PROPERTY KEYS concatenated, in declaration order
3 plain text fragment (entity name or filename)
1 single control byte equal to a label id
```
Length of the invalid name:
```
n=24 min=0 p50=54 max=327
```
The `min=0` case is an empty name. The single-control-byte case was `\x01`,
which equals the `id` of `_ag_label_vertex` in this graph.
#### Not one bad backend
Occurrences are spread across backends rather than concentrated in a
poisoned session:
```
distinct backends that hit it ~10
distinct backends in the window 397
most occurrences in one backend 2
```
So a session does not appear to "go bad" and stay bad. Each occurrence looks
independent.
#### No STATEMENT is logged with the error
`log_min_error_statement = error`, yet these errors appear in the log with
no accompanying `STATEMENT:` line, unlike other errors from the same instance
in the same window. We do not know whether that is meaningful to you, but it
was unexpected and may indicate the error is raised in a context where the
statement is not attached.
The faulting operation is known from the client side regardless: it is
always the edge upsert shown in the original report, never a read.
#### What we have ruled out since filing
Nothing has changed our earlier conclusions. In particular we eliminated
roughly 31,000 failed `create_graph()` calls per hour from our client (it was
calling it on every pool checkout) and the failure rate did not move, measured
per unit of work:
```
3.3 failures per 100 ingested documents (day of the change)
5.0 failures per 100 ingested documents (following day)
```
Happy to capture anything else that would help. A stack trace with debug
symbols is feasible for us if you can suggest where to break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
