> I see, it only affects the TTL. We can acknowledge the messages to
clean up the data. Then we have to document the `brokerInterceptors`
config clearly to say TTL might not work if
`AppendBrokerTimestampMetadataInterceptor` is not configured.
Yes, and LinLin figured out a better solution to mitig
I see, it only affects the TTL. We can acknowledge the messages to
clean up the data. Then we have to document the `brokerInterceptors`
config clearly to say TTL might not work if
`AppendBrokerTimestampMetadataInterceptor` is not configured.
Thanks,
Yunze
On Thu, Jan 18, 2024 at 10:41 AM PengHui
> I don't think we should provide such an option for users. Compared
with the serious security issue, the extra overhead (13 bytes) per
entry (not per message) should never be a concern.
It actually changed the stored data format. As far as I know, there are
some users
who read data from the offlo
> We should also improve the TTL:
> When client publish time > Ledger create time + Ledger rollover time,
> and `brokerPublishTime` is not set,
> we can make Ledger TTL time = Ledger create time + Ledger rollover time.
> This change can make sure entry expires when client clock is not right.
Thi
>From my perspective, it's a serious security issue. The client side
could easily make Pulsar's message expiry mechanism not work. Even if
it's not a malicious change made by users, it would be hard to figure
out what makes the message not expired.
> Users can still have a way to disable it if the
Only enabled `AppendBrokerTimestampMetadataInterceptor` is not enough.
We should also improve the TTL:
When client publish time > Ledger create time + Ledger rollover time,
and `brokerPublishTime` is not set,
we can make Ledger TTL time = Ledger create time + Ledger rollover time.
This change c
> User disabling `AppendBrokerTimestampMetadataInterceptor` does not mean
that they allow this bug.
This is a configuration, not an API. It is difficult to use documentation
to regulate user behavior.
Actually, they need to know why they want to disable `
AppendBrokerTimestampMetadataInterceptor`.
User disabling `AppendBrokerTimestampMetadataInterceptor` does not mean that
they allow this bug.
This is a configuration, not an API. It is difficult to use documentation to
regulate user behavior.
Maybe we can add a new field (BrokerTimestamp) to save the timestamp on the
Broker side.
The ti
IMO, we should enable `AppendBrokerTimestampMetadataInterceptor` by default.
Users can still have a way to disable it if they care about the additional
metadata stored
in each entry.
For the `hasBrokerPublishTime` API. The topic might also have historical
data without
broker publish time. So, it s
Now, if the message's metadata does not set a broker side timestamp, the
ledger expiration check is based on the client's publish time.
When the client machine's clock is incorrect (eg: set to 1 year later) ,
the ledger can not be cleaned up. Issue
https://github.com/apache/pulsar/issues/21347
`A
10 matches
Mail list logo
