Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.
>You should start a new bookie with the new configuration. If the DirectEntryLogger needs to set up a separate bookie service, it would be hard for users to switch from the DefaultEntryLogger to DriectEntryLogger. >It is not generally possible to guarantee compatibility between different implementations. The DirectEntryLogger is designed to be compatible with the DefaultEntryLogger, and can switch from the DefaultEntryLogger to DirectEntryLogger smoothly. If we switch from DefaultEntryLogger to DirectEntryLogger, but can't switch back, it will be hard to apply the DirectEntryLogger in the existing BookKeeper cluster. The main reason we can't switch from DirectEntryLogger back to DefaultEntryLogger is that the EntryLogId generator is different and the new EntryLogId generator in DirectEntryLogger can cover more cases than the old one. The drawback of the old EntryLogId generator is as follows: - The last entrylogId is stored in the `lastId` file, and if this file is broken or be tampered, the new generated entrylogId will runs into unexpected value - The new entrylogId is generated with +1 logic, and if the entrylogId reaches Integer.MAX_VALUE, it will be reset to 0 [1]. If the 0.log still exists (may be impossible in common case), the old 0.log file will be overridden and cause data loss. We just need to change the last entryId loader logic in the DefaultEntryLogger to be compatible with the DirectEntryLogger, which is valuable for this work. [1] https://github.com/apache/bookkeeper/blob/3bf08fef12979bf3cff49ee953899f26c6281ad9/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLoggerAllocator.java#L147-L151 Thanks, Hang Yan Zhao 于2023年7月31日周一 17:57写道： > > > You should never do this in production. > > But the user may not know the limitation, they may believe that the switching > is compatible, not introduce some problems. > > > BTW do you have a possible solution? > > see https://github.com/apache/bookkeeper/pull/4041
Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.
> You should never do this in production. But the user may not know the limitation, they may believe that the switching is compatible, not introduce some problems. > BTW do you have a possible solution? see https://github.com/apache/bookkeeper/pull/4041
Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.
Il giorno lun 31 lug 2023 alle ore 02:40 Yan Zhao ha scritto: > > I am concerned about a possible scenario where a user tries to use DirectIO, > runs it for a period of time, and experiences some problem on the bookie > server side, but is unsure of the cause and suspects it may be related to > DirectIO. As a result, the user disables DirectIO and restarts. In this case, > data may be lost. You should never do this in production. BTW do you have a possible solution? Enrico > > > > On 2023/07/30 17:40:25 Enrico Olivelli wrote: > > I think that you should not do this kind of things. > > You should start a new bookie with the new configuration. > > > > It is not generally possible to guarantee compatibility between different > > implementations. > > > > I tend to close the case as 'won't fix' > > > > > > Enrico > > > > > > Il Dom 30 Lug 2023, 13:00 horizonzy ha scritto: > > > > > Hi, the followers. > > > I would need to talk about a serious matter. Especially, if you use > > > DirectEntryLogger in a production environment. > > > > > > Switching from DefaultEntryLogger to DirectEntryLogger and switching back > > > from DirectEntryLogger to DefaultEntryLogger may result in data loss. > > > > > > Here is the detailed issue > > > https://github.com/apache/bookkeeper/issues/4040 > > > > >
Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.
I am concerned about a possible scenario where a user tries to use DirectIO, runs it for a period of time, and experiences some problem on the bookie server side, but is unsure of the cause and suspects it may be related to DirectIO. As a result, the user disables DirectIO and restarts. In this case, data may be lost. On 2023/07/30 17:40:25 Enrico Olivelli wrote: > I think that you should not do this kind of things. > You should start a new bookie with the new configuration. > > It is not generally possible to guarantee compatibility between different > implementations. > > I tend to close the case as 'won't fix' > > > Enrico > > > Il Dom 30 Lug 2023, 13:00 horizonzy ha scritto: > > > Hi, the followers. > > I would need to talk about a serious matter. Especially, if you use > > DirectEntryLogger in a production environment. > > > > Switching from DefaultEntryLogger to DirectEntryLogger and switching back > > from DirectEntryLogger to DefaultEntryLogger may result in data loss. > > > > Here is the detailed issue > > https://github.com/apache/bookkeeper/issues/4040 > > >
Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.
I think that you should not do this kind of things. You should start a new bookie with the new configuration. It is not generally possible to guarantee compatibility between different implementations. I tend to close the case as 'won't fix' Enrico Il Dom 30 Lug 2023, 13:00 horizonzy ha scritto: > Hi, the followers. > I would need to talk about a serious matter. Especially, if you use > DirectEntryLogger in a production environment. > > Switching from DefaultEntryLogger to DirectEntryLogger and switching back > from DirectEntryLogger to DefaultEntryLogger may result in data loss. > > Here is the detailed issue > https://github.com/apache/bookkeeper/issues/4040 >
