What will happen when PENDING_NO_OP is updated and committed, but (due to the no-force policy) the update to the metadata index is not reflected to disk except the update log and the commit log are written to disk. If the system crashes in this situation, the recovery must redo the update to the metadata index.
BTW, before we consider going to this update or upsert direction, it might be good to think about another way to prevent the situation you mentioned: "the stats are already persisted before 3) starts executing". Best, Young-Seok On Mon, Feb 15, 2016 at 12:03 AM, Ildar Absalyamov < [email protected]> wrote: > But the caller in this case (e.g. > QueryTranslator.handleCreateIndexStatement()) does not need to rollback to > the version of the record (i.e. index) prior to update(). If anything goes > wrong during that process the whole index creation is aborted and whatever > was written gets cleaned up (undo). Index creation operation is aborted and > there is no need to redo. > Does that make any difference? > > > On Feb 14, 2016, at 21:26, Young-Seok Kim <[email protected]> wrote: > > > > Considering the comment in BTree.update() method, the caller should take > > care of deleting the existing entry as well if it is required by the > upper > > level's semantic or need. > > In your situation, the upper level's semantic or need is to capture the > > update operation correctly from the transaction point of view such that > the > > update could be done "all or nothing" manner. > > As you pointed out in the first email, this is essentially about how the > > logging should be done correctly when you use BTree.update() method > instead > > of delete() followed by insert(). > > Considering this, what you want to do is actually implementing true > update > > or upsert instead of delete() followed by insert(). > > That being said, questions that you want to ask are > > Q1) what is "a" logical undo operation of update or upsert operation such > > that the logical undo operation correctly removes the effect of the > update > > or upsert operation when there is an abort request. > > Q2) what is "a" logical redo operation of update or upsert operation such > > that the logical redo operation correctly redo the effect of the update > or > > upsert operation when the operation was committed but the system failed > > before the committed effect was durable. > > > > Also, if the required logging informations for the logical undo and redo > > operations are different, the existing log record format needs to be > > modified to capture both informations in a single log record. > > Since the current logical operations (in transaction perspective) that we > > support are insert and delete (neither update nor upsert), their log > > records only keep the inserted entry or deleted entry for undo/redo > > operations. > > > > Best, > > Young-Seok > > > > > > On Sun, Feb 14, 2016 at 3:17 PM, Mike Carey <[email protected]> wrote: > > > >> That's what I figured. I think you're right! How hard to try it and > see? > >> Cheers, > >> Mike > >> > >> > >> On 2/14/16 3:06 PM, Ildar Absalyamov wrote: > >> > >>> I am not talking about user-facing insert operator, which Abdullah > >>> implemented, but rather LSMBtree-level upset, which seems to be a > separate > >>> index-modification operation according to > org.apache.hyracks.storage.am > >>> .common.ophelpers.IndexOperation > >>> According to > >>> > https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java#L329-L331 > >>> < > >>> > https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java#L329-L331 > > > >>> it seems that it is safe to make changes to the fields as long as they > are > >>> not keys, which is indeed the case with metadata changes I was talking > >>> about. > >>> > >>> On Feb 14, 2016, at 14:49, Mike Carey <[email protected]> wrote: > >>>> > >>>> An upsert would be fine, I would think; perhaps we didn't have it back > >>>> when that was done. > >>>> (Note that depending on which layer you are talking about, upsert is > >>>> synonymous with a > >>>> delete and then a (re)insert - though you are probably many layers > below > >>>> my world at the > >>>> user level where that's the case. :-)) > >>>> > >>>> On 2/14/16 2:47 PM, Ildar Absalyamov wrote: > >>>> > >>>>> A number of metadata-related changes, like creating a new index > >>>>> involves several stages: > >>>>> 1) Create an index with PENDING_ADD_OP > >>>>> 2) Bulkload the index > >>>>> 3) Delete the index with PENDING_ADD_OP and reinsert it with > >>>>> PENDING_NO_OP > >>>>> > >>>>> The last operation causes the issue with stats collection for > >>>>> particular index: sometimes the stats are already persisted before 3) > >>>>> starts executing, so they are become a subject to cascade delete, > hence are > >>>>> lost. > >>>>> I was wondering why an upset is not an option for step 3 instead of > >>>>> insert-delete? Are there any complications from transaction logging > >>>>> perspective? > >>>>> > >>>>> Best regards, > >>>>> Ildar > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Best regards, > >>> Ildar > >>> > >>> > >>> > >>> > >>> > >>> > >> > > Best regards, > Ildar > > > > >
