Hello Patrick, On Sep 8, 2011, at 9:59 , Lukas Zeller wrote:
> PS: I'm in the code now :-) Done, but not tested beyond not interfering with normal operation in my context. Here's the patch, I'm currently shuffling my repos around so I haven't pushed it yet. Best Regards, Lukas ---- the patch ---- >From 25f7b2bf8187544cf357a13214f65b2902f14bc8 Mon Sep 17 00:00:00 2001 From: Lukas Zeller <l...@plan44.ch> Date: Thu, 8 Sep 2011 16:41:44 +0200 Subject: [PATCH] engine: added more merge options for DB implementations to ask engine for. Up to now, only DB_DataMerged (207) had a special meaning when returned by the DB backend for a server add. Now the following options exist for server add operations: - DB backend returns DB_DataMerged (207): This means that the backend has augmented the item with additional data, either from an external source, or by merging with another item in the sync set. The backend must return the localID for the augmented item. In case the augmented item was created by combining with a existing item already in the sync set, the localID of that existing item must be returned (and not a new ID!) The client will be updated with the augmented version. Additionally, if the new version was created by merge with an existing item in the sync set, a delete is sent to the client to remove the old (pre-merge) version of the item from the client. - DB backend returns DB_DataReplaced (208): This means that the backend has stored the item, but doing so, an existing item from the sync set was replaced. The backend must return the localID for the replaced item which must also become the localID of the newly stored version. The client item will not be updated (as the item was stored as received), but otherwise the processing is the same as described for DB_DataMerged (207) above. Additionally, for both server and client, add and replace, the following new option exists: - DB backend returns DB_Conflict (409) This means that the backend did NOT store the item, but has detected a need to merge the current version stored in the backend with the new version. The engine will read the current version, merge it with the incoming version, and issue an updateItem operation on the backend with the merge result. In server add case, after the merge the same processing occurs as for DB_DataMerged (see above). --- src/DB_interfaces/api_db/dbapi.cpp | 11 ++-- src/sysync/customimplds.cpp | 91 ++++++++++++++++++++++++++++++++--- src/sysync/customimplds.h | 2 + src/sysync/localengineds.cpp | 13 +++-- src/sysync/localengineds.h | 5 +- src/sysync_SDK/Sources/syerror.h | 11 ++++- src/sysync_SDK/Sources/sync_dbapi.h | 22 +++++++-- 7 files changed, 128 insertions(+), 27 deletions(-) diff --git a/src/DB_interfaces/api_db/dbapi.cpp b/src/DB_interfaces/api_db/dbapi.cpp index a4296ef..de6a04b 100755 --- a/src/DB_interfaces/api_db/dbapi.cpp +++ b/src/DB_interfaces/api_db/dbapi.cpp @@ -1889,7 +1889,7 @@ TSyError TDB_Api::InsertItem( cAppCharP aItemData, cAppCharP parentID, InsItemSFunc p= (InsItemSFunc)dm->ds.dsData.str.InsertItem; TSyError err= p( fContext, aItemData, &a ); - if (!err || err==DB_DataMerged) { + if (!err || err==DB_DataMerged || err==DB_DataReplaced || err==DB_Conflict) { Assign_ItemID( newID, a, parentID ); } // if @@ -1902,8 +1902,10 @@ TSyError TDB_Api::InsertItem( cAppCharP aItemData, TDB_Api_Str &newItemID ) { TDB_Api_ItemID nID; TSyError err= InsertItem( aItemData, "",nID ); - if (!err || - err==DB_DataMerged) GetItemID( nID, newItemID ); + if (!err || err==DB_DataMerged || err==DB_DataReplaced || err==DB_Conflict) { + GetItemID( nID, newItemID ); + } // if + return err; } // InsertItem @@ -1920,8 +1922,7 @@ TSyError TDB_Api::InsertItemAsKey( KeyH aItemKey, cAppCharP parentID, InsItemKFunc p= (InsItemKFunc)dm->ds.dsData.key.InsertItemAsKey; TSyError err= p( fContext, aItemKey, &a ); - if (!err || - err==DB_DataMerged) { + if (!err || err==DB_DataMerged || err==DB_DataReplaced || err==DB_Conflict) { Assign_ItemID( newID, a, parentID ); } // if diff --git a/src/sysync/customimplds.cpp b/src/sysync/customimplds.cpp index 497f554..f4092d4 100755 --- a/src/sysync/customimplds.cpp +++ b/src/sysync/customimplds.cpp @@ -2720,6 +2720,41 @@ localstatus TCustomImplDS::implProcessMap(cAppCharP aRemoteID, cAppCharP aLocalI +/// helper to merge database version of an item with the passed version of the same item +TMultiFieldItem *TCustomImplDS::mergeWithDatabaseVersion(TSyncItem *aSyncItemP) +{ + TStatusCommand dummy(fSessionP); + TMultiFieldItem *dbVersionItemP = (TMultiFieldItem *)newItemForRemote(aSyncItemP->getTypeID()); + if (!dbVersionItemP) return NULL; + // - set IDs + dbVersionItemP->setLocalID(aSyncItemP->getLocalID()); + dbVersionItemP->setRemoteID(aSyncItemP->getRemoteID()); + // - result is always a replace (item exists in DB) + dbVersionItemP->setSyncOp(sop_wants_replace); + // - try to get from DB + bool ok=logicRetrieveItemByID(*dbVersionItemP,dummy); + if (ok && dummy.getStatusCode()!=404) { + // item found in DB, merge with original item + bool changedNewVersion, changedDBVersion; + aSyncItemP->mergeWith(*dbVersionItemP, changedNewVersion, changedDBVersion, this); + PDEBUGPRINTFX(DBG_DATA,( + "Merged incoming item (winning,relevant,%smodified) with version from database (loosing,to-be-replaced,%smodified)", + changedNewVersion ? "" : "NOT ", + changedDBVersion ? "" : "NOT " + )); + } + else { + // no item found, we cannot force a conflict + PDEBUGPRINTFX(DBG_ERROR,("Could not retrieve database version of item, DB status code = %hd",dummy.getStatusCode())); + delete dbVersionItemP; + dbVersionItemP=NULL; + return NULL; + } + return dbVersionItemP; +} // TCustomImplDS::mergeWithDatabaseVersion + + + /// process item (according to operation: add/delete/replace - and for future: copy/move) /// @note data items will be sent only after StartWrite() bool TCustomImplDS::implProcessItem( @@ -2740,6 +2775,7 @@ bool TCustomImplDS::implProcessItem( // %%% bool RemoteIDKnown=false; TMapContainer::iterator mappos; TSyncOperation sop=sop_none; + TMultiFieldItem *augmentedItemP = NULL; TP_DEFIDX(li); TP_SWITCH(li,fSessionP->fTPInfo,TP_database); @@ -2795,15 +2831,28 @@ bool TCustomImplDS::implProcessItem( } // add item and retrieve new localID for it sta = apiAddItem(*myitemP,localID); + myitemP->setLocalID(localID.c_str()); // possibly following operations need to be based on new localID returned by add + // check for backend asking engine to do a merge + if (sta==DB_Conflict) { + // DB has detected item conflicts with data already stored in the database and + // request merging current data from the backend with new data before storing. + augmentedItemP = mergeWithDatabaseVersion(myitemP); + if (augmentedItemP==NULL) + sta = DB_Error; // no item found, DB error + else { + sta = apiUpdateItem(*augmentedItemP); // store augmented version back to DB + // in server case, further process like backend merge (but no need to fetch again, we just keep augmentedItemP) + if (IS_SERVER && sta==LOCERR_OK) sta = DB_DataMerged; + } + } if (IS_SERVER) { #ifdef SYSYNC_SERVER - if (sta==DB_DataMerged) { + if (sta==DB_DataMerged || sta==DB_DataReplaced) { // while adding, data was merged with pre-existing data from... // ..either data external from the sync set, such as augmenting a contact with info from a third-party lookup // ..or another item pre-existing in the sync set. PDEBUGPRINTFX(DBG_DATA,("Database adapter indicates that added item was merged with pre-existing data (status 207)")); - myitemP->setLocalID(localID.c_str()); // following searches need to be based on new localID returned by add - // check if the item resulting from merrge is known by the client already (in it's pre-merge form, that is) + // check if the item resulting from merge is known by the client already (in it's pre-merge form, that is) TMapContainer::iterator conflictingMapPos = findMapByLocalID(localID.c_str(), mapentry_normal); bool remoteAlreadyKnowsItem = conflictingMapPos!=fMapTable.end(); // also check if we have a (pre-merge) operation pending for that item already @@ -2841,14 +2890,30 @@ bool TCustomImplDS::implProcessItem( } } } - // now create a replace command to update the item added from the client with the merge result - // - this is like forcing a conflict, i.e. this loads the item by local/remoteid and adds it to - // the to-be-sent list of the server. - forceConflict(myitemP); + // if backend has not replaced, but merely merged data, we're done. Otherwise, client needs to be updated with + // merged/augmented version of the data + if (sta!=DB_DataReplaced) { + // now create a replace command to update the item added from the client with the merge result + // - this is like forcing a conflict, i.e. this loads the item by local/remoteid and adds it to + // the to-be-sent list of the server. + if (augmentedItemP) { + // augmented version was created in engine, just add that version to the list of items to be sent + SendItemAsServer(augmentedItemP); // takes ownership of augmentedItemP + augmentedItemP = NULL; + } + else { + // augmented version was created in backend, fetch it now and add to list of items to be sent + augmentedItemP = (TMultiFieldItem *)SendDBVersionOfItemAsServer(myitemP); + } + } sta = LOCERR_OK; // otherwise, treat as ok } #endif } // server + // - we don't need the augmented item any more if it still exists at this point + if (augmentedItemP) { + delete augmentedItemP; augmentedItemP = NULL; + } if (sta!=LOCERR_OK) { aStatusCommand.setStatusCode(sta); ok=false; @@ -2878,6 +2943,18 @@ bool TCustomImplDS::implProcessItem( myitemP->setLocalID(localID.c_str()); // update item sta = apiUpdateItem(*myitemP); + if (sta==DB_Conflict) { + // DB has detected item conflicts with data already stored in the database and + // request merging current data from the backend with new data before storing. + augmentedItemP = mergeWithDatabaseVersion(myitemP); + if (augmentedItemP==NULL) + sta = DB_Error; // no item found, DB error + else { + sta = apiUpdateItem(*augmentedItemP); // store augmented version back to DB + delete augmentedItemP; // forget now + } + } + // now check final status if (sta!=LOCERR_OK) { aStatusCommand.setStatusCode(sta); ok=false; diff --git a/src/sysync/customimplds.h b/src/sysync/customimplds.h index d8884d0..9587eb9 100755 --- a/src/sysync/customimplds.h +++ b/src/sysync/customimplds.h @@ -772,6 +772,8 @@ protected: // - Queue the data needed for finalisation (usually - relational link updates) // as a item copy with only finalisation-required fields void queueForFinalisation(TMultiFieldItem *aItemP); + /// helper to merge database version of an item with the passed version of the same item + TMultiFieldItem *mergeWithDatabaseVersion(TSyncItem *aSyncItemP); public: // - get last to-remote sync time lineartime_t getPreviousToRemoteSyncCmpRef(void) { return fPreviousToRemoteSyncCmpRef; }; diff --git a/src/sysync/localengineds.cpp b/src/sysync/localengineds.cpp index 4cb65bd..0e269bd 100644 --- a/src/sysync/localengineds.cpp +++ b/src/sysync/localengineds.cpp @@ -5144,8 +5144,9 @@ bool TLocalEngineDS::engProcessSyncOpItem( // Server Case // =========== -// helper to force a conflict -TSyncItem *TLocalEngineDS::forceConflict(TSyncItem *aSyncItemP) +// helper to cause database version of an item (as identified by aSyncItemP's ID) to be sent to client +// (aka "force a conflict") +TSyncItem *TLocalEngineDS::SendDBVersionOfItemAsServer(TSyncItem *aSyncItemP) { TStatusCommand dummy(fSessionP); // - create new item @@ -5171,7 +5172,7 @@ TSyncItem *TLocalEngineDS::forceConflict(TSyncItem *aSyncItemP) conflictingItemP=NULL; } return conflictingItemP; -} // TLocalEngineDS::forceConflict +} // TLocalEngineDS::SendDBVersionOfItemAsServer @@ -5302,7 +5303,7 @@ bool TLocalEngineDS::engProcessRemoteItemAsServer( // Note: a forced conflict can still occur even if item is rejected // (this has the effect of unconditionally letting the server item win) if (fForceConflict && syncop!=sop_add) { - conflictingItemP = forceConflict(aSyncItemP); + conflictingItemP = SendDBVersionOfItemAsServer(aSyncItemP); // Note: conflictingitem is always a replace if (conflictingItemP) { if (syncop==sop_delete) { @@ -5393,7 +5394,7 @@ bool TLocalEngineDS::engProcessRemoteItemAsServer( if (!echoItemP) conflictingItemP = getConflictingItemByRemoteID(aSyncItemP); // do not check conflicts if we have already created an echo // - check if we must force the conflict if (!conflictingItemP && fForceConflict) { - conflictingItemP=forceConflict(aSyncItemP); + conflictingItemP=SendDBVersionOfItemAsServer(aSyncItemP); } if (conflictingItemP) { // conflict only if other party has replace @@ -5506,7 +5507,7 @@ bool TLocalEngineDS::engProcessRemoteItemAsServer( if (!echoItemP) conflictingItemP = getConflictingItemByRemoteID(aSyncItemP); // - check if we must force the conflict if (!conflictingItemP && fForceConflict) { - conflictingItemP=forceConflict(aSyncItemP); + conflictingItemP=SendDBVersionOfItemAsServer(aSyncItemP); } bool deleteconflict=false; if (conflictingItemP) { diff --git a/src/sysync/localengineds.h b/src/sysync/localengineds.h index 0c918f3..7d34917 100755 --- a/src/sysync/localengineds.h +++ b/src/sysync/localengineds.h @@ -1118,8 +1118,9 @@ protected: void adjustLocalIDforSize(string &aLocalID, sInt32 maxguidsize, sInt32 prefixsize); /// for received GUIDs (Map command), obtain real GUID (might be temp GUID due to maxguidsize restrictions) void obtainRealLocalID(string &aLocalID); - /// helper to force a conflict (i.e. have a particular item in the sync set) - TSyncItem *forceConflict(TSyncItem *aSyncItemP); + /// helper to cause database version of an item (as identified by aSyncItemP's ID) to be sent to client + /// (aka "force a conflict") + TSyncItem *SendDBVersionOfItemAsServer(TSyncItem *aSyncItemP); #endif // SYSYNC_SERVER /// helper to save resume state either at end of request or explicitly at reception of a "suspend" SUPERDS_VIRTUAL localstatus engSaveSuspendState(bool aAnyway); diff --git a/src/sysync_SDK/Sources/syerror.h b/src/sysync_SDK/Sources/syerror.h index fd7a351..84c68b6 100644 --- a/src/sysync_SDK/Sources/syerror.h +++ b/src/sysync_SDK/Sources/syerror.h @@ -38,21 +38,28 @@ enum TSyErrorEnum { /** no content / end of file / end of iteration / empty/NULL value */ DB_NoContent = 204, - /** external data has been merged */ + + /** while adding item, additional data (from external source or other syncset item) has been merged */ DB_DataMerged = 207, + /** adding item has replaced an existing item */ + DB_DataReplaced = 208, + /** not authorized */ DB_Unauthorized = 401, /** forbidden / access denied */ DB_Forbidden = 403, /** object not found / unassigned field */ DB_NotFound = 404, - /** command not allowed (possibly: only at this time)*/ + /** command not allowed (possibly: only at this time) */ DB_NotAllowed = 405, /** proxy authentication required */ DB_ProxyAuth = 407, + /** conflict, item was not stored because it requires merge (on part of the engine) first */ + DB_Conflict = 409, + /** item already exists */ DB_AlreadyExists = 418, diff --git a/src/sysync_SDK/Sources/sync_dbapi.h b/src/sysync_SDK/Sources/sync_dbapi.h index 71964a6..192d330 100755 --- a/src/sysync_SDK/Sources/sync_dbapi.h +++ b/src/sysync_SDK/Sources/sync_dbapi.h @@ -876,11 +876,23 @@ _ENTRY_ TSyError StartDataWrite( CContext aContext ); * @param <aID> Database key of the new dataset. * * @return error code - * - LOCERR_OK ( =0 ), if successful - * - DB_DataMerged ( =207 ), if successful, but "ReadItem" requested to - * inform about updates - * - DB_Forbidden ( =403 ), if \<aItemData> can't be resolved - * - DB_Full ( =420 ), if not enough space in the DB + * - LOCERR_OK ( =0 ), if successful + * - DB_DataMerged ( =207 ), if successful, but item actually stored + * was updated with data from + * external source or another + * sync set item. Engine will + * request updated version + * using ReadItem. + * (server add case only) + * - DB_DataReplaced ( =208 ), if successful, but item added replaces + * another item that already + * existed in the sync set. + * (server add case only) + * - DB_Conflict ( =409 ), if database requests engine to merge + * existing data with the + * to-be stored data first. + * - DB_Forbidden ( =403 ), if \<aItemData> can't be resolved + * - DB_Full ( =420 ), if not enough space in the DB * - ... or any other SyncML error code, see Reference Manual * * NOTE: The memory for \<aItemID> must be allocated locally. -- 1.7.5.4+GitX _______________________________________________ os-libsynthesis mailing list os-libsynthesis@synthesis.ch http://lists.synthesis.ch/mailman/listinfo/os-libsynthesis