Hi, Ok I think I have post to quickly. So I run more test.
UPSERT document with Operator Enable test1 : upsert the document {"id1":1,$set{"value":"abc"}}, PutMongo updateQueryKey=id1 => ok test2 : upsert the document {"id1":1,"id2":1,$set:{"value":"abc"}}, PutMongo updateQueryKey=id1,id2 line 220 : removeUpdateKeys() doen't remove anything because this method deals only with property with daot (p1.p2) line 229 : update.remove() doesn't remove anything because trying to remove many keys from the map So the update parameter send to mongodb is : {"id1":1,"id2":1,$set:{"value":"abc"}} and mongo throws Invalid BSON field name id1 exception test3 : upsert the document {"ids.id1":1,"value":"abc"}, PutMongo updateQueryKey=ids.id1 => ok test4 : upsert the document {"ids.id1":1,"ids.id2":1,"value":"abc"}, PutMongo updateQueryKey=ids.id1,ids.id2 => ok test5 : upsert the document {"id1":1,"ids.id2":1,"value":"abc"}, PutMongo updateQueryKey=id1,ids.id2 line 220 : removeUpdateKeys() remove "ids.id2" from the document ine 229 : update.remove() doesn't remove anything because trying to remove many keys from the map So the update parameter send to mongodb is : {"id1":1,$set:{"value":"abc"}} and mongo throws Invalid BSON field name id1 exception UPSERT document With whole document test 6 : upsert the document {"id1":1,"value":"abc"}, PutMongo updateQuertKey=id1 => ok test 7 : upsert the document {"id1":1,"id2":1,"value":"abc"}, PutMongo updateQuertKey=id1;id2 => ok test 8 : upsert the document {"ids.id1":1,"value":"abc"}, PutMongo updateQueryKey=ids.id1 query : {ids.id1=1} doc : {value=abc} inserted : { "_id" : ObjectId("5b44c5aeab8179ca72466a2f"), "value" : "abc" } test 9 : upsert the document {"ids.id1":1,"ids.id2":1,"value":"abc"}, PutMongo updateQueryKey=ids.id1,ids.id2 query : {ids.id1=1, ids.id2=1} doc : {value=abc} inserted : { "_id" : ObjectId("5b44c51cab8179ca7246648c"), "value" : "abc" } test 10 : upsert the document {"id1":1,"ids.id2":1,"value":"abc"}, PutMongo updateQueryKey=id1,ids.id2 query : {id1=1, value=abc}, doc : {id1=1, value=abc} inserted :{ "_id" : ObjectId("5b44c44bab8179ca72465c96"), "id1" : 1, "value" : "abc" } I think we should distinguish 2 update mode : - first use case : using updateQuery (not empty). In this case the query and the doc (flowfile) should be transmit to mongoDB without any change. - second use case : using udateQueryKey : the processor have to build the query based on the document and updateQueryKey paramter: then we have 2 mode : - WithOperatorEnable : the doc (flowfile) have the following form {"key":"a","keys.k1":"b",....,$set:{"v1":"abc","v.v1":"abcd"},$unset:{....}....} the updateQuertKey parameter : key,keys.k1 so the processor have to build the query parameter : {"key":"a","keys.k1":"b"} and remove the keys from de document. - WithWholeDocument : the json doc is {"key":"a","keys":{"k1":"b","k2":"c"},"v1":"abc","v":{"v1":"abcd"}...} the updateQuertKey parameter : key,keys.k1 so the processor have to extract the valurs of the different keys and build the query :{"key":"a","keys.k1":"b"}, the doc remains unchanged. I update the code of my custom processor, it works. Here is the java code part I have modified for putMongoProcessor. (See attached file: PutMongo-part.java) What do you think about that ? Thanks (Embedded image moved to file: pic04966.jpg) Yves HAMEL Direction Digital & Systèmes d’Information Groupe LM_DATA MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9 Tél. : +33 (0)5 49 09 36 06 Email : mon.em...@macif.fr / Pré Doyen 2 – bureau 999 www.macif.fr - Appli présente sur Google Play Store & Apple Store (Embedded image moved to file: pic07376.jpg) De : Mike Thomsen <mikerthom...@gmail.com> A : users@nifi.apache.org, Date : 09/07/2018 17:10 Objet : Re: Issue with PutMongoProcessor using upsert mode. removeUpdateKeys should be skipping your key because it's a simple key. private void removeUpdateKeys(String updateKeyParam, Map doc) { String[] parts = updateKeyParam.split(",[\\s]*"); for (String part : parts) { if (part.contains(".")) { doc.remove(part); } } } Are you sure the value is just identifiant and not something like something .identifiant? On Mon, Jul 9, 2018 at 9:42 AM Yves HAMEL <yha...@macif.fr> wrote: Hi, I try to "upsert" document in a mongodb collection. When the document exists, PutMongo processor update the document but when the document doesn't exist the processor do not create it. Here is the setting of my PutMongo processor : Mode : updateg Upsert : true Update Query Key : identifiant Update Query : null Update Mode : With Operator Enabled I have read the PutMongodb.java and notice a strange thing when preparing mongodb update request : if (!StringUtils.isBlank(updateKey)) { query = parseUpdateKey(updateKey, (Map)doc); removeUpdateKeys(updateKey, (Map)doc); } else { query = Document.parse(filterQuery); } When using updateKey, it builds the query with the updateKey and removes this key from the document so it creates the document without the key. The document is created but there is missing part of the document. When using update_query mode, it does remove anything form the document. I simply remove the removeUpdateKeys() method call in a test custum processor. And it works fine. Is that path correct ? Thank Yves HAMEL (Embedded image moved to file: pic07711.jpg) Yves HAMEL Direction Digital & Systèmes d’Information Groupe LM_DATA MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9 Tél. : +33 (0)5 49 09 36 06 Email : mon.em...@macif.fr / Pré Doyen 2 – bureau 999 www.macif.fr - Appli présente sur Google Play Store & Apple Store (Embedded image moved to file: pic28253.jpg)
PutMongo-part.java
Description: Binary data