Mike, Matt,
Thanks for the help. Mike - the one json thing was a typo.
I just got this working as the following:
File on disk, testQuery.json: {"$set":{"status":"Stage_2"}}
GetFile ----FlowFile---> PutMongo
NiFi Processor Properties:
Mode: update
Upsert: false
Update Query Key: No value set
Update Query: {"customId":"abc"}
Update Mode: With operators enabled
This feels like a pretty contrived example here that got it working. I'm
not aware of any of the current processors that output with the Mongo
Operators in a JSON file, which would mean if I'm operating on straight
JSON, I'd have to manipulate it in some way to get that in there. That
seems fairly complicated. Is there a simpler way to do this and I'm
missing it?
Just some background --
What I'm trying to do:
I have the ID of a record in the database, and I only want to update 1
field. I could take the ID and do a GetMongo -> JoltTransform -> PutMongo
and replace the entire document, but that seems like a lot of processors
for this use-case.
My initial approach/recommendation:
I initially thought the Update Query property would take what Mongo's
CLI is expecting {"customId":{"$customId"},{"$set": {"status":"Stage_2"}}.
Instead, it works by splitting on the comma - (1) Query as Property, (2)
Update with Operator as FlowFile. I think supporting an update using
operators that's set within the NiFi Processor Properties, vs in the
incoming FlowFile could make this processor a lot more flexible.
Thanks,
Ryan
On Wed, Jun 27, 2018 at 4:26 PM Matt Burgess <[email protected]> wrote:
> If "customId" is a flow file attribute (and the property supports
> expression language), then you just have your braces and $ swapped,
> try ${customId} instead of {$customId}
> On Wed, Jun 27, 2018 at 4:15 PM Mike Thomsen <[email protected]>
> wrote:
> >
> > > can you clarify if you mean the NiFi Processor Property "Update
> Query", or the FlowFile require proper json?
> >
> > Both.
> >
> > > I'm getting an error: MongoDB due to redstartDocument can only be
> called when CurrentBSONType is DOCUMENT, not when CurrentBSONType is Array.
> >
> > PutMongo does not support arrays.
> >
> > > I'm trying to do the latter, "specify a document that contains update
> operators"... On the mongo command line, the update would be:
> > > db.collection.update({"customId":{$customId},{$set:
> {"status":"Stage_2"}});
> >
> > I don' t know if that's a typo, but it should be
> {"customId":{$customId}} followed by {$set: {"status":"Stage_2"}}. Your
> version was one big document. What we would expect is a query like {
> "customId": "XYZ"}. As mentioned, $customId is not a valid JSON value.
> >
> > Let me know if that helps.
> >
> > Mike
> >
> > On Wed, Jun 27, 2018 at 3:15 PM Ryan Hendrickson <
> [email protected]> wrote:
> >>
> >> Hi Mike,
> >> Just curious - any other suggestions?
> >>
> >> Thanks,
> >> Ryan
> >>
> >> On Thu, Jun 21, 2018 at 5:23 PM Ryan Hendrickson <
> [email protected]> wrote:
> >>>
> >>> Thanks for the suggestions, can you clarify if you mean the NiFi
> Processor Property "Update Query", or the FlowFile require proper json?
> I'm not sure how to get proper json with the $set in there.
> >>>
> >>> I made the following modifications based it:
> >>>
> >>> NiFi Processors Properties:
> >>> Update Query: [{"customId":{$customId}},{"$set":
> {"status":"Stage_2"}}]
> >>> Update Mode: With operators enabled --- Confirmed that I've been
> using this.
> >>>
> >>> FlowFile Contents: [{"customId":{$customId}},{"$set":
> {"status":"Stage_2"}}]
> >>>
> >>> I'm getting an error: MongoDB due to redstartDocument can only be
> called when CurrentBSONType is DOCUMENT, not when CurrentBSONType is Array.
> >>>
> >>> The NiFi Docs for Put Mongo say [1]:
> >>> Update Query: Specify a full MongoDB query to be used for the lookup
> query to do an update/upsert. Supports Expression Language: true
> >>> Update Mode: Choose an update mode. You can either supply a JSON
> document to use as a direct replacement or specify a document that contains
> update operators like $set and $unset.
> >>>
> >>> I'm trying to do the latter, "specify a document that contains update
> operators"... On the mongo command line, the update would be:
> >>> db.collection.update({"customId":{$customId},{$set:
> {"status":"Stage_2"}});
> >>>
> >>> In the NiFi flow, all I have is the customId, and I want to set a
> status in the database when I receive it, but the database has a larger set
> of doc keys/values. I know I could do GetMongo -> JoltTransform for status
> -> PutMongo, but it seems silly to use 3 processors when this PutMongo
> looks like it can do it...
> >>>
> >>> Thanks,
> >>> Ryan
> >>>
> >>> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-mongodb-nar/1.6.0/org.apache.nifi.processors.mongodb.PutMongo/index.html
> >>>
> >>>
> >>>
> >>> On Thu, Jun 21, 2018 at 4:57 PM Mike Thomsen <[email protected]>
> wrote:
> >>>>
> >>>> Two things:
> >>>>
> >>>> 1. You need to use valid JSON. Your query is not a valid JSON example
> because some of the values are not quoted.
> >>>> 2. You need to make sure the update option is set to use operators,
> not use document.
> >>>>
> >>>> Let us know if that helps.
> >>>>
> >>>> Mike
> >>>>
> >>>> On Thu, Jun 21, 2018 at 3:19 PM Ryan Hendrickson <
> [email protected]> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> I can't seem to figure out the right combo of parameters to get a
> document to update in Mongo using the PutMongo processor and the $set
> operator.
> >>>>>
> >>>>> Try 1:
> >>>>> The incoming flowfile contains the customId: abc
> >>>>>
> >>>>> NiFi Processor Properties:
> >>>>> Mode: update
> >>>>> Upsert: false
> >>>>> Update Query Key: No value set
> >>>>> Update Query: {"customId":{$customId}},{$set:
> {"status":"Stage_2"}}
> >>>>> Update Mode: With operators enabled
> >>>>>
> >>>>> This consistently fails, the abbreviated log output:
> >>>>> PutMongo Failed to insert into MongoDB due to
> com.mongodb.util.JSONParseException:
> >>>>> abc
> >>>>> ...
> >>>>> at com.mongodb.util.JSONParser.parse(JSON.java:230)
> >>>>> ...
> >>>>> at
> org.apache.nifi.processors.mongodb.PutMongo.onTrigger(PutMongo.java:201)
> >>>>>
> >>>>>
> https://github.com/apache/nifi/blob/rel/nifi-1.6.0/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/PutMongo.java#L201
> >>>>>
> >>>>> It looks like it's trying to parse the incoming flowfile as a JSON
> document with the above parameters set.
> >>>>>
> >>>>> Try 2:
> >>>>> With that in mind, I changed my input flowfile to be a json object,
> but I don't think it should need to be, because I'm using the Update Query
> with Operators.
> >>>>> New incoming flow file: {"customId":"abc"}
> >>>>>
> >>>>> This allows it to get line 225, before it fails with:
> >>>>> PutMongo Failed to insert into MongoDB due to
> java.lang.IllegalArgumentException: Invalid BSON field name customId:
> >>>>> at org.bson.AbstractBsonWriter.writeName(AbstractBsonWriter.java:494)
> >>>>> at
> org.apache.nifi.processors.mongodb.PutMongo.onTrigger(PutMongo:225)
> >>>>>
> >>>>>
> https://github.com/apache/nifi/blob/rel/nifi-1.6.0/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/PutMongo.java#L225
> >>>>>
> >>>>> Line 225: //collection.updateOne(query, update, new
> UpdateOptions().upsert(upsert));
> >>>>> query = {"customId":{$customId}},{$set: {"status":"Stage_2"}}
> >>>>> update = {"customId":"abc"}
> >>>>> It looks like the 'update' variable, is my incoming flowfile. I'm
> not sure why it would be, based on my understanding of the processor
> properties works.
> >>>>>
> >>>>> If anyone has any insight on how to set this up for using the
> operators to update a document, I'd really appreciate the insight. I'm
> lost in debugging.
> >>>>>
> >>>>> Thanks,
> >>>>> Best,
> >>>>> Ryan
> >>>>>
>
