Re: Query syntax error in GetMongo processor?
Does your cluster have different users per database? Or to put it a
different way, is the user that you are authenticating as have at least
read/write privileges on every database?
The connection is created when the processor is scheduled so if you are
authenticating against a specific database, then the DB needs to be in the
connection string. If your authenticating as a user that can authenticate
against the admin DB, then you don't need to know the DB a priori.
That's my (probably over-simplistic) understanding anyways.
-Jason
On Tue, Aug 1, 2017 at 8:38 AM, James McMahon wrote:
> Follow up:
> 1- changing the query to be simple json without any db.find etc resolved
> that issue re: the invalid query. Thanks again to Pierre V. for pointing
> this out.
>
> 2- To eliminate the authentication problem I had to include the database
> name in the URI. So this
> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
> had to be this
> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144/
> DataServiceAudit
> I made the assumption that because NiFi specifically calls for Mongo
> Database Name in the second parameter of GetMongo, it means that the
> database name must be left off the Mongo URI. To me if you want it in the
> URI, then why have it at all as a separate parameter? But clearly that is
> not the case.
>
> It worked with these changes.
>
> On Sun, Jul 30, 2017 at 12:31 PM, James McMahon
> wrote:
>
>> Hello. I cannot get a simple test query to work against MongoDB from
>> GetMongo. Here is what I attempt to do:
>>
>> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
>> Mongo Database Name DataServiceAudit
>> Mongo Collection Name Audit
>> SSL Context Service No value set
>> Client Auth NONE
>> Query db.find({'Call':'Marko'})
>>
>> I leave the collection out (ie, db.Audit.find...) because I have
>> expressly stated the collection name in the processor.
>> I am trying to get this working in its simplest form before I tackle
>> themore complex, and so am not using SSL at this time.
>>
>> I am using NiFi 1.3.0 in this case.
>>
>> In the UI the processor indicates this problem:
>> 'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
>> org.bson.json.JSONParseException
>>
>> Why does it throw that error?
>>
>> If I remove the Query entirely, I can get the processor to enter a run
>> state. However then in the nifi-app.log it says there is an authentication
>> issue:
>>
>> 2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed
>> FlowFile Repository with 2313 records in 122 milliseconds
>> 2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
>> o.a.nifi.processors.mongodb.GetMongo
>> GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11]
>> Failed to execute query null due to com.mongodb.MongoTimeoutException:
>> Timed out after 3 ms while waiting for a server that matches
>> ReadPreferenceServerSelector{readPreference=primary}. Client view of
>> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
>> type=UNKNOWN, state=CONNECTING,
>> exception={com.mongodb.MongoSecurityException:
>> Exception authenticating MongoCredential{mechanism=null,
>> userName='nifi', source='admin', password=,
>> mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException:
>> Command failed with error 18: 'Authentication failed.' on server
>> 34.227.51.144:27017. The full response is { "ok" : 0.0, "errmsg" :
>> "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed"
>> }}}]: com.mongodb.MongoTimeoutException: Timed out after 3 ms while
>> waiting for a server that matches
>> ReadPreferenceServerSelector{readPreference=primary}.
>> Client view of cluster state is {type=UNKNOWN, servers=[{address=
>> 34.227.51.144:27017, type=UNKNOWN, state=CONNECTING,
>> exception={com.mongodb.MongoSecurityException: Exception authenticating
>> MongoCredential{mechanism=null, userName='nifi', source='admin',
>> password=, mechanismProperties={}}}, caused by
>> {com.mongodb.MongoCommandException: Command failed with error 18:
>> 'Authentication failed.' on server 34.227.51.144:27017. The full
>> response is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
>> "codeName" : "AuthenticationFailed" }}}]
>> com.mongodb.MongoTimeoutException: Timed out after 3 ms while
>> waiting for a server that matches
>> ReadPreferenceServerSelector{readPreference=primary}.
>> Client view of cluster state is {type=UNKNOWN, servers=[{address=
>> 34.227.51.144:27017, type=UNKNOWN, state=CONNECTING,
>> exception={com.mongodb.MongoSecurityException: Exception authenticating
>> MongoCredential{mechanism=null, userName='nifi', source='admin',
>> password=, mechanismProperties={}}}, caused by
>> {com.mongodb.MongoCommandException: Command failed with error 18:
>> 'Authentication fa
Re: Query syntax error in GetMongo processor?
Follow up:
1- changing the query to be simple json without any db.find etc resolved
that issue re: the invalid query. Thanks again to Pierre V. for pointing
this out.
2- To eliminate the authentication problem I had to include the database
name in the URI. So this
Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
had to be this
Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
/DataServiceAudit
I made the assumption that because NiFi specifically calls for Mongo
Database Name in the second parameter of GetMongo, it means that the
database name must be left off the Mongo URI. To me if you want it in the
URI, then why have it at all as a separate parameter? But clearly that is
not the case.
It worked with these changes.
On Sun, Jul 30, 2017 at 12:31 PM, James McMahon
wrote:
> Hello. I cannot get a simple test query to work against MongoDB from
> GetMongo. Here is what I attempt to do:
>
> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
> Mongo Database Name DataServiceAudit
> Mongo Collection Name Audit
> SSL Context Service No value set
> Client Auth NONE
> Query db.find({'Call':'Marko'})
>
> I leave the collection out (ie, db.Audit.find...) because I have expressly
> stated the collection name in the processor.
> I am trying to get this working in its simplest form before I tackle
> themore complex, and so am not using SSL at this time.
>
> I am using NiFi 1.3.0 in this case.
>
> In the UI the processor indicates this problem:
> 'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
> org.bson.json.JSONParseException
>
> Why does it throw that error?
>
> If I remove the Query entirely, I can get the processor to enter a run
> state. However then in the nifi-app.log it says there is an authentication
> issue:
>
> 2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository
> Successfully checkpointed FlowFile Repository with 2313 records in 122
> milliseconds
> 2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
> o.a.nifi.processors.mongodb.GetMongo
> GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11]
> Failed to execute query null due to com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]: com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]
> com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
> for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}.
> Client view of cluster state is {type=UNKNOWN, servers=[{address=
> 34.227.51.144:27017, type=UNKNOWN, state=CONNECTING,
> exception={com.mongodb.MongoSecurityException: Exception authenticating
> MongoCredential{mechanism=null, userName='nifi', source='admin',
> password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException:
> Command failed with error 18: 'Authentication failed.' on server
> 34.227.51.144:27017. The full response is { "ok" : 0.0, "errmsg" :
> "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed"
> }}}]
> at com.mongodb.connection.BaseCluster.createTimeoutException(
> BaseCluster.java:369)
> at com.mongodb.connection.BaseCluster.selectServer(
> BaseCluster.java:101)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:75)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:71)
> at com.mongodb.binding.ClusterBinding.getReadConnectionSource(
> ClusterBinding.java:63)
> at com.mongodb.operation.Operati
Re: Query syntax error in GetMongo processor?
Hi James,
The query needs to be a valid JSON. You can have a look into the test class
to get some examples:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoTest.java
In your case, I assume the query parameter should only be:
{'Call':'Marko'}
Regarding the authentication issue, not sure I can help.
Hope this helps,
Pierre.
2017-07-30 18:31 GMT+02:00 James McMahon :
> Hello. I cannot get a simple test query to work against MongoDB from
> GetMongo. Here is what I attempt to do:
>
> Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
> Mongo Database Name DataServiceAudit
> Mongo Collection Name Audit
> SSL Context Service No value set
> Client Auth NONE
> Query db.find({'Call':'Marko'})
>
> I leave the collection out (ie, db.Audit.find...) because I have expressly
> stated the collection name in the processor.
> I am trying to get this working in its simplest form before I tackle
> themore complex, and so am not using SSL at this time.
>
> I am using NiFi 1.3.0 in this case.
>
> In the UI the processor indicates this problem:
> 'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
> org.bson.json.JSONParseException
>
> Why does it throw that error?
>
> If I remove the Query entirely, I can get the processor to enter a run
> state. However then in the nifi-app.log it says there is an authentication
> issue:
>
> 2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository
> Successfully checkpointed FlowFile Repository with 2313 records in 122
> milliseconds
> 2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
> o.a.nifi.processors.mongodb.GetMongo
> GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11]
> Failed to execute query null due to com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]: com.mongodb.MongoTimeoutException:
> Timed out after 3 ms while waiting for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}. Client view of
> cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
> type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException:
> Exception authenticating MongoCredential{mechanism=null, userName='nifi',
> source='admin', password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException: Command failed with error 18:
> 'Authentication failed.' on server 34.227.51.144:27017. The full response
> is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
> "codeName" : "AuthenticationFailed" }}}]
> com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
> for a server that matches
> ReadPreferenceServerSelector{readPreference=primary}.
> Client view of cluster state is {type=UNKNOWN, servers=[{address=
> 34.227.51.144:27017, type=UNKNOWN, state=CONNECTING,
> exception={com.mongodb.MongoSecurityException: Exception authenticating
> MongoCredential{mechanism=null, userName='nifi', source='admin',
> password=, mechanismProperties={}}}, caused by
> {com.mongodb.MongoCommandException:
> Command failed with error 18: 'Authentication failed.' on server
> 34.227.51.144:27017. The full response is { "ok" : 0.0, "errmsg" :
> "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed"
> }}}]
> at com.mongodb.connection.BaseCluster.createTimeoutException(
> BaseCluster.java:369)
> at com.mongodb.connection.BaseCluster.selectServer(
> BaseCluster.java:101)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:75)
> at com.mongodb.binding.ClusterBinding$
> ClusterBindingConnectionSource.(ClusterBinding.java:71)
> at com.mongodb.binding.ClusterBinding.getReadConnectionSource(
> ClusterBinding.java:63)
> at com.mongodb.operation.OperationHelper.withConnection(
> OperationHelper.java:210)
> at com.mongodb.operation.FindOperation.execute(
> FindOperation.java:482)
> at com.mongodb.operation.FindOperation.execute(
> FindOperation.java:79)
> at com.mongodb.Mongo.execute(Mongo.java:772)
> at com.mongodb.Mongo$2.execute(Mongo.java:759)
> at
Query syntax error in GetMongo processor?
Hello. I cannot get a simple test query to work against MongoDB from
GetMongo. Here is what I attempt to do:
Mongo URI mongodb://nifi:[my pwd]@34.227.51.144
Mongo Database Name DataServiceAudit
Mongo Collection Name Audit
SSL Context Service No value set
Client Auth NONE
Query db.find({'Call':'Marko'})
I leave the collection out (ie, db.Audit.find...) because I have expressly
stated the collection name in the processor.
I am trying to get this working in its simplest form before I tackle
themore complex, and so am not using SSL at this time.
I am using NiFi 1.3.0 in this case.
In the UI the processor indicates this problem:
'Query' validated against 'db.find({'Call':'Marko'})' is invalid because
org.bson.json.JSONParseException
Why does it throw that error?
If I remove the Query entirely, I can get the processor to enter a run
state. However then in the nifi-app.log it says there is an authentication
issue:
2017-07-30 16:27:27,606 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 2313 records in 122 milliseconds
2017-07-30 16:27:41,147 ERROR [Timer-Driven Process Thread-7]
o.a.nifi.processors.mongodb.GetMongo
GetMongo[id=100c11db-1fb9-1138-1050-0f63cda85d11] Failed to execute query
null due to com.mongodb.MongoTimeoutException: Timed out after 3 ms
while waiting for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]:
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting
for a server that matches
ReadPreferenceServerSelector{readPreference=primary}. Client view of
cluster state is {type=UNKNOWN, servers=[{address=34.227.51.144:27017,
type=UNKNOWN, state=CONNECTING,
exception={com.mongodb.MongoSecurityException: Exception authenticating
MongoCredential{mechanism=null, userName='nifi', source='admin',
password=, mechanismProperties={}}}, caused by
{com.mongodb.MongoCommandException: Command failed with error 18:
'Authentication failed.' on server 34.227.51.144:27017. The full response
is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18,
"codeName" : "AuthenticationFailed" }}}]
at
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at
com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:210)
at
com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
at
com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
at
org.apache.nifi.processors.mongodb.GetMongo.onTrigger(GetMongo.java:158)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.ap
