RE: REST provenance/lineage API
Hello matt
Almost perfect !
I call nifi REST api with the following JSON body in the post ( just indicating
the flow uuid)
--
{ "lineage":{
"request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc"
} } }
--
And results are records like this but I would expect the ProcessorID (
componentID?) and Processor Name involved in each record ? is it possible ?
Philippe
{
"id":"322",
"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
"type":"EVENT",
"eventType":"RECEIVE",
"millis":1479375558360,
"timestamp":"11/17/2016 09:39:18.360 UTC”
}
From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: users@nifi.apache.org
Subject: REST provenance/lineage API
Hello Matt, I have the same problem with the provenance/lineage REST call
curl 'http://localhost:8080/nifi-api/provenance/lineage/..
very difficult for me (sorry ☹ ) to find the right json body to send in the
POST call .
if you can give an example again
Philippe
Best regards
-
Thanks a lot Matt
It works nicely !
From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org
Subject: Re: REST provenance api search options
Philippe,
Here's an example command for initiating a provenance search:
curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01'
--data-binary
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00
EST","endDate":"11/15/2016 23:59:59
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed
The available searchable fields are defined in your nifi.properties file under
the following properties:
nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes
Because the searches can be long running they are performed asynchronously.
These means that the curl command above creates the search request but does not
wait for it to complete. Instead, you'll need to get the uuid to the search
request to continue to GET it until the search completes. Once completed, you
should DELETE the search request. Open up the Dev Tools in your browser to seen
this sequence of requests in action.
Let me know if you have any more questions. Thanks.
Matt
On Tue, Nov 15, 2016 at 2:47 AM,
mailto:philippe.gib...@orange.com>> wrote:
Hello,
My SW context : nifi 1.0.0/ubuntu
I am trying to use the provenance search options . I have the Id of my
processor ie (ProcessorID ) but it’s not very clear for me how to fill the
searchableFields.
Is something as the following to be right ? and in this case where do i put
the ProcessorID? in the field or in the Id ? and what for label?
---
curl -i -X PUT -H 'Content-Type: application/json' -d
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
ID","type":"STRING"}]}
' http://localhost:8080/nifi-api/provenance/search-options’
--
please can you clarify ? with an example ?
Best regards
phil
_
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
Controller services visibility problem
Hello all, We are using NiFi 1.0 over http without any authentication, so all our users are coming to NiFi as anonymous. The problem we are having is of controller services visibility. So if we create a controller service (say database connection) from the top right option of the canvas, this is not visible within a group of processors. Also the contrary is also true, if we create a controller service for a group of processors, this is not visible to the rest of the canvas. Is there a way to assign visibility for a controller service, e.g. set global visibility for a service, so that we don't have to recreate it in all the groups of processors that need it? As a side note, we didn't have this problem with NiFi versions before 1.0. Many thanks, Panos
Re: REST provenance/lineage API
Phil,
Those details are available in the provenance event which can be accessed
through this endpoint:
GET /nifi-api/provenance-events/
If your instance is clustered however, you'll also need to include
clusterNodeId as a query parameter.
Matt
On Thu, Nov 17, 2016 at 8:46 AM, wrote:
> Hello matt
>
> Almost perfect !
>
> I call nifi REST api with the following JSON body in the post ( just
> indicating the flow uuid)
>
> --
>
> { "lineage":{ "request":{"lineageRequestType":"FLOWFILE"
> ,"uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } }
>
> --
>
> And results are records like this but I would expect the ProcessorID (
> componentID?) and Processor Name involved in each record ? is it possible ?
>
> Philippe
>
>
>
> {
>
> "id":"322",
>
> "flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
>
> "type":"EVENT",
>
> "eventType":"RECEIVE",
>
> "millis":1479375558360,
>
> "timestamp":"11/17/2016 09:39:18.360 UTC”
>
> }
>
>
>
>
>
>
>
> *From:* GIBERT Philippe IMT/OLPS
> *Sent:* mercredi 16 novembre 2016 17:15
> *To:* users@nifi.apache.org
> *Subject:* REST provenance/lineage API
>
>
>
> Hello Matt, I have the same problem with the provenance/lineage REST call
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance/lineage/..
>
>
>
> very difficult for me (sorry L ) to find the right json body to send
> in the POST call .
>
> if you can give an example again
>
>
>
> Philippe
>
> Best regards
>
>
>
>
> *-*
>
> Thanks a lot Matt
>
> It works nicely !
>
>
>
>
>
> *From:* Matt Gilman [mailto:matt.c.gil...@gmail.com
> ]
> *Sent:* mardi 15 novembre 2016 15:08
> *To:* users@nifi.apache.org
> *Subject:* Re: REST provenance api search options
>
>
>
> Philippe,
>
>
>
> Here's an example command for initiating a provenance search:
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
> application/json' -H 'Accept: application/json, text/javascript, */*;
> q=0.01' --data-binary '{"provenance":{"request":{"
> maxResults":1000,"startDate":"11/15/2016 00:00:00
> EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"
> FlowFileUUID":"","Filename":" flowfile>","ProcessorID":""' --compressed
>
>
>
> The available searchable fields are defined in your nifi.properties file
> under the following properties:
>
>
>
> nifi.provenance.repository.indexed.fields
>
> nifi.provenance.repository.indexed.attributes
>
>
>
> Because the searches can be long running they are performed
> asynchronously. These means that the curl command above creates the search
> request but does not wait for it to complete. Instead, you'll need to get
> the uuid to the search request to continue to GET it until the search
> completes. Once completed, you should DELETE the search request. Open up
> the Dev Tools in your browser to seen this sequence of requests in action.
>
>
>
> Let me know if you have any more questions. Thanks.
>
>
>
> Matt
>
>
>
>
>
> On Tue, Nov 15, 2016 at 2:47 AM, wrote:
>
> Hello,
>
> My SW context : nifi 1.0.0/ubuntu
>
> I am trying to use the provenance search options . I have the Id of my
> processor ie (ProcessorID ) but it’s not very clear for me how to fill the
> searchableFields.
>
> Is something as the following to be right ? and in this case where do i
> put the ProcessorID? in the field or in the Id ? and what for label?
>
> ---
>
> curl -i -X PUT -H 'Content-Type: application/json' -d
> '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
> ID","type":"STRING"}]}
>
> ' http://localhost:8080/nifi-api/provenance/search-options’
>
> --
>
> please can you clarify ? with an example ?
>
> Best regards
>
> phil
>
>
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
>
RE: REST provenance/lineage API
Thanks , very simple at the end ☺ !
Last question , If I can ..
Initially I have a process group named PG1 with some processors.
I want to Execute the DFlow located in PG1
and then with and external program call via REST lineage/provenance API to
get the complete report about the flow.
My question is :
Is there a way to start with the ProcessgroupName ( ie PG1) and get the
flow_uid executed in the process Group with 1 REST API ?
Best regards
Phil
From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: jeudi 17 novembre 2016 15:25
To: users@nifi.apache.org
Subject: Re: REST provenance/lineage API
Phil,
Those details are available in the provenance event which can be accessed
through this endpoint:
GET /nifi-api/provenance-events/
If your instance is clustered however, you'll also need to include
clusterNodeId as a query parameter.
Matt
On Thu, Nov 17, 2016 at 8:46 AM,
mailto:philippe.gib...@orange.com>> wrote:
Hello matt
Almost perfect !
I call nifi REST api with the following JSON body in the post ( just indicating
the flow uuid)
--
{ "lineage":{
"request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc"
} } }
--
And results are records like this but I would expect the ProcessorID (
componentID?) and Processor Name involved in each record ? is it possible ?
Philippe
{
"id":"322",
"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
"type":"EVENT",
"eventType":"RECEIVE",
"millis":1479375558360,
"timestamp":"11/17/2016 09:39:18.360 UTC”
}
From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: users@nifi.apache.org
Subject: REST provenance/lineage API
Hello Matt, I have the same problem with the provenance/lineage REST call
curl 'http://localhost:8080/nifi-api/provenance/lineage/..
very difficult for me (sorry ☹ ) to find the right json body to send in the
POST call .
if you can give an example again
Philippe
Best regards
-
Thanks a lot Matt
It works nicely !
From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org
Subject: Re: REST provenance api search options
Philippe,
Here's an example command for initiating a provenance search:
curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01'
--data-binary
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00
EST","endDate":"11/15/2016 23:59:59
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed
The available searchable fields are defined in your nifi.properties file under
the following properties:
nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes
Because the searches can be long running they are performed asynchronously.
These means that the curl command above creates the search request but does not
wait for it to complete. Instead, you'll need to get the uuid to the search
request to continue to GET it until the search completes. Once completed, you
should DELETE the search request. Open up the Dev Tools in your browser to seen
this sequence of requests in action.
Let me know if you have any more questions. Thanks.
Matt
On Tue, Nov 15, 2016 at 2:47 AM,
mailto:philippe.gib...@orange.com>> wrote:
Hello,
My SW context : nifi 1.0.0/ubuntu
I am trying to use the provenance search options . I have the Id of my
processor ie (ProcessorID ) but it’s not very clear for me how to fill the
searchableFields.
Is something as the following to be right ? and in this case where do i put
the ProcessorID? in the field or in the Id ? and what for label?
---
curl -i -X PUT -H 'Content-Type: application/json' -d
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
ID","type":"STRING"}]}
' http://localhost:8080/nifi-api/provenance/search-options’
--
please can you clarify ? with an example ?
Best regards
phil
_
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law
Re: Controller services visibility problem
Hi Panos, You are correct in that Controller Services that are created in the top-right corner will not be available to Processors. These are "controller-level" services and are available only to Reporting Tasks and other Controller Services. If you want to use a Controller Service for Processors, then you need to create the service at the Process Group level (in the Operate palette). A Controller Service that is created here is available to any Processor in this group and any child group. So if you want a Controller Service to be available to all Processors, you can simply create the Controller Service at the Root Process Group, and all child groups will then have access to it. Thanks -Mark On Nov 17, 2016, at 9:10 AM, Panos Geo mailto:panospanos1...@outlook.com>> wrote: Hello all, We are using NiFi 1.0 over http without any authentication, so all our users are coming to NiFi as anonymous. The problem we are having is of controller services visibility. So if we create a controller service (say database connection) from the top right option of the canvas, this is not visible within a group of processors. Also the contrary is also true, if we create a controller service for a group of processors, this is not visible to the rest of the canvas. Is there a way to assign visibility for a controller service, e.g. set global visibility for a service, so that we don’t have to recreate it in all the groups of processors that need it? As a side note, we didn't have this problem with NiFi versions before 1.0. Many thanks, Panos
Re: Controller services visibility problem
Hello Panos, With the 1.0.0 update, security policies were added to process groups and controller level features. This means that there is now a difference between creating a Controller Service (CS) which is scoped to the controller (ie. for Reporting tasks) and creating a CS which is scoped to a process group. So in order to create a CS that is available to all processors, just create it in the root process group. All processors will be able to reference it since they are all within the scope of the root group. Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com On Thursday, November 17, 2016 9:11 AM, Panos Geo wrote: Hello all, We are using NiFi 1.0 over http without any authentication, so all our users are coming to NiFi as anonymous. The problem we are having is of controller services visibility. So if we create a controller service (say database connection) from the top right option of the canvas, this is not visible within a group of processors. Also the contrary is also true, if we create a controller service for a group of processors, this is not visible to the rest of the canvas. Is there a way to assign visibility for a controller service, e.g. set global visibility for a service, so that we don’t have to recreate it in all the groups of processors that need it? As a side note, we didn't have this problem with NiFi versions before 1.0. Many thanks, Panos
SplitJson:GC Overhead Limit Exceeded
Hi All, I have a flowfile containing a JSON array with 30k objects that I am trying to split into separate flowfiles for down stream processing. The problem is the processor reports a GC Overhead Limit Exceeded warning and administratively yields. Is there anyway of setting up a back pressure option or some changes to the nifi config to best address this. Thanks, Mike
Re: SplitJson:GC Overhead Limit Exceeded
..just for info in bootstrap.conf my heap size is as follows: java.arg.2=-Xms512m java.arg.3=-Xmx512m Would it be a simple case of increasing this? The size of the flowfile json array is 35MB. Mike On 17 November 2016 at 15:47, Mike Harding wrote: > Hi All, > > I have a flowfile containing a JSON array with 30k objects that I am > trying to split into separate flowfiles for down stream processing. > > The problem is the processor reports a GC Overhead Limit Exceeded warning > and administratively yields. > > Is there anyway of setting up a back pressure option or some changes to > the nifi config to best address this. > > Thanks, > Mike >
Resource directory paths are malformed: docs
Hello all, I rebuild nifi from scratch and after the deploy Jetty won't restartslogs returns this error: at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_92] at org.apache.nifi.NiFi.(NiFi.java:146) ~[nifi-runtime-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] at org.apache.nifi.NiFi.main(NiFi.java:243) ~[nifi-runtime-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] Caused by: java.lang.IllegalStateException: Resource directory paths are malformed: docs at org.apache.nifi.web.server.JettyServer.createDocsWebApp(JettyServer.java:540) ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] at org.apache.nifi.web.server.JettyServer.loadWars(JettyServer.java:321) ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] at org.apache.nifi.web.server.JettyServer.(JettyServer.java:144) ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] any idea where to look / how to fix this ?
Re: SplitJson:GC Overhead Limit Exceeded
Hi Mike, Certainly, I would recommend trying to change the max heap to say 2 GB and see if that gives you what you need. Looking at the code, it does look like this Processor may not be the most efficient in how it is parsing the JSON. There are libraries, for example, that provide a "Streaming JSON" interface, but this Processor loads the entire JSON into heap and then creates an Object Model from it. Also, what do you have set for the Max Concurrent Tasks? If you have multiple threads simultaneously running, you could have each one using up quite a lot of heap. Thanks -Mark On Nov 17, 2016, at 10:54 AM, Mike Harding mailto:mikeyhard...@gmail.com>> wrote: ..just for info in bootstrap.conf my heap size is as follows: java.arg.2=-Xms512m java.arg.3=-Xmx512m Would it be a simple case of increasing this? The size of the flowfile json array is 35MB. Mike On 17 November 2016 at 15:47, Mike Harding mailto:mikeyhard...@gmail.com>> wrote: Hi All, I have a flowfile containing a JSON array with 30k objects that I am trying to split into separate flowfiles for down stream processing. The problem is the processor reports a GC Overhead Limit Exceeded warning and administratively yields. Is there anyway of setting up a back pressure option or some changes to the nifi config to best address this. Thanks, Mike
Re: SplitJson:GC Overhead Limit Exceeded
The backing library of the Json processors does indeed require loading the entire doc into memory. We should make sure this consideration is documented if not already. Could be an interesting idea to not tie SplitJson to this library given that it might not need all the functionalities of JsonPath and would likely be a good candidate for streaming. On Thu, Nov 17, 2016 at 11:23 Mark Payne wrote: > Hi Mike, > > Certainly, I would recommend trying to change the max heap to say 2 GB and > see if that gives you what you need. > Looking at the code, it does look like this Processor may not be the most > efficient in how it is parsing the JSON. > There are libraries, for example, that provide a "Streaming JSON" > interface, but this Processor loads the entire JSON > into heap and then creates an Object Model from it. > > Also, what do you have set for the Max Concurrent Tasks? If you have > multiple threads simultaneously running, you could > have each one using up quite a lot of heap. > > Thanks > -Mark > > > On Nov 17, 2016, at 10:54 AM, Mike Harding wrote: > > ..just for info in bootstrap.conf my heap size is as follows: > > java.arg.2=-Xms512m > > java.arg.3=-Xmx512m > > Would it be a simple case of increasing this? The size of the flowfile > json array is 35MB. > > Mike > > > > On 17 November 2016 at 15:47, Mike Harding wrote: > > Hi All, > > I have a flowfile containing a JSON array with 30k objects that I am > trying to split into separate flowfiles for down stream processing. > > The problem is the processor reports a GC Overhead Limit Exceeded warning > and administratively yields. > > Is there anyway of setting up a back pressure option or some changes to > the nifi config to best address this. > > Thanks, > Mike > > > >
Re: SplitJson:GC Overhead Limit Exceeded
If we consider streaming for SplitJson (or a new version of it), we wouldn't be able to support the "micro-batch" functionality as is in SplitJson today (like the fragment.count attribute, for example). Might not be a concern, or might warrant a new processor (SplitJsonStreaming, e.g.) . Regards, Matt On Thu, Nov 17, 2016 at 11:36 AM, Aldrin Piri wrote: > The backing library of the Json processors does indeed require loading the > entire doc into memory. We should make sure this consideration is documented > if not already. > > Could be an interesting idea to not tie SplitJson to this library given that > it might not need all the functionalities of JsonPath and would likely be a > good candidate for streaming. > On Thu, Nov 17, 2016 at 11:23 Mark Payne wrote: >> >> Hi Mike, >> >> Certainly, I would recommend trying to change the max heap to say 2 GB and >> see if that gives you what you need. >> Looking at the code, it does look like this Processor may not be the most >> efficient in how it is parsing the JSON. >> There are libraries, for example, that provide a "Streaming JSON" >> interface, but this Processor loads the entire JSON >> into heap and then creates an Object Model from it. >> >> Also, what do you have set for the Max Concurrent Tasks? If you have >> multiple threads simultaneously running, you could >> have each one using up quite a lot of heap. >> >> Thanks >> -Mark >> >> >> On Nov 17, 2016, at 10:54 AM, Mike Harding wrote: >> >> ..just for info in bootstrap.conf my heap size is as follows: >> >> java.arg.2=-Xms512m >> >> java.arg.3=-Xmx512m >> >> Would it be a simple case of increasing this? The size of the flowfile >> json array is 35MB. >> >> Mike >> >> >> >> On 17 November 2016 at 15:47, Mike Harding wrote: >>> >>> Hi All, >>> >>> I have a flowfile containing a JSON array with 30k objects that I am >>> trying to split into separate flowfiles for down stream processing. >>> >>> The problem is the processor reports a GC Overhead Limit Exceeded warning >>> and administratively yields. >>> >>> Is there anyway of setting up a back pressure option or some changes to >>> the nifi config to best address this. >>> >>> Thanks, >>> Mike >> >> >> >
Re: Resource directory paths are malformed: docs
Hi Alessio, Could you copy/paste the full stack trace? Thanks! Pierre 2016-11-17 17:03 GMT+01:00 Alessio Palma : > Hello all, > > I rebuild nifi from scratch and after the deploy Jetty won't restartslogs > returns this error: > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > ~[na:1.8.0_92] > at org.apache.nifi.NiFi.(NiFi.java:146) > ~[nifi-runtime-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.NiFi.main(NiFi.java:243) ~[nifi-runtime-1.0.0-SNAPSHOT. > jar:1.0.0-SNAPSHOT] > Caused by: java.lang.IllegalStateException: Resource directory paths are > malformed: docs > at > org.apache.nifi.web.server.JettyServer.createDocsWebApp(JettyServer.java:540) > ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.web.server.JettyServer.loadWars(JettyServer.java:321) > ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > at org.apache.nifi.web.server.JettyServer.(JettyServer.java:144) > ~[nifi-jetty-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT] > > any idea where to look / how to fix this ? > > > >
