RE: REST provenance/lineage API

2016-11-17 Thread philippe.gibert 

Thanks , very simple at the end ☺ !

Last question  , If I can ..
Initially I have a  process group named  PG1 with some processors.
I want to Execute the DFlow  located in PG1
and then  with and external program  call via REST lineage/provenance  API to 
get the complete  report about the flow.
My question is :
Is there a way to start  with the ProcessgroupName ( ie PG1)   and get the  
flow_uid  executed in the process Group with 1 REST API ?

Best regards
Phil


From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: jeudi 17 novembre 2016 15:25
To: users@nifi.apache.org
Subject: Re: REST provenance/lineage API

Phil,

Those details are available in the provenance event which can be accessed 
through this endpoint:

GET /nifi-api/provenance-events/
If your instance is clustered however, you'll also need to include 
clusterNodeId as a query parameter.

Matt

On Thu, Nov 17, 2016 at 8:46 AM, 
<philippe.gib...@orange.com<mailto:philippe.gib...@orange.com>> wrote:
Hello matt
Almost perfect !
I call nifi REST api with the following JSON body in the post ( just indicating 
the flow uuid)
--
{ "lineage":{  
"request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc"
 } } }
--
And  results are records like this but I would expect  the ProcessorID ( 
componentID?) and  Processor Name involved in each record ? is it possible ?
Philippe

{
"id":"322",
"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
"type":"EVENT",
"eventType":"RECEIVE",
"millis":1479375558360,
"timestamp":"11/17/2016 09:39:18.360 UTC”
}



From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: REST provenance/lineage API

Hello Matt, I have the same problem with the provenance/lineage REST call

curl 'http://localhost:8080/nifi-api/provenance/lineage/..

very difficult for me (sorry ☹ ) to  find the right  json  body to send in the 
POST call .
if you can give an example again

Philippe
Best regards

-
Thanks a lot Matt
It works nicely !


From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: REST provenance api search options

Philippe,

Here's an example command for initiating a provenance search:

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: 
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' 
--data-binary 
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 
EST","endDate":"11/15/2016 23:59:59 
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed

The available searchable fields are defined in your nifi.properties file under 
the following properties:

nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes

Because the searches can be long running they are performed asynchronously. 
These means that the curl command above creates the search request but does not 
wait for it to complete. Instead, you'll need to get the uuid to the search 
request to continue to GET it until the search completes. Once completed, you 
should DELETE the search request. Open up the Dev Tools in your browser to seen 
this sequence of requests in action.

Let me know if you have any more questions. Thanks.

Matt


On Tue, Nov 15, 2016 at 2:47 AM, 
<philippe.gib...@orange.com<mailto:philippe.gib...@orange.com>> wrote:

Hello,

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my 
processor ie (ProcessorID ) but it’s not very clear for me how to fill the 
searchableFields.

Is something  as the following to be right  ? and in this case where do i put 
the ProcessorID? in the field or  in the Id ? and what for label?

---

curl -i -X PUT -H 'Content-Type: application/json' -d 
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
 ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options’

--

​ please can you clarify ? with an example ?

Best regards

phil



_



Ce message et ses pieces jointes peuvent contenir des informations 
confi

Re: REST provenance/lineage API

2016-11-17 Thread Matt Gilman 
Phil,

Those details are available in the provenance event which can be accessed
through this endpoint:

GET /nifi-api/provenance-events/

If your instance is clustered however, you'll also need to include
clusterNodeId as a query parameter.

Matt

On Thu, Nov 17, 2016 at 8:46 AM, <philippe.gib...@orange.com> wrote:

> Hello matt
>
> Almost perfect !
>
> I call nifi REST api with the following JSON body in the post ( just
> indicating the flow uuid)
>
> --
>
> { "lineage":{  "request":{"lineageRequestType":"FLOWFILE"
> ,"uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } }
>
> --
>
> And  results are records like this but I would expect  the ProcessorID (
> componentID?) and  Processor Name involved in each record ? is it possible ?
>
> Philippe
>
>
>
> {
>
> "id":"322",
>
> "flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
>
> "type":"EVENT",
>
> "eventType":"RECEIVE",
>
> "millis":1479375558360,
>
> "timestamp":"11/17/2016 09:39:18.360 UTC”
>
> }
>
>
>
>
>
>
>
> *From:* GIBERT Philippe IMT/OLPS
> *Sent:* mercredi 16 novembre 2016 17:15
> *To:* users@nifi.apache.org
> *Subject:* REST provenance/lineage API
>
>
>
> Hello Matt, I have the same problem with the provenance/lineage REST call
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance/lineage/..
>
>
>
> very difficult for me (sorry L ) to  find the right  json  body to send
> in the POST call .
>
> if you can give an example again
>
>
>
> Philippe
>
> Best regards
>
>
>
>
> *-*
>
> Thanks a lot Matt
>
> It works nicely !
>
>
>
>
>
> *From:* Matt Gilman [mailto:matt.c.gil...@gmail.com
> <matt.c.gil...@gmail.com>]
> *Sent:* mardi 15 novembre 2016 15:08
> *To:* users@nifi.apache.org
> *Subject:* Re: REST provenance api search options
>
>
>
> Philippe,
>
>
>
> Here's an example command for initiating a provenance search:
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
> application/json' -H 'Accept: application/json, text/javascript, */*;
> q=0.01' --data-binary '{"provenance":{"request":{"
> maxResults":1000,"startDate":"11/15/2016 00:00:00
> EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"
> FlowFileUUID":"","Filename":" flowfile>","ProcessorID":""' --compressed
>
>
>
> The available searchable fields are defined in your nifi.properties file
> under the following properties:
>
>
>
> nifi.provenance.repository.indexed.fields
>
> nifi.provenance.repository.indexed.attributes
>
>
>
> Because the searches can be long running they are performed
> asynchronously. These means that the curl command above creates the search
> request but does not wait for it to complete. Instead, you'll need to get
> the uuid to the search request to continue to GET it until the search
> completes. Once completed, you should DELETE the search request. Open up
> the Dev Tools in your browser to seen this sequence of requests in action.
>
>
>
> Let me know if you have any more questions. Thanks.
>
>
>
> Matt
>
>
>
>
>
> On Tue, Nov 15, 2016 at 2:47 AM, <philippe.gib...@orange.com> wrote:
>
> Hello,
>
> My SW context : nifi 1.0.0/ubuntu
>
>  I am trying to use the provenance search options . I have the Id of my
> processor ie (ProcessorID ) but it’s not very clear for me how to fill the
> searchableFields.
>
> Is something  as the following to be right  ? and in this case where do i
> put the ProcessorID? in the field or  in the Id ? and what for label?
>
> ---
>
> curl -i -X PUT -H 'Content-Type: application/json' -d
> '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
> ID","type":"STRING"}]}
>
> ' http://localhost:8080/nifi-api/provenance/search-options’
>
> --
>
> ​ please can you clarify ? with an example ?
>
> Best regards
>
> phil
>
>
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

RE: REST provenance/lineage API

2016-11-17 Thread philippe.gibert 
Hello matt
Almost perfect !
I call nifi REST api with the following JSON body in the post ( just indicating 
the flow uuid)
--
{ "lineage":{  
"request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc"
 } } }
--
And  results are records like this but I would expect  the ProcessorID ( 
componentID?) and  Processor Name involved in each record ? is it possible ?
Philippe

{
"id":"322",
"flowFileUuid":"00b40d7d-50ac-48a9-8926-ba884a89bd28",
"type":"EVENT",
"eventType":"RECEIVE",
"millis":1479375558360,
"timestamp":"11/17/2016 09:39:18.360 UTC”
}



From: GIBERT Philippe IMT/OLPS
Sent: mercredi 16 novembre 2016 17:15
To: users@nifi.apache.org
Subject: REST provenance/lineage API

Hello Matt, I have the same problem with the provenance/lineage REST call

curl 'http://localhost:8080/nifi-api/provenance/lineage/..

very difficult for me (sorry ☹ ) to  find the right  json  body to send in the 
POST call .
if you can give an example again

Philippe
Best regards

-
Thanks a lot Matt
It works nicely !


From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: REST provenance api search options

Philippe,

Here's an example command for initiating a provenance search:

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: 
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' 
--data-binary 
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 
EST","endDate":"11/15/2016 23:59:59 
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed

The available searchable fields are defined in your nifi.properties file under 
the following properties:

nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes

Because the searches can be long running they are performed asynchronously. 
These means that the curl command above creates the search request but does not 
wait for it to complete. Instead, you'll need to get the uuid to the search 
request to continue to GET it until the search completes. Once completed, you 
should DELETE the search request. Open up the Dev Tools in your browser to seen 
this sequence of requests in action.

Let me know if you have any more questions. Thanks.

Matt


On Tue, Nov 15, 2016 at 2:47 AM, 
<philippe.gib...@orange.com<mailto:philippe.gib...@orange.com>> wrote:

Hello,

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my 
processor ie (ProcessorID ) but it’s not very clear for me how to fill the 
searchableFields.

Is something  as the following to be right  ? and in this case where do i put 
the ProcessorID? in the field or  in the Id ? and what for label?

---

curl -i -X PUT -H 'Content-Type: application/json' -d 
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
 ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options’

--

​ please can you clarify ? with an example ?

Best regards

phil



_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Re: REST provenance/lineage API

2016-11-16 Thread Matt Gilman 
Phil,

The lineage endpoints follow the same asynchronous model that the
provenance endpoints do. Here is an example curl command to initiate a
lineage search request:

curl 'https://localhost:8443/nifi-api/provenance/lineage' -H 'Content-Type:
application/json' -H 'Accept: application/json, text/javascript, */*;
q=0.01' --data-binary
'{"lineage":{"request":{"lineageRequestType":"FLOWFILE","uuid":"05f547fe-e15d-4186-a645-fbed6621aee1","eventId":"0"}}}'
--compressed

The contents of this request comes from the details of a selected
provenance event. I'd suggest using your browser's Dev Tools to watch the
requests in action while you are searching provenance and viewing lineage
graphs.

Let me know if you have any other questions. Thanks!

Matt

On Wed, Nov 16, 2016 at 11:15 AM,  wrote:

> Hello Matt, I have the same problem with the provenance/lineage REST call
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance/lineage/..
>
>
>
> very difficult for me (sorry L ) to  find the right  json  body to send
> in the POST call .
>
> if you can give an example again
>
>
>
> Philippe
>
> Best regards
>
>
> *-*
>
> Thanks a lot Matt
>
> It works nicely !
>
>
>
>
>
> *From:* Matt Gilman [mailto:matt.c.gil...@gmail.com
> ]
> *Sent:* mardi 15 novembre 2016 15:08
> *To:* users@nifi.apache.org
> *Subject:* Re: REST provenance api search options
>
>
>
> Philippe,
>
>
>
> Here's an example command for initiating a provenance search:
>
>
>
> curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
> application/json' -H 'Accept: application/json, text/javascript, */*;
> q=0.01' --data-binary '{"provenance":{"request":{"
> maxResults":1000,"startDate":"11/15/2016 00:00:00
> EST","endDate":"11/15/2016 23:59:59 EST","searchTerms":{"
> FlowFileUUID":"","Filename":" flowfile>","ProcessorID":""' --compressed
>
>
>
> The available searchable fields are defined in your nifi.properties file
> under the following properties:
>
>
>
> nifi.provenance.repository.indexed.fields
>
> nifi.provenance.repository.indexed.attributes
>
>
>
> Because the searches can be long running they are performed
> asynchronously. These means that the curl command above creates the search
> request but does not wait for it to complete. Instead, you'll need to get
> the uuid to the search request to continue to GET it until the search
> completes. Once completed, you should DELETE the search request. Open up
> the Dev Tools in your browser to seen this sequence of requests in action.
>
>
>
> Let me know if you have any more questions. Thanks.
>
>
>
> Matt
>
>
>
>
>
> On Tue, Nov 15, 2016 at 2:47 AM,  wrote:
>
> Hello,
>
> My SW context : nifi 1.0.0/ubuntu
>
>  I am trying to use the provenance search options . I have the Id of my
> processor ie (ProcessorID ) but it’s not very clear for me how to fill the
> searchableFields.
>
> Is something  as the following to be right  ? and in this case where do i
> put the ProcessorID? in the field or  in the Id ? and what for label?
>
> ---
>
> curl -i -X PUT -H 'Content-Type: application/json' -d
> '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
> ID","type":"STRING"}]}
>
> ' http://localhost:8080/nifi-api/provenance/search-options’
>
> --
>
> ​ please can you clarify ? with an example ?
>
> Best regards
>
> phil
>
>
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

REST provenance/lineage API

2016-11-16 Thread philippe.gibert 
Hello Matt, I have the same problem with the provenance/lineage REST call

curl 'http://localhost:8080/nifi-api/provenance/lineage/..

very difficult for me (sorry ☹ ) to  find the right  json  body to send in the 
POST call .
if you can give an example again

Philippe
Best regards
-
Thanks a lot Matt
It works nicely !


From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org
Subject: Re: REST provenance api search options

Philippe,

Here's an example command for initiating a provenance search:

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: 
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' 
--data-binary 
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 
EST","endDate":"11/15/2016 23:59:59 
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed

The available searchable fields are defined in your nifi.properties file under 
the following properties:

nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes

Because the searches can be long running they are performed asynchronously. 
These means that the curl command above creates the search request but does not 
wait for it to complete. Instead, you'll need to get the uuid to the search 
request to continue to GET it until the search completes. Once completed, you 
should DELETE the search request. Open up the Dev Tools in your browser to seen 
this sequence of requests in action.

Let me know if you have any more questions. Thanks.

Matt


On Tue, Nov 15, 2016 at 2:47 AM, 
> wrote:

Hello,

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my 
processor ie (ProcessorID ) but it’s not very clear for me how to fill the 
searchableFields.

Is something  as the following to be right  ? and in this case where do i put 
the ProcessorID? in the field or  in the Id ? and what for label?

---

curl -i -X PUT -H 'Content-Type: application/json' -d 
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
 ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options’

--

​ please can you clarify ? with an example ?

Best regards

phil



_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.