Provenance doesn't work with FetchS3Object
I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object. "Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)" Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile? Regards, Ben The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Provenance doesn't work with FetchS3Object
Ben I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling. Have you tried it without incoming FlowFiles? Cheers Oleg On Oct 15, 2015, at 3:38 PM, Ben Meng mailto:ben.m...@lifelock.com>> wrote: I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object. "Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)" Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile? Regards, Ben The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Provenance doesn't work with FetchS3Object
Oleg, Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t. Regards, Ben From: Oleg Zhurakousky Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Date: Thursday, October 15, 2015 at 12:49 PM To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Subject: Re: Provenance doesn't work with FetchS3Object Ben I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling. Have you tried it without incoming FlowFiles? Cheers Oleg On Oct 15, 2015, at 3:38 PM, Ben Meng mailto:ben.m...@lifelock.com>> wrote: I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object. "Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)" Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile? Regards, Ben The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Provenance doesn't work with FetchS3Object
Ben, Since FetchS3Object is not creating the FlowFile, it should not be emitting a RECEIVE event. This is certainly a bug. I have created a ticket for this: https://issues.apache.org/jira/browse/NIFI-1038 <https://issues.apache.org/jira/browse/NIFI-1038> Thanks -Mark > On Oct 15, 2015, at 3:57 PM, Ben Meng wrote: > > Oleg, > > Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it > just didn’t generate any output. I’ve also confirmed the behavior by > inspecting the codes. The first thing it does is to check if there’s an > existing FlowFile, and returns if there isn’t. > > Regards, > Ben > > > From: Oleg Zhurakousky > Reply-To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > Date: Thursday, October 15, 2015 at 12:49 PM > To: "users@nifi.apache.org <mailto:users@nifi.apache.org>" > Subject: Re: Provenance doesn't work with FetchS3Object > > Ben > > I don’t think it needs an incoming FlowFile. It is a scheduled component and > will retrieve contents based on how you configure scheduling. > Have you tried it without incoming FlowFiles? > > Cheers > Oleg > >> On Oct 15, 2015, at 3:38 PM, Ben Meng > <mailto:ben.m...@lifelock.com>> wrote: >> >> I understand that FetchS3Object processor requires an incoming FlowFile to >> trigger it. The problem is that FetchS3Object emits a RECEIVE provenance >> event for the existing FlowFile. That event causes following error when I >> try to open the lineage chart for a simple flow: GenerateFlowFile -> >> FetchS3Object. >> >> "Found cycle in graph. This indicates that multiple events were registered >> claiming to have generated the same FlowFile (UUID = >> 40f58407-ea10-4843-b8d1-be0e24f685aa)" >> >> Should FetchS3Object create a new FlowFile for each fetched object? If so, >> does it really require an incoming FlowFile? >> >> Regards, >> Ben >> The information contained in this transmission may contain privileged and >> confidential information. It is intended only for the use of the person(s) >> named above. If you are not the intended recipient, you are hereby notified >> that any review, dissemination, distribution or duplication of this >> communication is strictly prohibited. If you are not the intended recipient, >> please contact the sender by reply email and destroy all copies of the >> original message. > > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended recipient, > please contact the sender by reply email and destroy all copies of the > original message.
Re: Provenance doesn't work with FetchS3Object
Thanks Mark. That makes sense. Regards, Ben From: Mark Payne Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Date: Thursday, October 15, 2015 at 1:09 PM To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Subject: Re: Provenance doesn't work with FetchS3Object Ben, Since FetchS3Object is not creating the FlowFile, it should not be emitting a RECEIVE event. This is certainly a bug. I have created a ticket for this: https://issues.apache.org/jira/browse/NIFI-1038 Thanks -Mark On Oct 15, 2015, at 3:57 PM, Ben Meng mailto:ben.m...@lifelock.com>> wrote: Oleg, Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t. Regards, Ben From: Oleg Zhurakousky Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Date: Thursday, October 15, 2015 at 12:49 PM To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" Subject: Re: Provenance doesn't work with FetchS3Object Ben I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling. Have you tried it without incoming FlowFiles? Cheers Oleg On Oct 15, 2015, at 3:38 PM, Ben Meng mailto:ben.m...@lifelock.com>> wrote: I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object. "Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)" Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile? Regards, Ben The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
