That was it, thanks! From: Munagala Ramanath [mailto:[email protected]] Sent: Tuesday, December 20, 2016 8:52 AM To: [email protected] Subject: Re: Data duplication between operators
Are you reusing objects in the input operator ? If so, try creating a new object for each tuple. Ram On Tue, Dec 20, 2016 at 5:38 AM, Doyle, Austin O. <[email protected]<mailto:[email protected]>> wrote: Also as a follow up, its not just repetition of some of the data points, it’s also just not sending some of the data points. When I try a 100 record one for example (1 – 100) it only sends the data record 100 a hundred times instead of 1 through 100. From: Doyle, Austin O. [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 20, 2016 8:14 AM To: [email protected]<mailto:[email protected]> Subject: RE: Data duplication between operators The downstream operator doesn’t seem to be failing and through some local tests I can confirm that each operator works separately. Could it be something else? -Austin From: Vlad Rozov [mailto:[email protected]] Sent: Monday, December 19, 2016 6:40 PM To: [email protected]<mailto:[email protected]> Subject: Re: Data duplication between operators This will be a bug unless the downstream operator constantly fails and is restored to a checkpoint in which case it is expected that it may get the same tuple multiple times. Thank you, Vlad On 12/19/16 11:33, Doyle, Austin O. wrote: I’m trying to send some sequential data between an S3 Input Operator and a CSV Parser operator. I added logging to see what the outputPort is emitting and it seems to be straightforward (data points 1 to 1000). I added logging on the input of the CSV Parser which receives 1000 data points but not the correct data points. It actually receives random data points multiple times (like point 57 twenty or so times). Has anyone seen anything like this? Thanks, Austin
