Hi Thomas, In our case we are writing the results back to maprstreams topic based on some validations.
Thanks Jaspal On Thursday, October 6, 2016, Thomas Weise <t...@apache.org> wrote: > Hi, > > which operators in your application are writing to external systems? > > When you look at the example from the blog (https://github.com/ > DataTorrent/examples/tree/master/tutorials/exactly-once), there is Kafka > input, which is configured to be idempotent. The results are written to > JDBC. That operator by itself supports exactly-once through transactions > (in conjunction with idempotent input), hence there is no need to configure > the processing mode at all. > > Thomas > > > On Thu, Oct 6, 2016 at 1:52 PM, Bandaru, Srinivas < > srinivas.band...@optum.com > <javascript:_e(%7B%7D,'cvml','srinivas.band...@optum.com');>> wrote: > >> Thanks Thomas. We have set the set all three operator properties, Kafka >> input operator has set to “Exactly_once” and downstream two operators set >> as AT_MOST_ONCE same as mentioned in documentation. We have also referred >> the documentation you have shared but couldn’t find the right properties >> for exactly_once scenario. Is there anything we are missing or need to add >> additional properties if someone else used this functionality. >> >> >> >> *Any help would be greatly appreciated??* >> >> >> >> We tried with different combinations and trying to set all three >> properties with “EXACTLY_ONCE”, But application throwing an with error >> saying *“If the processing mode for an operator is specified as >> EXACTLY_ONCE then the processing mode for all downstream operators should >> be specified as AT_MOST_ONCE otherwise it will result in an error.*” >> Application behavior is same as the Datatorrent documentation but it’s not >> working. >> >> >> >> Thanks, >> >> Srinivas >> >> >> >> >> >> *From:* Thomas Weise [mailto:t...@apache.org >> <javascript:_e(%7B%7D,'cvml','t...@apache.org');>] >> *Sent:* Thursday, October 06, 2016 3:03 PM >> *To:* users@apex.apache.org >> <javascript:_e(%7B%7D,'cvml','users@apex.apache.org');> >> *Subject:* Re: Datatorrent fault tolerance >> >> >> >> Hi, >> >> >> >> It would be necessary to know a bit more about your application for >> specific recommendations, but from what I see above, a few things don't >> look right. >> >> >> >> It appears that you are setting the processing mode on the input >> operator, which only reads from Kafka. Exactly-once is important for >> operators that affect the state of external systems, for example when >> writing to a database, or, producing messages to Kafka. >> >> >> >> There is also a problem with the documentation that may contribute to the >> confusion. The processing mode you are trying to use isn't really >> exactly-once and probably will be deprecated in a future release. Here are >> some better resources on fault tolerance and processing semantics: >> >> >> >> http://apex.apache.org/docs.html >> >> https://www.youtube.com/watch?v=FCMY6Ii89Nw >> >> http://www.slideshare.net/ApacheApexOrganizer/webinar-fault- >> toleranceandprocessingsemantics >> >> https://www.datatorrent.com/blog/end-to-end-exactly-once-wit >> h-apache-apex/ >> >> >> >> HTH, >> >> Thomas >> >> >> >> >> >> On Thu, Oct 6, 2016 at 11:26 AM, Bandaru, Srinivas < >> srinivas.band...@optum.com >> <javascript:_e(%7B%7D,'cvml','srinivas.band...@optum.com');>> wrote: >> >> Hi, Need some help! >> >> Need your advice in implementing the fault tolerance mechanism using >> datatorrent. Currently we are trying to implement with EXACTLY_ONCE >> scenario. >> >> We are referring the below documentation. The highlighted line says if we >> need to use the EXACTLY_ONCE processing mode, the downstream operators >> should have AT_MOST_ONCE. We added the operator level attributes in the >> properties.xml (attached below) and when we launch the application, no >> messages are coming through the last operator *topicUpdate *but we are >> able to see the messages when we comment those attributes. >> >> *Please have a look at it, either we are specifying the attributes >> incorrectly or is there any additional step to be done*. >> >> >> >> >> >> · *PROCESSING_MODE* >> >> static final Attribute >> <https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Attribute.html><Operator.ProcessingMode >> >> <https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Operator.ProcessingMode.html>> >> PROCESSING_MODE >> >> The payload processing mode for this operator - at most once, exactly >> once, or default at least once. If the processing mode for an operator is >> specified as AT_MOST_ONCE and no processing mode is specified for the >> downstream operators if any, the processing mode of the downstream >> operators is automatically set to AT_MOST_ONCE. If a different processing >> mode is specified for the downstream operators it will result in an error. If >> the processing mode for an operator is specified as EXACTLY_ONCE then the >> processing mode for all downstream operators should be specified as >> AT_MOST_ONCE otherwise it will result in an error. >> >> >> >> >> >> >> >> >> >> >> >> >> >> Thanks, >> >> Srinivas >> >> >> This e-mail, including attachments, may include confidential and/or >> proprietary information, and may be used only by the person or entity >> to which it is addressed. If the reader of this e-mail is not the intended >> recipient or his or her authorized agent, the reader is hereby notified >> that any dissemination, distribution or copying of this e-mail is >> prohibited. If you have received this e-mail in error, please notify the >> sender by replying to this message and delete this e-mail immediately. >> >> >> >> >> This e-mail, including attachments, may include confidential and/or >> proprietary information, and may be used only by the person or entity >> to which it is addressed. If the reader of this e-mail is not the intended >> recipient or his or her authorized agent, the reader is hereby notified >> that any dissemination, distribution or copying of this e-mail is >> prohibited. If you have received this e-mail in error, please notify the >> sender by replying to this message and delete this e-mail immediately. >> > >
