Saransh I would like to have a discussion, if we can record it, publish it in youtube for example and share the link here again, in order to allow the community to be in the loop.
Does it work? Regards El lun, 25 oct 2021 a las 22:37, Saransh Sharma (<sara...@muellners.com>) escribió: > Timestamping could also be added as part of the key. > > I agree with Victor here, we need to keep it simple, Hey @Victor Romero > <victor.rom...@fintecheando.mx> would you like to discuss this and > flesh it out together ? > > On Tue, Oct 26, 2021 at 9:01 AM VICTOR MANUEL ROMERO RODRIGUEZ < > victor.rom...@fintecheando.mx> wrote: > >> I will extend the previous email, >> >> At asynchronous transactions: >> >> The trigger (or POST) for doing asynchronous transactions while sending >> notifications (email, sms, push) over MQ/JMS queues, the user/customer >> experience will be affected by receiving twice or more payment >> reminders, payment notification (they can think that the payment was >> applied twice), account creations, loan applications, etc and the cost >> associated for each message sent over the notification channel. >> >> At journey entry: >> >> The trigger (or POST) for doing any transactions (monetary or non >> monetary) only 1 entry must be recorded in all the systems so then the >> logging and monitoring systems (fraud, alerting, etc) can correlate the >> transaction (using surrogate keys) and later the data scientist or data >> engineers won't be blaming the CBS engineering team. >> >> At transaction caching: >> >> We got a use case where the loan origination process was taking some time >> to execute the process and because of the demand of the financial product >> people was trying to execute several times to have a loan, so then the >> business requirement was to keep on cache the same "loan request" if the >> people ID was the same (in our case the National Population ID) and not >> executing the query to the external systems (Credit Bureau, Black List had >> a high cost per each query). So then we saved execution time and money. >> >> ****TL;TR**** >> What I am trying to express is that the level, use cases, and user >> history (or whatever it can be the concept about is it use) have a wide >> range of "idempotent" implementations for the potential solutions should be >> evaluated carefully depending on them. >> >> I think that the ID of the "idempotent" key should have enough entropy to >> avoid any collision in the middleware, cache, db or messaging level and the >> recording of it must be at UTC time either if it handles synchronous or >> asynchronous API calls. >> >> ******** >> >> Best regards >> >> Victor >> >> >> El lun, 25 oct 2021 a las 19:39, VICTOR MANUEL ROMERO RODRIGUEZ (< >> victor.rom...@fintecheando.mx>) escribió: >> >>> Hello, >>> >>> I agree that a key must be used on the server side (Consumer) to avoid >>> any duplicate record, most of the time I used to look at the Software >>> Patterns like the Martin Fowler's articles: >>> >>> >>> https://martinfowler.com/articles/patterns-of-distributed-systems/idempotent-receiver.html >>> >>> >>> The pattern described there is more complex than the Saransh's proposal >>> (which is a good starting point), there are many approaches to add this >>> feature (Idempotency) on Apache Fineract, which I think should be flexible >>> enough to handle business rules. >>> >>> Idempotency can be at different levels (even using the POST/PUT), >>> example: >>> >>> At DB level: >>> >>> The POST can be executed to create a new customer and the DB table has >>> as a key or unique the email used in the account creating, then a >>> duplicated entry at DB level should be thrown and the API Rest should send >>> this error. Then this API Rest is Idempotent because at DB level there is a >>> restriction. >>> >>> At API REST per business rule level: >>> >>> The POST for a login should be idempotent because only one session is >>> created at a time per user/customer and in case of X number of failed >>> retries the account must be locked for a period of time or unlocked by a >>> manual/automatic process. Once the session is created, if a POST for login >>> sends a 200 status, then for a period of time the possibility to have >>> another login must be blocked until the current active session expires >>> (expiry time is set by the digital delivery channel owner like Internet or >>> Mobile banking). >>> >>> At API REST per consistency rule level: >>> >>> The POST for doing a payment (third party same bank or interbank >>> transfer) the idempotency key must be used for keeping the track of the >>> transaction status, holding the amount, settlement or reversal of the >>> amount, digital receipt, digital invoice generation, accounting record, >>> etc) in order to avoid duplication of charges in the origin account. Even >>> with confirmation/warning pop ups/windows errors occur. So then the handle >>> of the idempotency key must be handled and implemented in the Front end >>> application, the Server (and the internal consumer). >>> >>> And by the way the transactions must be done at UTC time, not at TZ >>> (another change in the Apache Fineract's tables should be done for the >>> offset time zone recording) . >>> >>> Best regards >>> >>> Victor >>> >>> El sáb, 23 oct 2021 a las 0:05, Saransh Sharma (<sara...@muellners.com>) >>> escribió: >>> >>>> This is a curious problem in computer science and especially in >>>> distributed networks. >>>> >>>> API Idempotency is a major issue for software like Fineract, unless >>>> the API handles drop off and recognize the second call or any n call as >>>> either duplicate or important in any given case, like how stripe does this >>>> by specifying a key from the client end ensures that even in case of >>>> failure the charge does not happen twice! >>>> >>>> This can lead to financial loss in case of real payment taking place! >>>> >>>> I think, as of right now there is no such mechanism in Fineract 1.x >>>> that eliminates such risk. Except, on the validation side, but that’s not >>>> enough. >>>> >>>> To have this implemented we have done some work on this last year , >>>> maybe we can present some code also ! >>>> >>>> I have used some cryptography libs to have the client-side send a >>>> unique key that’s generated by the private key infrastructure and the >>>> client-side has to send these as headers and on the server-side, the >>>> signatures are validated thus making the idempotency in POST and PUT >>>> request. >>>> >>>> The code on the client side >>>> >>>> var keys = { >>>> alice: >>>> '38f93bdda21a5c4a7bae4eb75bb7811cbc3eb627176805c1009ff2099263c6ad', >>>> bob: '09880c962437080d72f72c8c63a69efd65d086c9e7851a87b76373eb6ce9aab5'}; >>>> // GET >>>> for(k in keys) { >>>> var url = 'APIURL'; >>>> var dataToSign = url; >>>> var options = { >>>> url: url, >>>> headers: { >>>> 'x-identity': auth.getPublicKeyFromPrivateKey(keys[k]), >>>> 'x-signature': auth.sign(dataToSign, keys[k]) >>>> } >>>> }; >>>> >>>> >>>> The above code generates these Keys are now passed on each request to >>>> the server, and the server using the middleware in case of Fineract have >>>> to do two things >>>> >>>> 1. VerifySignature >>>> 2. GetSinFromThePublicKey >>>> >>>> I started writing some code for this and quickly realized that it's >>>> actually doable, it requires around 20 hours of work, There are some >>>> dependency that needs to be added on the server side and some quick changes >>>> on the middleware and its done. >>>> >>>> We presented a paper on this last year on Apache Con ! Remember >>>> something about using zero knowledge of proof and password less auth . >>>> >>>> Let me know , I would like to make sure that this gets added in this >>>> upcoming release. >>>> >>>> >>>> >>>> >>>> >>>> On Friday, October 22, 2021, James Dailey <jamespdai...@gmail.com> >>>> wrote: >>>> >>>>> Devs - >>>>> >>>>> Through a conversation with a computer science person, I've learned >>>>> that Fineract should have a clearly articulated approach with regard to >>>>> idempotency. This is particularly true when thinking about how fineract >>>>> interacts with payment systems. >>>>> >>>>> "When building a system to move money, it is paramount that operations >>>>> that move money are idempotent. Failure to do this might result in >>>>> accidentally double charging your customer, or paying a vendor multiple >>>>> times. The risk of this happening is elevated based on the way software is >>>>> typically built today, since developers take advantage of scalable systems >>>>> that process multiple items in parallel. " >>>>> ( >>>>> https://www.moderntreasury.com/journal/why-idempotency-matters-in-payments >>>>> ) >>>>> >>>>> Or this article on Stripe engineering calling for liberal use of >>>>> Idempotency: >>>>> >>>>> "The easiest way to address inconsistencies in distributed state >>>>> caused by failures is to implement server endpoints so that they’re >>>>> idempotent, which means that they can be called any number of times while >>>>> guaranteeing that side effects only occur once." ( >>>>> https://stripe.com/blog/idempotency) >>>>> >>>>> and this article in Apache Camel >>>>> >>>>> https://camel.apache.org/camel-kafka-connector/latest/user-guide/idempotency.html >>>>> >>>>> So, my understanding may be wrong and we've got this covered, or maybe >>>>> we just consider this overkill. Perhaps there is something in the "magic" >>>>> that's developed in the CQRS plus in-memory database queuing and our >>>>> clever >>>>> transaction state management, but... I don't know. >>>>> >>>>> Or perhaps we haven't really had to think about this because we >>>>> haven't had that many large scale deployments and the scalability issues >>>>> associated with multiple parallel calls haven't come to the fore, or >>>>> haven't come back to the list. So, that's the intent of this email. >>>>> >>>>> Do we have a need now for layering on Idempotency checks? What is the >>>>> right approach? Has anyone already implemented this, or decided not to >>>>> for >>>>> very good reasons? What would be the right approach to this? Implement >>>>> where? Time limited to 3 hrs? Etc... >>>>> >>>>> So, this is a call to discuss. I'm far from an expert on this. >>>>> >>>>> >>>>> > > -- > > Saransh Sharma > *Research Partner* > *Muellner Internet Pvt Ltd * > > > ---------------------------------------------------------------------------------------------- > > The idea of separation of me and you is amazing. > > ---------------------------------------------------------------------------------------------- > *Company Website <https://www.muellners.com/> **Company Linkedin > <https://www.linkedin.com/company/muellners> *Company Facebook > <https://www.facebook.com/muellners> > > This mail is governed by Muellner® Internet Private Limited's IT policy. > The information contained in this e-mail and any accompanying documents > may contain information that is confidential or otherwise protected from > disclosure. If you are not the intended recipient of this message, or if > this message has been addressed to you in error, please immediately alert > the sender by reply e-mail and then delete this message, including any > attachments. Any dissemination, distribution or other use of the contents > of this message by anyone other than the intended recipient is strictly > prohibited. All messages sent to and from this e-mail address may be > monitored as permitted by applicable law and regulations to ensure > compliance with our internal policies and to protect our business. E-mails > are not secure and cannot be guaranteed to be error free as they can be > intercepted, amended, lost or destroyed, or contain viruses. You are deemed > to have accepted these risks if you communicate with us by e-mail. >
