Ok, so basically what we want is different kind of processing when http is used as this feature unwraps the data within cloud event and send the cloud attribute as headers.
El mar, 24 oct 2023, 16:02, Walter Medvedeo <[email protected]> escribió: > Hi Enrique, > in general knative eventing deliver the events to a target service in a > format like this: > > > https://github.com/cloudevents/spec/blob/main/cloudevents/bindings/http-protocol-binding.md#31-binary-content-mode > > See the example below, and what we have is that all extensions that you > have in the cloud event are send as http-headers, like > ce-myExtension1 > ce-myExtension2 > > etc. > > > This is why if we do > > objectMapper.readValue(message.getPayload().toString(), MyClass.class) > > It might happen that some values are not cached up, because all the cloud > event extensions goes as headers and not as part of the > message.getPayload(). > > > HTTP/1.1 200 OK > ce-specversion: 1.0 > ce-type: com.example.someevent > ce-time: 2018-04-05T03:56:24Z > ce-id: 1234-1234-1234 > > ce-myExtension1: my value1 > > ce-myExtension2: my value2 > > ce-source: /mycontext/subcontext > .... further attributes ... > Content-Type: application/json; charset=utf-8 > Content-Length: nnnn > > { > ... application data ... > } > > > > > Cheers, > Walter. > > El mar, 24 oct 2023 a las 13:38, Enrique Gonzalez Martinez (< > [email protected]>) escribió: > > > I think is fair point. From my perspective there is no problem. In the > end > > we just need to duplicate fields in what u call binary format and unwrap > > those changes in the data index to make it work. > > > > Unfortunately the events requires need the context of the event as it is > > big event was replace by raw diff events. > > > > My guess is that knative unwrap the cloud event and send that wrapped > data > > (otherwise) there is no eventpublisher which sends no cloud event afaik.. > > is that correct ? > > > > El mar, 24 oct 2023, 13:25, Walter Medvedeo <[email protected]> > > escribió: > > > > > Hello Guys, > > > > > > I just wanted to raise up a tricky part of the cloud events that I > think > > > it's important to take into account when we introduce modifications on > > the > > > events produced / received by our infrastructure in general. > > > > > > When we send cloud events over the wire, there are basically two modes > > that > > > can be used to send them, "structured" and "binary", all good. In > > general, > > > every time our code send those events, it does by using the > "structured" > > > mode. And, in general, our services like the data-index, and the > > > jobs-service will receive them in "structured" mode too. > > > > > > For instance, if we use kafka, and the process produces an event, that > > > event will be delivered in the "structured" mode to the data-index. > > > > > > BUT, there are some scenarios, where even when we produce these events > > in " > > > structured" mode, we don't have 100% control, on which mode they will > be > > > delivered. That is the case of the knative eventing infrastructure. > When > > we > > > have a installation that relies on knative eventing, very simplified, > we > > > have something like this: > > > > > > Process --> (produce event in "structured" mode) --> knative broker > > > (receives the event) --> knative trigger (delivers the event in > "binary" > > > mode) --> Data Index (receives the event in binary mode) > > > > > > To deliver the originally event created in structured mode, in binary > > mode > > > instead, is something internal to the knative eventing system, and they > > > have good reasons for that. On reason, is for example, that by managing > > the > > > events in the binary mode internally, it's easier to do > filtering/routing > > > operations, based on the ce-xxx headers that are used by the binary > mode. > > > > > > Why I am commenting this? > > > > > > As a side effect of the "binary" mode delivery, we must warrantee that > > our > > > services, like Data Index, and Jobs Service a prepared to understand > > those > > > events in both formats. > > > > > > Why? well, because if our architecture is deployed by using the knative > > > eventing infrastructure (instead of kafka), those events will arrive in > > > binary mode. > > > > > > Not sure if I could explain the "problem" clearly, but I'm happy to do > a > > > demo, or whatever of a full setup in the knative system, and show the > > > involved components to ensure the events delivery, etc, np. > > > > > > Now, going the concrete situation, I think this last change in the > events > > > reception at the data-index: > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1881, is > > > introducing an issue when we are in knative eventing system. (and we > > > receive the events in the binary format) > > > > > > Before, far from perfect there was kind of "tricky" processing that > tried > > > to couple with both formats, and more or less worked. Now after the > > change, > > > I can see some errors when we are in knative. > > > > > > And in parallel, we had an open PR > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1889, with > more > > > improvements in particular on the treatment on this events to improve > > this > > > "tricky" code I was mentioning. > > > > > > So we had a kind of too many changes at the same piece of the pie, > > > basically. I think > > > https://github.com/apache/incubator-kie-kogito-apps/pull/1889 should > > > probably have been merged first since it provided an improvement on the > > > binary/structured, and thus, cloud be used for the processing of the > new > > > events defined in the other PR. > > > > > > To be clear, it's no a complain, but I wanted to introduce the topic > > > because I think it's important to keep in mind that both "binary" and > > > "structured" mode for future changes. > > > > > > I'll now try to resolve all the merge conflicts on the PR and see how > to > > be > > > sure both modes continue working. > > > > > >
