Hello Guys, I just wanted to raise up a tricky part of the cloud events that I think it's important to take into account when we introduce modifications on the events produced / received by our infrastructure in general.
When we send cloud events over the wire, there are basically two modes that can be used to send them, "structured" and "binary", all good. In general, every time our code send those events, it does by using the "structured" mode. And, in general, our services like the data-index, and the jobs-service will receive them in "structured" mode too. For instance, if we use kafka, and the process produces an event, that event will be delivered in the "structured" mode to the data-index. BUT, there are some scenarios, where even when we produce these events in " structured" mode, we don't have 100% control, on which mode they will be delivered. That is the case of the knative eventing infrastructure. When we have a installation that relies on knative eventing, very simplified, we have something like this: Process --> (produce event in "structured" mode) --> knative broker (receives the event) --> knative trigger (delivers the event in "binary" mode) --> Data Index (receives the event in binary mode) To deliver the originally event created in structured mode, in binary mode instead, is something internal to the knative eventing system, and they have good reasons for that. On reason, is for example, that by managing the events in the binary mode internally, it's easier to do filtering/routing operations, based on the ce-xxx headers that are used by the binary mode. Why I am commenting this? As a side effect of the "binary" mode delivery, we must warrantee that our services, like Data Index, and Jobs Service a prepared to understand those events in both formats. Why? well, because if our architecture is deployed by using the knative eventing infrastructure (instead of kafka), those events will arrive in binary mode. Not sure if I could explain the "problem" clearly, but I'm happy to do a demo, or whatever of a full setup in the knative system, and show the involved components to ensure the events delivery, etc, np. Now, going the concrete situation, I think this last change in the events reception at the data-index: https://github.com/apache/incubator-kie-kogito-apps/pull/1881, is introducing an issue when we are in knative eventing system. (and we receive the events in the binary format) Before, far from perfect there was kind of "tricky" processing that tried to couple with both formats, and more or less worked. Now after the change, I can see some errors when we are in knative. And in parallel, we had an open PR https://github.com/apache/incubator-kie-kogito-apps/pull/1889, with more improvements in particular on the treatment on this events to improve this "tricky" code I was mentioning. So we had a kind of too many changes at the same piece of the pie, basically. I think https://github.com/apache/incubator-kie-kogito-apps/pull/1889 should probably have been merged first since it provided an improvement on the binary/structured, and thus, cloud be used for the processing of the new events defined in the other PR. To be clear, it's no a complain, but I wanted to introduce the topic because I think it's important to keep in mind that both "binary" and "structured" mode for future changes. I'll now try to resolve all the merge conflicts on the PR and see how to be sure both modes continue working.
