Hi Andrew, This is indeed a tricky case since Flink doesn't provide non-shaded JAR for flink-json. One hacky solution in my mind is like:
1. Create a module let's say "wikimedia-event-utilities-shaded" that relocates Jackson in the same way and uses the same Jackson version as flink-shaded-jackson 2. Deploy the module to a local or remote Maven repository 3. Let your custom format depend on the "wikimedia-event-utilities-shaded" module, then all Jackson dependencies are relocated in the same way. Another solution is that you can serialize then deserialize the "different" ObjectNode to do the conversion but this sacrifices the performance. Hope this could be helpful! Best regards, Qingsheng On Thu, Jun 9, 2022 at 8:29 PM Andrew Otto <o...@wikimedia.org> wrote: > > Hi all, > > I'm working on an integration project trying to write some library code that > will allow us at the Wikimedia Foundation to use Flink with our 'Event > Platform'. Specifically, I'm trying to write a reusable step near the end of > a pipeline that will ensure our JSON events satisfy some criteria before > producing them to Kafka. Details here. > > I'm experimenting with writing my own custom format to do this. But all I > really need to do is override JsonRowDataSerializationSchema's serialize > method and augment and validate the ObjectNode before it is serialized to > byte[]. > > I'm running into an issue where the ObjectNode that is used by Flink here is > the shaded one: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode, > whereas the WMF code I want to use to augment the ObjectNode is using a > regular non shaded one. I can't pass the shaded ObjectNode instance to a > function that takes a non shaded one, and I can't cast the shaded ObjectNode > to non shaded either. > > My Q is: is there a way to extend Flink APIs that use shaded dependencies? I > suppose I could copy/paste the whole of the "json" format code that I need > into my project and just make it my own, but this feels quite obnoxious. > > Thank you! > -Andrew Otto > Wikimedia Foundation > >
