I think the read path could be feasible with a List-style processor that had the smarts to process the Delta commit log, which is simple and doesn’t necessarily require Spark.
The write path is significantly more complex since now you have to make changes to that commit log potentially while Spark is too. I haven’t followed closely enough to know if this is exposed in the Delta Lake APIs in a way that could be used outside Spark. To get a sense of the lift for the read, there’s a _delta_log dir that has a bunch of JSON files and roll up in Parquet. If you look at those you can get the set of Parquet files that belong to the current possible world. Provided someone hasn’t just done a vacuum, you know what can be transferred out in the List. The need for the write path is arguably weak. NiFi can deliver data to cloud object stores or message queues and Structured Streaming can handle that data flow and do the inserts into Delta tables. -joey > On Jun 15, 2022, at 11:27 AM, Mike Thomsen <[email protected]> wrote: > > I looked into the feasibility of doing a direct integration and found > it to not be a really good fit because the other "integrations" I > found with Delta involved running Spark locally in the background > IIRC. Maybe that's changed, but the impression I got was that Delta is > more or less tied to how Spark works and doesn't fit in well with > different systems like NiFi (or Hive, as I think Hive was their first > "integration" attempt). > >> On Wed, Jun 15, 2022 at 12:42 PM scott <[email protected]> wrote: >> >> Hi community, >> I was wondering if there is an official effort to add delta table format ( >> delta.io ) into NiFi? I saw there were some discussions a few years back, >> but not sure what came of that. Not looking for a workaround where I have to >> use spark to do the actual read/write into the delta table, just pure NiFi( >> Java ) solution. >> >> Thanks, >> Scott
