Shawn, There are also Redis and Couchbase distributed map cache clients already in NiFi. I don't see any Jiras or PRs related to a DynamoDB or JDBC ones. I thought about making ones for JDBC, Hazelcast and/or Nitrite [1] (with or without a DataGate server), but never got around to it. I think DynamoDB and JDBC implementations would be helpful, the latter could support DynamoDB in the meantime using the Simba JDBC driver [2].
While thinking about the JDBC one, I figured it might be nice to be able to cache the table locally for X amount of time or N number of entries, in case you pre-populate the cache and are just reading it with the client. Any write (from the NiFi client) would invalidate the cache and the table would be re-fetched on the next read operation. I did something similar for the DatabaseRecordLookupService, but that's a read-only service so I didn't have to worry about writes, I was just trying to improve performance where possible. Regards, Matt [1] https://www.dizitart.org/nitrite-database.html [2] https://www.simba.com/drivers/dynamodb-odbc-jdbc/ On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks <[email protected]> wrote: > > Has anyone already done some work on adding new services for distribute map > cache? I’m looking at moving to aws and I really don’t want to have to run > emr just for hbase. I’ve been thinking about starting on either a DynamoDB or > simple jdbc implementation. > > Thanks > Shawn > > Sent from my iPhone
