GitHub user kaspersorensen opened a pull request:
https://github.com/apache/metamodel/pull/178
A new module for Apache Kafka "querying" using MetaModel
Hi all,
I've been playing around with this idea for a while and decided to try it
out ... I'm curious what you think!
This is a new MM module which allows you to query Apache Kafka. By "query"
I mean: Each topic is represented as a table. And fields in here include
`partition`, `offset`, `key`, `value` and `timestamp`.
When querying, the MM module creates a new unique Kafka consumer. This
works, but I have one worry, which I don't really know how to mitigate:
Consumers are registered in Kafka and cannot be deleted without pulling some
tricks that go all the way into Zookeeper. This worries me because then every
query fired will leave behind a new consumer ID in Zookeeper. If you use this
module a lot then, then it becomes a lot of garbage.
Except for the issue described above, I think this works quite well.
Queries are very fast and I've also added ability for the module to insert
stuff.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kaspersorensen/metamodel kafka
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metamodel/pull/178.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #178
----
commit 5b8176e18bd77e15317c6d7f7c22f7916a2db7c6
Author: Kasper Sørensen <i.am.kasper.sorensen@...>
Date: 2018-01-28T01:45:32Z
Initial draft at a Kafka module for Apache MetaModel
commit c6be1aea314a62fc26e1212248cf9abc9f915bca
Author: Kasper Sørensen <i.am.kasper.sorensen@...>
Date: 2018-01-28T20:41:50Z
Added ability to query by partition and/or offset
commit 26641cc435db46646e5ce5732df09c58781b4ce8
Author: Kasper Sørensen <i.am.kasper.sorensen@...>
Date: 2018-01-28T23:05:43Z
Toggled Kafka integration tests using the properties file
commit 9f4d15e4df90d8555bbb5e0e17aaa3a3b924d3df
Author: Kasper Sørensen <i.am.kasper.sorensen@...>
Date: 2018-01-28T23:27:08Z
Added mocked unit tests for base query and partition-based query
commit 0e7545e74d80985c2b90fff6288b90f8e509ea40
Author: Kasper Sørensen <i.am.kasper.sorensen@...>
Date: 2018-01-28T23:50:04Z
Added (insert-only) UpdateableDataContext capabilities
----
---
