Apache Pinot Daily Email Digest (2020-12-22)

Pinot Slack Email Digest Tue, 22 Dec 2020 18:00:27 -0800

#general

@ketan_popat: @ketan_popat has joined the channel
@sarthak.ag: @sarthak.ag has joined the channel
@hello: @hello has joined the channel
@huodon: @huodon has joined the channel
@sudhir: @sudhir has joined the channel

#random

#troubleshooting

@ketan_popat: @ketan_popat has joined the channel
@sarthak.ag: @sarthak.ag has joined the channel
@yash.agarwal: Hi, How would I go about creating a custom filesystem plugin ?
@ken: Hi @yash.agarwal - you should be able to fork the pinot repo from , then add your own plugin directory at `<repo>/pinot-plugins/pinot-file-system/<your plugin>`. It would automatically get picked up if it’s located in the `plugins/` directory inside of a Pinot distribution.
@ken: You’d add your plugin module name to the list of modules in the `…/pinot-file-system/pom.xml` file so it gets built. I believe it’s also possible to code this up completely separate from the Pinot project, but I haven’t tried that (ie just a regular project with appropriate dependencies on Pinot jars)
@ken: And at the code level, you’re implementing a class that extends `PinotFS`.
@yash.agarwal: yeah. I was hoping if it could be a separate code. We already our working on the way you mentioned. Thanks!
@yash.agarwal: Btw ```-Dplugins.dir=/opt/pinot/plugins``` we provide this path in our docker image for picking plugins but I get exceptions when loading plugins.
@ken: @yash.agarwal - what exceptions? And for regular plugins, or your custom plugin?
@hello: @hello has joined the channel
@huodon: @huodon has joined the channel
@sudhir: @sudhir has joined the channel

#pinot-dev

@himanshu.rathore: @himanshu.rathore has joined the channel
@ken: I’m working on , and had some questions about what the final jar should contain. The pinot-integration-tests-0.7.0-SNAPSHOT.jar is empty, and the pinot-integration-tests-0.7.0-SNAPSHOT-tests.jar has both the code that would be useful for 3rd party projects wanting to test against Pinot (BaseClusterIntegrationTest and associated classes) AND all of the integration test code and resources used by Pinot internally. I’m wondering if it makes sense to try to include only the externally useful code in the -tests jar, or move that code into `src/main/java` so that it winds up in the pinot-integration-tests-0.7.0-SNAPSHOT.jar. Thoughts?
@ken: I’m leaning towards the first option, as being more consistent (put the externally useful code in the tests.jar file), even though (a) the name is somewhat confusing (blah-tests-version-tests.jar) and (b) it’s more fragile, as I’ll need to select specific packages from the src/test/java directory to include.
@ken: Though this makes it much harder to build a shaded jar with just the dependencies needed for the externally useful code.

#presto-pinot-streaming

@himanshu.rathore: @himanshu.rathore has joined the channel

#presto-pinot-connector