Apache Pinot Daily Email Digest (2020-12-17)

Pinot Slack Email Digest Thu, 17 Dec 2020 18:00:44 -0800

#general

@slackbot: This message was deleted.
@mayanks: Hey, you can provide this feedback using the LinkedIn app.
@chinmay.cerebro: Thanks to everyone who voted on the 2021 roadmap poll (50+ participants) ! We're keeping it open for 1 more week. You can still vote if you haven't here:

#feat-text-search

@taylorboren86: @taylorboren86 has joined the channel

#troubleshooting

@elon.azoulay: Hi, we wanted to know if changing kafka consumer properties in the realtime config requires restarting the servers so that the consumer can pick up the new properties. Anyone familiar with this?
@npawar: for an existing consumer actively consuming, the changes won’t get applied. But for any new consumers created, it will get automatically applied. With every new segment, the table config is fetched again to make the consumer
@npawar: so for existing consumers, restart is required
@elon.azoulay: thanks @npawar!

#pinot-dev

@ken: I see that pinot-spi’s pom has a dependency on log4j-slf4j-impl. I don’t think this is right - it should only depend on slf4j-api. Because of this dependency, the pinot-java-client’s dependency on pinot-spi pulls in the logging implementation, which is not what you want because it means an external project using the client often needs to exclude those logging jars.
@g.kishore: Agree...
@ken: OK, files
@g.kishore: Thanks... should we do this in two phases? First make the dependency provided and then remove it?
@ken: I think for pinot-spi we can just change the dependency to be on slf4j-api. Everyone who uses pinot-spi should have their own logging implementation.
@g.kishore: Okay... that’s simple
@taylorboren86: @taylorboren86 has joined the channel

#feat-pravega-connector

@fpj: @g.kishore @npawar On the Kinesis integration document, I might be able to produce a Pravega connector with the batch API of Pravega rather than the the event stream API. With the batch API, I'm able to get a list of segments within the bounds of stream cuts (a reference to a position in the stream across segments): I have one question about `PartitionGroupMetadata` in the document, and I thought the description was too long to do it as a comment in the document, but if you prefer like that, then I can raise it in the document directly. `PartitionGroupMetadata` has a `groupID` attribute, which I don't understand how it maps to a group of shards/segments. For instance, say that I'm able to list segments via the iterator that `getSegments` returns: ```StreamSegmentsIterator getSegments(Stream stream, StreamCut fromStreamCut, StreamCut toStreamCut); ``` How do Pinot map those segments to a partition group? Also, such a list of segments is flat, meaning that it is not preserving a predecessor-successor order of segments. When you have scaling, there is a natural order of segments that needs to be followed to avoid breaking per-key order. The event stream API in Pravega makes such a guarantee, but the batch API doesn't. How do you do it for Kinesis?
@g.kishore: its upto the implementation on how to map segments to partition group, by default its 1-1 mapping which is the only things we support today but we this redesign we are trying to support grouping multiple stream shards into one segment in pinot
@fpj: > its upto the implementation on how to map segments to partition group where would you implement this logic? in the implementation of the `PartitionGroupMetadata` interface? also, could you please comment on this: > such a list of segments is flat, meaning that it is not preserving a predecessor-successor order of segments. When you have scaling, there is a natural order of segments that needs to be followed to avoid breaking per-key order. The event stream API in Pravega makes such a guarantee, but the batch API doesn't. How do you do it for Kinesis?
@g.kishore: where would you implement this logic? in the implementation of the `PartitionGroupMetadata` interface? • yes
@g.kishore: > such a list of segments is flat, meaning that it is not preserving a predecessor-successor order of segments. When you have scaling, there is a natural order of segments that needs to be followed to avoid breaking per-key order. The event stream API in Pravega makes such a guarantee, but the batch API doesn't. How do you do it for Kinesis? we provide the current segments to the interface, so the implementation should not return the new segments until the old segments have reached EOF
@fpj: interesting, that's essentially the logic that the event stream reader implements, which I can't use because it requires direct access to the segments. it feels like pravega is being punished for doing the right thing because others don't do it...
@fpj: I have to reimplement that logic as part of the pinot spi implementation to be able to preserve the predecessor-successor order, that's what I'm hearing, is that right?
@g.kishore: > interesting, that's essentially the logic that the event stream reader implements, which I can't use because it requires direct access to the segments. it feels like pravega is being punished for doing the right thing because others don't do it... :disappointed: I think there are challenges with both models. Its basically comes down to exposing the right API
@g.kishore: in some cases, high level abstraction is good but for systems like Pinot, KSQLdb etc low level API is better
@fpj: direct access to segments/partitions give an idea of simplicity, but managing it isn't simple. given a list of segments, the implementation needs to figure out the order of segments and if pinot asks for a replay for a single segment, then the interface implementation now needs to figure out the history of successors to be able to serve the right data. and, as the stream scales up or down, there needs to be some coordination across groups otherwise groups might end up with an unbalanced load. implementing this logic is non-trivial and from this discussion it is necessary, so I'm not convinced that it is a good idea to expect this logic to be implemented as part of the pinot spi rather than relying on the streaming system to do it. the upstream streaming system should have this information and do a better job at the coordination. I'm unclear right now as how it affects the other requirements you had for which you decided to pursue this low-level approach.
@ssubrama: A dumb question here: Is a pravega `segment` same as a kinesis `shard` (or, kafka `partition`)? Just so I get the terms right
@fpj: A Pravega segment is closer to a Kinesis shard in that it can be sealed (closed) and rescaled. A segment in Pravega stores a sequence of bytes, so it is possible to write and read bytes without events or messages. A Pravega segment is as sequence of events when using the event stream API.
@fpj: For the sake of progress and give that conversations are ephemeral on free slack, I'm wondering what the best way to proceed is at this point. I feel that the current API is making it difficult to use Pravega features and if we follow the direction we are discussing here, the one of using the batch API, then it will be underutilized and Pinot won't be able to benefit as much from it. I'm particularly concerned about the direction of dealing with segments/shards and order of the same at the spi level. In general, Pravega is open to suggestions of API changes and I'd love to see Pravega fully utilized with Pinot. Let me know how you want to proceed.
@g.kishore: i have been thinking about this a lot, cant come up with a solution that will work without making lot of changes in Pinot
@g.kishore: I was going back to the other suggestion you had made
@g.kishore: one consumer consumer from pravega and then writes to Pinot
@g.kishore: we plan to add a write api soon
@g.kishore: may be we should integrate with Pravega once we have write API?

#pinot-docs