#general
@avinashnayak: @avinashnayak has joined the channel
@girishbhat.m7: @girishbhat.m7 has joined the channel
@calvin.mwenda: @calvin.mwenda has joined the channel
@karinwolok1: :partying_face: Congrats to the new #ApachePinot committers!!! :tada: @chinmay.cerebro & @yupeng Thank you for all of your contributions and continuous effort to give back to the community!:wine_glass: :clap1: The community thanks you! :clap1:
@kundu.abhishek: @kundu.abhishek has joined the channel
@karinwolok1: :tada: Hooray! :tada: We've surpassed 1,000 members on the Pinot Slack :wine_glass: :slack: This is a great milestone and we expect much more growth in the future! :chart_with_upwards_trend: :pray: Thank you to all our PMC's, committers, contributors, and community members! :pray: You are all vital in helping move the needle and industry-standards in fast analytics! :bar_chart: :chart_with_downwards_trend: Art credit: Apache Pinot PMC & Committer @npawar
@m.h.dugas: @m.h.dugas has joined the channel
#random
@avinashnayak: @avinashnayak has joined the channel
@girishbhat.m7: @girishbhat.m7 has joined the channel
@calvin.mwenda: @calvin.mwenda has joined the channel
@kundu.abhishek: @kundu.abhishek has joined the channel
@srini: hey folks! :wave: Thanks to folks like @kenny Pinot and Superset have a great romance :sunny: We just published our manifesto on open source BI, :pray: would appreciate any social media boosts!
@m.h.dugas: @m.h.dugas has joined the channel
#troubleshooting
@avinashnayak: @avinashnayak has joined the channel
@girishbhat.m7: @girishbhat.m7 has joined the channel
@calvin.mwenda: @calvin.mwenda has joined the channel
@kundu.abhishek: @kundu.abhishek has joined the channel
@m.h.dugas: @m.h.dugas has joined the channel
#discuss-validation
@chinmay.cerebro: @ssubrama @jackie.jxt we're using this doc to track all remaining validations:
@chinmay.cerebro: we can also use those 2 categories for enforcing how strict the validations are
@chinmay.cerebro: one PR in progress related to this is :
@chinmay.cerebro: which is trying to enforce a schema on table config - this will be the most stringent check and will probably belong in the SUGGESTED cateogry
#segment-write-api
@npawar: kishore was suggesting we change the `init(PinotConfiguration conf)` to `init(URI controllerURI, String tableNameWithType)` so that the API is self descriptive. Any concerns about that @yupeng @fx19880617 ?
@yupeng: are those params sufficient?
@npawar: everything we want should ideally come from the table config
@yupeng: do we plan a namespace concept in the future?
@yupeng: btw, there are a number of configs i planned in the flink connector
@yupeng:
@yupeng: like parallelism and flush condition etc
@npawar: ```The SegmentWriter has an internal buffer with a configured threshold for flushing``` is this needed? the SegmentWriter need not have a threshold for flushing rt? The Sink will invoke flush
@yupeng: i see
@yupeng: that’s one option
@yupeng: i thought the writer will also have its own option
@yupeng: given it shall have some config for buffer size etc
@npawar: hmm, sure we can enhance it to do that.
@npawar: what do you mean by namespace concept?
@yupeng: tables from different tenants can have same name
@npawar: hmm, probably not
@yupeng: we probably want to relax this constraint in the future
@yupeng: it’s common to have db.schema.table hierachy
@npawar: in that case, we can make `init(URI controllerURI, TableConfig config)`
@yupeng: whats TableConfig?
@npawar: the pinot table config
@yupeng: ok. so the sink first fetches the config from controller, and then call init?
@npawar: yes
@yupeng: i see. i’m ok. just to check what will uniquely identify a table in future
@yupeng: btw, somehow i feel the init will be `init(controllerURI, tableName, (optional) tenantName, writerConfig)`
@npawar: tenantName is part of the tableConfig. So if we use tableConfig, we dont need to provide tableName and tenantName
@yupeng: thats one point
@yupeng: another is `writerConfig`
@npawar: Sure, lets keep writer config in the init. writerConfig should also be part of ingestionConfig in the tableConfig. And we can provide way to override `init(controllerURI, TableConfig config, SegmentWriterConfig override)`
@yupeng: hmm really?
@yupeng: can you give me some writerConfig example?
@yupeng: i thought writerConfig is a client-side config
@yupeng: sink configures based on the client-side resource etc
@npawar: for example, flushThreshold should be in the table config rt?
@yupeng: i see
@yupeng: and sink can override it?
@yupeng: then it sounds good
@npawar: yes. For instance, if you have a long running setup, you can have flushThreshold, segmentName generator config in the tableConfig. But now lets say you want to backfill 1 day. Then, you would want to override the segmentName generator config, and give specific name
@yupeng: btw, could you check if this fits in the existing spark configs?
@yupeng: the example makes sense
@npawar: btw did you get a chance to try out the branch?
@yupeng: not yet
@yupeng: spent some time setting up the env
@yupeng: also, the cross project dependency is a bit tricky
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
