#general
@waqasdilawardaha: @waqasdilawardaha has joined the channel
@maitreyi.kv: @maitreyi.kv has joined the channel
@xiangfu0: In case you are using MacOs Monterey. They introduce a new feature of airplay receiver which uses port 5000 and 7000. However 7000 is used in pinot server query port by default for Quickstart. The resolution is to disable this feature in System Preference -> Sharing -> uncheck AirPlay Receiver
@kchavda: Hello all, I'm looking at this
@kchavda: Hi @npawar, any additional documentation on this?
@npawar: i believe that one is a config common across all minion tasks. minion documentation in general is pretty low right now, so might have bee missed.
@npawar: for now, this example that you saw is the only reference i see
@npawar: we can add some
@kchavda: Thanks Neha. Do you know if the `schedule` is required in table config or the controller conf change handles that?
@npawar: it is certainly not required. Just the controller config `controller.task.frequencyInSeconds` is sufficient. However, this controller config is a setting across the cluster. Versus the `schedule` lets you override it per table. So if you need finer control than the controller setting, this is the one
@kchavda: Thank you, that helps!
@diogo.baeder: Hi folks! A question about `dateTimeFieldSpecs`: Is the `TIMESTAMP` type something that will stay in Pinot? Because @kchavda gave me a nice hint about that but we don't see that in the docs, so I'm unsure whether I can safely use it and know that it won't get removed on future Pinot versions
@npawar: it is here to stay. It’s part of the datatypes in this page
@diogo.baeder: Ah, nice! Thanks!
@diogo.baeder: @npawar strange that it's not defined here though:
@npawar: good catch :slightly_smiling_face: will fix it, and maybe also add some examples
@diogo.baeder: Awesome, thanks a lot! :heart:
@diogo.baeder: @npawar what format should be used though when publishing values through Kafka? I'm using, for example, `2020-04-04 00:00:00 UTC` , as a string, and it's not saving on Pinot
@npawar: should be ``` "dateTimeFieldSpecs": [ { "name": "time_col_name", "dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyyMMdd HH:mm:ss z", "granularity": "1:MILLISECONDS" } ]```
@diogo.baeder: Oh... I thought the dataType would be `TIMESTAMP`, am I wrong?
@npawar: @jackie.jxt does this `2020-04-04 00:00:00 UTC` string also work for `TIMESTAMP`?
@jackie.jxt: No, you need to remove `UTC`
@npawar: so it’s `yyyyMMdd HH:mm:ss` and `epoch millis` that can use `TIMESTAMP` ?
@jackie.jxt: Yes
@jackie.jxt: Standard `TIMESTAMP` format would be `yyyy-MM-dd HH:mm:ss.SSS`
@diogo.baeder: Oh, just `2020-04-04 00:00:00` then? And in terms of performance, how does that compare to using `SIMPLE_DATE_FORMAT` with a date-time format?
@diogo.baeder: I mean, comparing the dataTypes, it would be `TIMESTAMP` vs `STRING`, would there be a significant performance difference?
@jackie.jxt: `TIMESTAMP` is stored as long (millis since epoch), so it has better performance than string
@diogo.baeder: Ah, that's very relevant! :slightly_smiling_face:
@diogo.baeder: You guys rock so hard! Thank you very very much! :heart:
@jackie.jxt: The performance improvement is depending on the queries, but long should always be faster than string
@jackie.jxt: Pleasure to help
@diogo.baeder: Indeed, cause it's numeric and all that... awesome!
#random
@waqasdilawardaha: @waqasdilawardaha has joined the channel
@maitreyi.kv: @maitreyi.kv has joined the channel
@farnood: @farnood has left the channel
#troubleshooting
@waqasdilawardaha: @waqasdilawardaha has joined the channel
@maitreyi.kv: @maitreyi.kv has joined the channel
@kchavda: Hi all, I have a realtime table which completes loading all data from source ( using debezium > kafka). I compared the kafka connect logs and total records from snapshot match total records in Pinot table, however a few minutes later there are less records in Pinot. Nothing in the pinot-controller/server/broker logs. Anyone else experience this?
@ssubrama: Do you think the retention manager kicked in , removing old data? You may want to check the value of time column in your data. When you say "nothing" in the logs, I suppose you mean no errors/warnings?
@kchavda: Hi Subbu, yep, by "nothing" I meant errors/warning. Thanks for the suggestion, let me check that. If that's the case, increasing retentionTimeValue should work, correct?
@kchavda: It seems to be random. Compared Pinot table to source and I see missing primary key values in Pinot table. Where does the retention manager dump the removed data?
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
