#general
@bhat.shruti: @bhat.shruti has joined the channel
@ysuo: Hi, is it supported to use JSONEXTRACTSCALAR function in the transformConfigs?
@kharekartik: Hi No that is not supported but you can use `jsonPath` function which does a similar thing. If you require a specific result data type you can use its variations such as `jsonPathLong`, `jsonPathDouble` etc.
@ysuo: @kharekartik Thanks. Another question, does jsonPath supports regular _expression_?
@kharekartik: Not sure. You can test out here -
@ysuo: Got it. Thanks, Kartik.
@satyam.raj: Hey guys, Is there a way to connect Pinot to a secure Kafka cluster using Jaas configuration?
@mayanks: @slack1 might have some insights here.
@slack1: Hi Satyam, yes. I remember @patidar.rahul8392 was settings up something similar some time ago. If you look up his posts via search you’ll find a number of actual configs
@francois: Hi getting deeper in my
@mark.needham: at the moment you can't replace a real-time segment, but I think someone might be working on it. @mayanks will have more info.
@francois: I can see it’s replacing my REALTIME to OFFLINE for the table name but don’t know where :confused:
@mark.needham: at the moment you can only replace offline segments, real-time ones won't work
@mark.needham: not sure why it'd be renaming your table
@mark.needham: i thought it threw an error indicating that you can't replace real-time segments
@francois: Not really it complains about segment non existing (Still in realTime Table )`org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 410 (Gone) with reason: "Cannot refresh non-existing segment, aborted uploading segment: actions__0__0__20220405T0951Z of table: actions_OFFLINE" while sending request:
@francois: ok I will switch my logic to get the segment in OFFLINE table an then purge them :wink:
@mark.needham: There's a simple example showing one way to do that here -->
@francois: Yes already use it :wink:
@mark.needham: ah cool
@mrpringle: Any plans for pinot to support 'RealtimeToOfflineSegementsTask' for upsert enabled tables. Use case is this: 1. Normal queries; key contains a business date; only want to see the last value for a day (as per current upsert config works well) 2. Detail / as of queries; want to see record up to a given timestamp for a given business date (skipUpsert and LASTWITHTIME( col, asOfTime ). e.g. compare 11am yesterday v today 3. Historical want to move data beyond a week to OFFLINE 4. Historical want to drop data points e.g. milli seconds -> hours per day. Like reducing data resolution over time through some data cleaning job.
@luisfernandez: number 3 and 4 i believe you can do with the current RTToOffline job
@datanarch01: @datanarch01 has joined the channel
@ciro.zar: @ciro.zar has joined the channel
@weixiang.sun: It seems that we have health API for controller. Do we have health API for table and broker?
@mayanks: There should be one for broker. Swagger should list all the apis, you can check if there’s one for table there.
@evan.galpin: hey folks, does anyone have experience with “zombie” tables? I attempted to add a tableSpec but was missing a field in the config. This resulted in a null pointer exception when Pinot processed the config. As a result, the table doesn’t exist in the UI or by querying the list API, but I can’t create a table with the same name due to receiving an error that “table already exists”. Presumably the spec reached ZK or something, but no where else? What can be done to remove the zombie table?
@mayanks: Hmm this seems like a bug to fix. Could you please file a GH issue?
@bagi.priyank: i ended up in a similar situation earlier this week. pinot cluster couldn't route to kafka cluster so when i submitted table config to create the table, the table was never created externally or visible via UI. infact swagger api to get all tables didn't return it in the result. however if i tried creating the table again, cluster would respond back with an error saying that the table already exists.
@npawar: I'm assuming you already tried to invoke DELETE /tables/tablename Api but that didn't work? I've had this happen once before, but invoking a delete cleared out the zombie states
@bagi.priyank: Yep delete didn't work.
#random
@bhat.shruti: @bhat.shruti has joined the channel
@datanarch01: @datanarch01 has joined the channel
@ciro.zar: @ciro.zar has joined the channel
#troubleshooting
@bhat.shruti: @bhat.shruti has joined the channel
@datanarch01: @datanarch01 has joined the channel
@ciro.zar: @ciro.zar has joined the channel
@facundo.bianco: Hi All, I'm trying to use Pinot with AWS S3 integration but I got this error message (whole error message will be attached): ```2022/04/08 19:30:00.576 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 35.248 since launch java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:526) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... 11 more Caused by: java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... 11 more``` I configured Pinot's Kubernetes as ``` controller: data: dir:
@mayanks: The issue seems like PinotFs has not been initialized. Also, any reason to not use the latest release 0.10.0?
@luisfernandez: anyone familiar with this error while executing the QuickStart from IDE ```2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main] Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108 -brokerPort 8000 -queryType sql -query select count(*) from baseballStats limit 1 java.net.NoRouteToHostException: No route to host (Host unreachable)```
@mayanks: Is broker up and running at port 8000 of 0.0.26.108?
#getting-started
@datanarch01: @datanarch01 has joined the channel
@ciro.zar: @ciro.zar has joined the channel
#releases
@siddhartha.varma: @siddhartha.varma has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
