#general
@weili99: @weili99 has joined the channel
@weili99: :wave: Hi everyone!
@jvfsmrarki: @jvfsmrarki has joined the channel
@jvfsmrarki: Hi :)
@sevvy_yusuf: @sevvy_yusuf has joined the channel
@richard892: I posted a blog post about Pinot's new range index here
@kevin.dethelot: @kevin.dethelot has joined the channel
#random
@weili99: @weili99 has joined the channel
@jvfsmrarki: @jvfsmrarki has joined the channel
@sevvy_yusuf: @sevvy_yusuf has joined the channel
@kevin.dethelot: @kevin.dethelot has joined the channel
#troubleshooting
@weili99: @weili99 has joined the channel
@francois: Hi I’m struggling to get a JSON col in a string way to apply a JSONIndex on it. I’ve tried several things and I keep having a null value in the col :disappointed: Schema extract `{` `"name": "fulldata",` `"dataType": "STRING",` `"maxLength": 2147483647` `}` Table config extract `{` `"columnName": "fulldata",` `"transformFunction": "JSONFORMAT(meta)"` `}`
@mark.needham:
@mark.needham:
@francois: I’ve start to work based on these examples but no différence on what I tried. I’m looking to extact a full bloc a string
@mark.needham: so on your table extract config you reference a field in your source data called 'meta' - does that field exist?
@francois: yes of course my schema for my json event has at root level the meta key
@mark.needham: ok. I mean the table config/schema look ok to me
@mark.needham: at least the extracts
@francois: I keep have null values in the query console but when filtering on non null value I’ve got stuff back from the query (with null ) it’s strange :smile:
@francois: The only thing I see it’s that my meta field has complex type with neasted structure but no error anywere :confused:
@mark.needham: so does it load the data correctly for some rows?
@francois: nope. But I’ve tried to simplify my things and I’ve maybe found the good way to extract my cols
@francois: The field I’m looking for is in meta.relation. I was able to get meta as json and by applying another function to the meta i’m now able to access my relation value value :slightly_smiling_face: I was trying to do it in one transformation
@g.kishore: Can you share Sample input row?
@francois: Yes sure `{` `"meta": {` `"expectedValidator": [],` `"relation": [` `{` `"id": "2",` `"firstName": "Jules",` `"lastName": "HUGHUE"` `}` `],` `"expectedValidatorsForReject": [],` `"isEnabled": false,` `"version": "somevers",` `"isLogged": true,` `"language": "fr"` `},` `"data": {` `"id": "4",` `"type": "tiypes",` `"attributes": {` `"term": "2017-02",` `"creationDate": "2017-02-01T00:00:00+0100",` `"updateDate": "2017-02-01T00:00:00+0100",` `"informationComments": "someRandomtext",` `"closed": false,` `"state": "validated",` `"paid": true,` `"advance": 0,` `"actualExpenses": [],` `"fixedExpenses": [` `{` `<List of elems>` `}` `]` `}` `}` `}`
@francois: And I was looking to extract in one transformation relation from the meta bloc as a JSONSTRING
@g.kishore: that should have worked
@g.kishore: so when you do `select fulldata from T` you get all nulls?
@francois: yes so i’ve added somthing like this in my transormationsSpec : `{` `"columnName": "meta_str",` `"transformFunction": "JSONFORMAT(meta)"` `},` `{` `"columnName": "relation",` `"transformFunction": "JSONPATHARRAY(meta_str,'$.relation')"` `}`
@npawar: Does it work without the second transform function? I think the relation one is incorrect
@francois: The 1 plus 2 works as I expect. But I was expecting to get something like 1 and 2 in single line. It was not the the case :confused:
@jvfsmrarki: @jvfsmrarki has joined the channel
@sevvy_yusuf: @sevvy_yusuf has joined the channel
@sevvy_yusuf: Hi team :wave: I'm writing a custom Trino connector to read from our Pinot cluster and I'm running into an issue when trying to use `LASTWITHTIME` . The first time I run the query (`SELECT * FROM catalog.schema."SELECT LASTWITHTIME(dataColumnName, timeColumnName, 'STRING') FROM tableName WHERE column = 'someColumnValue' LIMIT 10";`) I get the following error: ```java.lang.ExceptionInInitializerError at org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106)``` Any consecutive runs of the query produce the following error: ```java.lang.NoClassDefFoundError: Could not initialize class org.apache.pinot.common.function.FunctionRegistry at org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106)``` Other queries, including dynamic queries work okay. I am using Pinot version 0.9.3. Has anyone come across this before? Any help would be appreciated, many thanks in advance!
@mayanks: Is that a nested query? Pinot doesn’t support nested queries at the moment.
@mayanks: Oh sorry didn’t realize you are using Trino connector
@sevvy_yusuf: It's a dynamic query where we're pushing down the whole query to Pinot. For example this works: ```SELECT * FROM catalog.schema."SELECT columnName FROM table LIMIT 1";```
@richard892: the class isn't loading, which is interesting
@richard892: this will mean one of two things: • non-atomic classloader attempts to load same class twice (I highly doubt this can affect pinot) • static initializer throws during intialization
@sevvy_yusuf: Hi @richard892 thanks for replying! I don't see any errors in the logs during initialization or running other queries, it seems to only throw when I run that particular query
@mayanks: Seems like the inner query runs on Pinot successfully, so the transform function is indeed part of 0.9.x
@mayanks: @elon.azoulay any idea on the connector?
@mayanks: @sevvy_yusuf can you paste the output of explain
@sevvy_yusuf: Yup - `explain analyze` throws this error too: ```Query 20220307_173536_00009_bawgb failed: java.lang.NoClassDefFoundError: Could not initialize class org.apache.pinot.common.function.FunctionRegistry```
@sevvy_yusuf: Looking at the stack trace the error arises when the connector tries to build a dynamic table from the PQL. This is where it throws: ```org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106```
@richard892: `NoClassDefFoundError` means that the class could not be loaded but not because the classfile is missing
@richard892: it normally means initialization failed if the classloader isn't buggy
@g.kishore: I think the LASTWITHTIME udf was added recently and is not part of the 0.9.3.. I might be wrong
@sevvy_yusuf: Hi @g.kishore the docs mention LASTWITHTIME in `0.9.0`. Could there have been extra functionality that is required for it added later maybe?
@kevin.dethelot: @kevin.dethelot has joined the channel
#pinot-dev
@gxm.monica: @gxm.monica has joined the channel
#pinot-realtime-table-rebalance
@tony: @tony has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
