npawar commented on issue #7262:
URL: https://github.com/apache/pinot/issues/7262#issuecomment-897903962
@HJK181 , i was able to get this working with your avro schema, avro data,
and Pinot schema. This does not use the complex config, but instead just
transform configs for all fields, as @Jackie-Jiang and I were suggesting.
Here's my schema:
```
{
"schemaName": "kpis",
"dimensionFieldSpecs": [
{
"name": "userQuery",
"dataType": "STRING"
},
{
"name": "masterQuery",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "clicks",
"dataType": "INT"
},
{
"name": "checkout",
"dataType": "INT"
},
{
"name": "cart",
"dataType": "INT"
}
],
"dateTimeFieldSpecs": [
{
"name": "timestamp",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
},
{
"name": "mappingTime",
"dataType": "LONG",
"format": "1:SECONDS:EPOCH",
"granularity": "15:MINUTES"
}
]
}
```
Here's my table:
```
{
"tableName": "kpis_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"timeColumnName": "timestamp",
"replication": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP"
},
"metadata": {},
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "userQuery",
"transformFunction": "jsonPathString(query, '$.userQuery')"
},
{
"columnName": "masterQuery",
"transformFunction": "jsonPathString(query, '$.masterQuery')"
},
{
"columnName": "mappingTime",
"transformFunction": "jsonPathLong(query, '$.mappingTime')"
},
{
"columnName": "timestamp",
"transformFunction": "Groovy({mappingTime == -1 ? 31536000 :
mappingTime}, mappingTime)"
},
{
"columnName": "clicks",
"transformFunction": "jsonPathDouble(kpis, '$.clicks')"
},
{
"columnName": "checkout",
"transformFunction": "jsonPathDouble(kpis, '$.checkout')"
},
{
"columnName": "cart",
"transformFunction": "jsonPathDouble(kpis, '$.cart')"
}
]
},
"isDimTable": false
}
```
And here's the query result:
I suggest you use transform configs to unblock yourself, while we improve
the documentation for the complex config. @yupeng9 would it be possible to
figure out what is the source of confusion/issue with the instructions for
using the complex config and fix up the documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
