#general
@libhojani: @libhojani has joined the channel
@humengyuk18: Can I use the `datetimeconvert` inbuilt function for ingestion transform in pinot? Is there any limitations when transforming time columns? I’m getting error when adding a transform function to table config, but no specific error msg is logged out.
@npawar: You cannot use datetimeconvert in the ingestion. But the existing dateTime functions listed on the docs should be enough to achieve what you want.
@npawar: You may have to chain them
#random
@libhojani: @libhojani has joined the channel
#troubleshooting
@libhojani: @libhojani has joined the channel
#getting-started
@richhickey: @richhickey has joined the channel
#minion-improvements
@laxman: *Status* 1. Cherry-picked NPE patch(Jackie), Memory consumption issue patch (Neha) to 0.7.1 branch. This release is available via jitpack for others usage if needed.
@laxman: *Example #1:* ``` // map of api callee name -> incoming call count map<string> api_callee_name_count = {};``` Above field got added recently and we are seeing the following error while converting older segments dated prior to the field addition. ``` "5": { "TASK_ID": "Task_RealtimeToOfflineSegmentsTask_1620500694484_6", "ASSIGNED_PARTICIPANT": "Minion_10.8.4.75_9514", "STATE": "TASK_ERROR", "START_TIME": "1620500720538", "NUM_ATTEMPTS": "1", "INFO": "org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of array in field api_callee_name_count__VALUES of record", "FINISH_TIME": "1620500724083" },```
@laxman: *Example #2:* Field added ``` union { null, string } backend_destination = null;``` Error ``` "0": { "ASSIGNED_PARTICIPANT": "Minion_10.8.4.75_9514", "NUM_ATTEMPTS": "1", "START_TIME": "1620500716291", "STATE": "TASK_ERROR", "TASK_ID": "Task_RealtimeToOfflineSegmentsTask_1620500694484_2", "INFO": "org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of string in field backend_destination of record", "FINISH_TIME": "1620500720538" },```
@jackie.jxt: I think the issue is that new added columns does not having default values properly generated
@jackie.jxt: Thanks for reporting this, will think of a solution and get back to you
@jackie.jxt: I feel the correct way to handle it should be backfill null values for existing nullable columns, and keep the record transformer
@jackie.jxt: This way the null value vector not preserved problem is also solved
@laxman: Do you mean we need to apply CompositeTransformer in Mapper phase?
@jackie.jxt: Yeah, I think that might be the correct way to go
@jackie.jxt: In case the latest config does not match the existing segment config, the transformer can apply the necessary changes
@laxman: > the transformer can apply the necessary changes Had gone through Composite transformer. Couldn’t figure out which transformer does this? Can you please point me to the code/class where this is done?
@laxman: > This way the null value vector not preserved problem is also solved I don’t think this is possible even if we apply NullValueTransformer. I actually tried it. I couldn’t rebuild that null vector. RT->OFL Segment Converter starts with a reading a segment and segment doesn’t preserve this null vector. So, I think there is no way to reconstruct the null vector from the segment file. I may be wrong and I’ll be happy to be proven wrong.
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
