#general
@sosyalmedya.oguzhan: I want to overwrite all data in my offline table, and i set `segmentPushType to REFRESH.` But when i try to overwrite data with new data, old data still be stored. This is my job executon spec; ```executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'input/' includeFileNamePattern: 'glob:**/*.json' outputDirURI: 'output' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'orders' schemaURI: '
@mayanks: For refresh, you need to ensure that the names and number of segments generated and pushed each time of refresh matches the names and number of segments in Pinot.
@sosyalmedya.oguzhan: Can i set segment names as tableName_seqId ? I guess i can overwrite all data with this way
@mayanks: Yes you can specify that
@sosyalmedya.oguzhan: how can i specify?
@sosyalmedya.oguzhan: Also i want to ask one more question; i put 100 segments for the first run with segment names tbl_0, tbl_1 and so on then i put 150 segments for the second run with segment names tbl_0, tbl_1 .. in that case, the first 100 segments will be overwritten, then last 50 segments will be added, right?
@mayanks:
@mayanks: yes, the first 100 will be overwritten only
@sosyalmedya.oguzhan: thank you much fixed doesn't work. i'm working with standalone job spec; java.lang.UnsupportedOperationException: Unsupported segment name generator type: fixed pinot version 0.6
@mayanks: Overwrite works by replacing segments of same names
#presto-pinot-streaming
@sukypanesar: @sukypanesar has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
