sajjad-moradi commented on code in PR #9377:
URL: https://github.com/apache/pinot/pull/9377#discussion_r968959896
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java:
##########
@@ -163,11 +172,20 @@ private static Map<String, String>
parsePrefixesToRename(TableConfig tableConfig
@Override
public GenericRow transform(GenericRow record) {
- flattenMap(record, new ArrayList<>(record.getFieldToValueMap().keySet()));
- for (String collection : _fieldsToUnnest) {
- unnestCollection(record, collection);
+ try {
Review Comment:
@Jackie-Jiang AFAIK, compiler doesn't optimize the code inside the try
block, but other than that there's no performance overhead for just using a try
block. Let me know if you have experienced otherwise.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -222,27 +226,42 @@ public void build()
TransformPipeline.Result reusedResult = new TransformPipeline.Result();
while (_recordReader.hasNext()) {
long recordReadStartTime = System.currentTimeMillis();
- long recordReadStopTime;
+ long recordReadStopTime = System.currentTimeMillis();
long indexStopTime;
reuse.clear();
- GenericRow decodedRow = _recordReader.next(reuse);
- recordReadStartTime = System.currentTimeMillis();
- _transformPipeline.processRow(decodedRow, reusedResult);
- recordReadStopTime = System.currentTimeMillis();
- _totalRecordReadTime += (recordReadStopTime - recordReadStartTime);
+ try {
+ GenericRow decodedRow = _recordReader.next(reuse);
+ recordReadStartTime = System.currentTimeMillis();
+ _transformPipeline.processRow(decodedRow, reusedResult);
+ recordReadStopTime = System.currentTimeMillis();
+ _totalRecordReadTime += (recordReadStopTime - recordReadStartTime);
+ } catch (Exception e) {
+ if (_continueOnError) {
+ incompleteRowsFound++;
+ LOGGER.debug("Error occurred while reading row during indexing",
e);
+ continue;
+ }
+ }
Review Comment:
Do we have documentation on what "Continue on Error" flag mean? I thought if
it's set to false or not set, we want to stop the ingestion. If that's true,
then we need to rethrow if _continueOnError is not set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
