[GitHub] npawar commented on issue #3748: Use the PinotFS instead of local file system to open a segment file t…

[GitBox]

npawar commented on issue #3748: Use the PinotFS instead of local file system 
to open a segment file t…
URL: https://github.com/apache/incubator-pinot/pull/3748#issuecomment-461218017
 
 
   I did some evaluation. You can introduce the new metadata upload
   **Step 1**
   Add a new endpoint in LLCSegmentCompletionHandler as
   ```
     @POST
     @Path(SegmentCompletionProtocol.MSG_TYPE_COMMIT_END_METADATA)
     @Produces(MediaType.APPLICATION_JSON)
     @Consumes(MediaType.MULTIPART_FORM_DATA)
     public String 
segmentCommitEndWithMetadata(@QueryParam(SegmentCompletionProtocol.PARAM_INSTANCE_ID)
 String instanceId,
         @QueryParam(SegmentCompletionProtocol.PARAM_SEGMENT_NAME) String 
segmentName,
         @QueryParam(SegmentCompletionProtocol.PARAM_SEGMENT_LOCATION) String 
segmentLocation,
         @QueryParam(SegmentCompletionProtocol.PARAM_OFFSET) long offset,
         @QueryParam(SegmentCompletionProtocol.PARAM_MEMORY_USED_BYTES) long 
memoryUsedBytes,
         @QueryParam(SegmentCompletionProtocol.PARAM_BUILD_TIME_MILLIS) long 
buildTimeMillis,
         @QueryParam(SegmentCompletionProtocol.PARAM_WAIT_TIME_MILLIS) long 
waitTimeMillis,
         @QueryParam(SegmentCompletionProtocol.PARAM_ROW_COUNT) int numRows,
         @QueryParam(SegmentCompletionProtocol.PARAM_SEGMENT_SIZE_BYTES) long 
segmentSizeBytes,
         FormDataMultiPart metadataFiles) {
   ```
   In the `FormDataMultiPart metadataFiles` multipart, we expect 
`metadata.properties` and `column.meta`. Extract those and copy them to a local 
temp folder. You can use that folder to create a new SegmentMetadataImpl (the 
way it's done in `PinotLLCRealtimeSegmentManager::extractSegmentMetadata`
   
   
   
   **Step 2**
   In `ServerSegmentCompletionProtocolHandler`, introduce a method 
`segmentCommitEndWithMetadata(SegmentCompletionProtocol.Request.Params params,
         File metadataPropertiesFile, File creationMetaFile)`. The url that 
gets constructed needs to change to POST. These 2 files need to be added as 
entities (similar to the way segment tar file is added in 
`getUploadFileRequest`)
   
   **Step 3**
   In `LLRealtimeSegmentDataManager::doSplitCommit`, change call from 
`segmentCommitEnd` to `segmentCommitEndWithMetadata`. The metadata files for 
this call can be extracted from the segmentTarFile.
   
   **Step 4**
   Conditions can be added, which check if the metadata was extracted 
successfully. If not, download the metadata from the segment store. Ensure path 
coming from non-split commit upload also works
   
   @chenboat @mcvsubbu 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org
For additional commands, e-mail: dev-h...@pinot.apache.org