lasdf1234 commented on PR #12241: URL: https://github.com/apache/gravitino/pull/12241#issuecomment-5114055971
@laserninja Thank you very much for your design document. Especially the open-ended questions are very profound. Your meticulous and striving spirit is truly admirable. Regarding the three open-ended questions, I will provide the answers in the next session. 1. Should batching depend on the scan plan cache? This is a very good question. I think for this design document, we can temporarily not address the changes related to caching. You can create a new issue to discuss whether caching should be enabled by default. (I prefer to enable caching. If an issue is raised, it can be discussed in that issue.) 2. Is 100 the right default? In my opinion, the design of the interface should take into account the actual usage scenarios. For many scenarios, the default size of the gateway/LB is 1MB. 100 items would result in approximately 0.3MB of data being returned. So for now, I think 100 is a feasible option. If issues are encountered during actual scenario testing (where the default value of 100 causes many problems), I believe we can open another issue. 3. Should tokens carry an expiry? In the source code and specifications of iceberg, there is no explanation for this thing called "token". Instead, it is more regarded as an opaque string. (The open API of iceberg: An opaque string provided by the REST server that represents a unit of work for generating file scan tasks for scan planning.) Therefore, I think this class should be named PlanTaskCodec instead of planTaskToken. This string is just an ordinary parameter (like tableName and catalogName, etc.), so I don't think there is a need to consider the issue of expiration. I'm not sure if these answers are appropriate. I'm still willing to continue the discussion with you. If you agree with these responses, you can rewrite the design document based on this content. Don't forget to create a new issue: Should the cache be set as the default value. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
