Hi, jinhui, Thanks for your information.
I understood the difference. It is time to do vote if nobody has more concerns. Best, Jingsong Jingsong Li <[email protected]>于2025年1月15日 周三22:23写道： > Hi xuanwo, thanks for the suggestion. > > We are talking about getting rid of the middle layer, and use COSN > directly. > > So if OpenDAL has no difference to COSN, it is better to use it directly. > > Best, > Jingsong > > 刘金辉 <[email protected]>于2025年1月15日 周三19:02写道： > >> hi，Jingsong teacher. >> Here is some information，，please check thanks >> Optimization methods of cosn: >> 1. Performance optimization: >> a. Combined with the architecture of COS fusion bucket, separate metadata >> flow and data flow to achieve high-performance file semantic access >> b. Based on COS distributed metadata engine, provide high-performance >> metadata access >> c. Provide high-concurrency data flow through pre-reading strategy, block >> processing and other means >> 2. Integrate Tencent Cloud service role authentication capabilities to >> improve access security >> 3. Integrate Tencent Cloud log reporting capabilities, support task-level >> log reporting, and improve observability >> >> >> >> >> >> S3 Architecture: >> >> >> >> >> >> >> >> >> >> >> Cosn Architecture >> >> liujinhui >> >> >> >> >> 刘金辉 >> [email protected] >> >> >> >> >> >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: >> "dev" >> < >> [email protected]>; >> 发送时间: 2025年1月9日(星期四) 下午4:20 >> 收件人: "dev"<[email protected]>; >> >> 主题: Re: [DISCUSS]paimon support tencent cosn >> >> >> >> Hi, >> >> That's interesting to come across this discussion. Apache OpenDAL offers >> native support for COS (although we primarily utilize it for native IAM >> support and are not currently aware of any performance gaps). Perhaps we >> could consider adding a new file IO class based on OpenDAL to support this? >> >> Additionally, OpenDAL provides native support for nearly all storage >> services, ranging from S3, Gcs to Azblob, which could be a valuable >> addition for us. >> >> For more information about OpenDAL: https://github.com/apache/opendal >> We also offer a native Java binding: >> https://github.com/apache/opendal/tree/main/bindings/java >> >> On Thu, Jan 9, 2025, at 11:05, Jingsong Li wrote: >> > Hi jinhui, >> > >> > Thanks for driving this, can you provide some information about "When >> > using the s3 protocol to access the fusion bucket, the traffic will >> > pass through the access layer twice", is there an official website? >> > >> > Best, >> > Jingsong >> > >> > On Thu, Jan 9, 2025 at 10:56 AM 刘金辉 <[email protected]> >> wrote: >> >> >> >> Introduction >> >> Tencent COS is a famous object storage system provided by >> Tencent Corp. Hadoop-COS is a client that makes the upper computing systems >> based on HDFS be able to use the COS as its underlying storage system. >> tencent cloud is top 2 cloud vendors in China market and the object store >> COS （https://intl.cloud.tencent.com/product/cos） is widely used among >> China’s cloud users but now it is hard for hadoop user to access data laid >> on COS storage&nbsp; >> >> >> >> >> >> We can use S3 SDK and COSN SDK to access COSN，but >> >> 1. When using the s3 protocol to access the fusion bucket, the >> traffic will pass through the access layer twice, the traffic will be >> doubled, and the performance will be impaired. This problem will not occur >> when using cosn. >> >> 2. Compared with cos ordinary buckets, fusion buckets have >> performance advantages in list, rename, truncate and other operations. >> >> 3. If you must use the s3 protocol to access cos, we do not >> recommend using fusion buckets, then the s3 protocol + cos ordinary bucket >> will encounter some of the above performance issues. >> >> 4. Other data lake frameworks also support access to cosn, oss, >> S3, etc. >> >> >> >> >> >> >> >> >> >> Therefore, in order to allow more users to use paimon with high >> performance, it is necessary to support cosn access to paimon. >> >> >> >> >> >> >> >> >> >> >> >> >> >> Please give your opinions. thanks >> >> >> >> >> >> >> >> >> >> >> >> 刘金辉 >> >> [email protected] >> >> >> >> >> >> >> >> &nbsp; >> >> -- >> Xuanwo >> >> https://xuanwo.io/ > >
