Hi Sumant, I have resumed work on the RSM implementations for Azure Storage. If you are interested in discussing the implementation details, please reach out to me directly and we can set up a call to discuss further.
Thanks. On Mon, Jul 12, 2021 at 11:12 PM Satish Duggana <[email protected]> wrote: > Hi Sumant, > We are tracking HDFS and S3 implementations as part of this project to > see the proposed APIs are addressing file/object stores. As Jun > mentioned in the earlier mail thread, no RSM implementation will be > part of Apache Kafka. This is in line with Kafka Connectors > development. Devs/Users can build their own implementations and host > them to share with others. > > We have HDFS/S3 RSMs which can be treated as reference > implementations. These can be referenced to build your own > implementations if needed. > > Thanks, > Satish. > > > On Tue, 13 Jul 2021 at 02:25, Sumant Tambe <[email protected]> wrote: > > > > Hi Israel, > > > > Linkedin is interested in evaluating KIP-405 for HDFS and S3 in the short > > term and Azure Blob Storage in the long run. You may already know that > Linkedin > > is migrating to Azure > > <https://engineering.linkedin.com/blog/2019/building-next-infra>. We > think > > that Blobs will provide us with the optimal cost/availability/operability > > trade-offs for Kafka in Azure. > > What's the context behind your interest in KIP-405 and Azure Blobs? Do > you > > have any data/experience of using Azure blobs at scale? > > > > Satish, any word on the KIP-405 and the RSM implementations? > > > > Regards, > > Sumant > > Kafka Dev, Linkedin > > > > On Mon, 15 Mar 2021 at 09:08, Satish Duggana <[email protected]> > > wrote: > > > > > Hi Israel, > > > Thanks for your interest in tiered storage. As mentioned by Jun > earlier, we > > > decided not to have any implementations in Apache Kafka repo like Kafka > > > connectors. We plan to have RSM implementations for HDFS, S3, GCP, and > > > Azure storages in a separate repo. We will let you know once they are > ready > > > for review. > > > > > > Best, > > > Satish. > > > > > > On Sat, 13 Mar 2021 at 01:27, Israel Ekpo <[email protected]> > wrote: > > > > > > > Thanks @Jun for the prompt response. > > > > > > > > That's ok and I think it is a great strategy just like the Connect > > > > ecosystem. > > > > > > > > However, I am still in search for repos and samples that demonstrate > > > > implementation for the KIP. > > > > > > > > I will keep searching but was just wondering if there were sample > > > > implementations for S3 or HDFS I could take a look at. > > > > > > > > Thanks. > > > > > > > > On Fri, Mar 12, 2021 at 2:19 PM Jun Rao <[email protected]> > > > wrote: > > > > > > > > > Hi, Israel, > > > > > > > > > > Thanks for your interest. As part of KIP-405, we have made the > decision > > > > not > > > > > to host any plugins for external remote storage directly in Apache > > > Kafka. > > > > > Those plugins could be hosted outside of Apache Kafka. > > > > > > > > > > Jun > > > > > > > > > > On Thu, Mar 11, 2021 at 5:15 PM Israel Ekpo <[email protected]> > > > > wrote: > > > > > > > > > > > Thanks Satish, Sriharsha, Suresh and Ying for authoring this KIP > and > > > > > thanks > > > > > > to everyone that participated in the review and discussion to > take it > > > > to > > > > > > where it is today. > > > > > > > > > > > > I would like to contribute by working on integrating Azure > Storage > > > > (Blob > > > > > > and ADLS) with Tiered Storage for this KIP > > > > > > > > > > > > I have created this issue to track this work > > > > > > https://issues.apache.org/jira/browse/KAFKA-12458 > > > > > > > > > > > > Are there any sample implementations for HDFS/S3 that I can > reference > > > > to > > > > > > get started? > > > > > > > > > > > > When you have a moment, please share. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > >
