Re: Kafka Connect - HDFS or FileStream
Appreciate if somebody has experience on the above and respond to the same. Logically it should work if we have a property set hdfs.url = file:///home/user/data On Wed, May 15, 2019 at 5:06 AM Vinay Jain wrote: > Redirecting to a file will not work, we would not be able to create a > different filenames after some time or filesize. HDFS connect sink already > has those options , also i could use some minor transformations while using > the connect > > On Tue, May 14, 2019 at 7:28 AM Hans Jespersen wrote: > >> Can you just use kafka-console-consumer and just redirect the output into >> a >> file? >> >> -hans >> >> >> On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: >> >> > Hi >> > >> > The data needs to be transferred to some other system in other network, >> and >> > due to some security reasons, the other systems cannot be exposed . So >> the >> > available mechanism is file based integration. Is there a production >> ready >> > Kafka connect adapter which can create files in local directory. >> > >> > Regards >> > Vinay Jain >> > >> > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt >> wrote: >> > >> > > Can you explain more about why you're writing a file with the data? >> > > Presumably, this is for another application to consume; could it not >> take >> > > the data from Kafka directly, whether with a native client or over the >> > REST >> > > proxy? >> > > Oftentimes local files are unnecessary 'duck tape' for integration >> that >> > can >> > > be done in a better way. >> > > >> > > >> > > -- >> > > >> > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff >> > > >> > > >> > > On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: >> > > >> > > > Hi >> > > > >> > > > I would like to consume a Topic and save the AVRO messages in local >> > > > directory in the AVRO file formats. As per Kafka Connect File Stream >> > > > documentation, it is not for production use. >> > > > >> > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am >> > not >> > > > sure if it can also write to the Local directory if we pass in the >> > > variable >> > > > hdfs.url the URL for local file system instead of HDFS path. >> > > > >> > > > Will this work or are there any other ready made options which can >> be >> > > used >> > > > for the same. >> > > > >> > > > Regards >> > > > >> > > > Vinay >> > > > >> > > >> > >> >
Re: Kafka Connect - HDFS or FileStream
Redirecting to a file will not work, we would not be able to create a different filenames after some time or filesize. HDFS connect sink already has those options , also i could use some minor transformations while using the connect On Tue, May 14, 2019 at 7:28 AM Hans Jespersen wrote: > Can you just use kafka-console-consumer and just redirect the output into a > file? > > -hans > > > On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > > > Hi > > > > The data needs to be transferred to some other system in other network, > and > > due to some security reasons, the other systems cannot be exposed . So > the > > available mechanism is file based integration. Is there a production > ready > > Kafka connect adapter which can create files in local directory. > > > > Regards > > Vinay Jain > > > > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt > wrote: > > > > > Can you explain more about why you're writing a file with the data? > > > Presumably, this is for another application to consume; could it not > take > > > the data from Kafka directly, whether with a native client or over the > > REST > > > proxy? > > > Oftentimes local files are unnecessary 'duck tape' for integration that > > can > > > be done in a better way. > > > > > > > > > -- > > > > > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > > > > > > > > On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: > > > > > > > Hi > > > > > > > > I would like to consume a Topic and save the AVRO messages in local > > > > directory in the AVRO file formats. As per Kafka Connect File Stream > > > > documentation, it is not for production use. > > > > > > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am > > not > > > > sure if it can also write to the Local directory if we pass in the > > > variable > > > > hdfs.url the URL for local file system instead of HDFS path. > > > > > > > > Will this work or are there any other ready made options which can be > > > used > > > > for the same. > > > > > > > > Regards > > > > > > > > Vinay > > > > > > > > > >
Re: Kafka Connect - HDFS or FileStream
Can you just use kafka-console-consumer and just redirect the output into a file? -hans On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > Hi > > The data needs to be transferred to some other system in other network, and > due to some security reasons, the other systems cannot be exposed . So the > available mechanism is file based integration. Is there a production ready > Kafka connect adapter which can create files in local directory. > > Regards > Vinay Jain > > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt wrote: > > > Can you explain more about why you're writing a file with the data? > > Presumably, this is for another application to consume; could it not take > > the data from Kafka directly, whether with a native client or over the > REST > > proxy? > > Oftentimes local files are unnecessary 'duck tape' for integration that > can > > be done in a better way. > > > > > > -- > > > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > > > > > On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: > > > > > Hi > > > > > > I would like to consume a Topic and save the AVRO messages in local > > > directory in the AVRO file formats. As per Kafka Connect File Stream > > > documentation, it is not for production use. > > > > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am > not > > > sure if it can also write to the Local directory if we pass in the > > variable > > > hdfs.url the URL for local file system instead of HDFS path. > > > > > > Will this work or are there any other ready made options which can be > > used > > > for the same. > > > > > > Regards > > > > > > Vinay > > > > > >
Re: Kafka Connect - HDFS or FileStream
Hi The data needs to be transferred to some other system in other network, and due to some security reasons, the other systems cannot be exposed . So the available mechanism is file based integration. Is there a production ready Kafka connect adapter which can create files in local directory. Regards Vinay Jain On Mon, May 13, 2019 at 3:41 PM Robin Moffatt wrote: > Can you explain more about why you're writing a file with the data? > Presumably, this is for another application to consume; could it not take > the data from Kafka directly, whether with a native client or over the REST > proxy? > Oftentimes local files are unnecessary 'duck tape' for integration that can > be done in a better way. > > > -- > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff > > > On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: > > > Hi > > > > I would like to consume a Topic and save the AVRO messages in local > > directory in the AVRO file formats. As per Kafka Connect File Stream > > documentation, it is not for production use. > > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am not > > sure if it can also write to the Local directory if we pass in the > variable > > hdfs.url the URL for local file system instead of HDFS path. > > > > Will this work or are there any other ready made options which can be > used > > for the same. > > > > Regards > > > > Vinay > > >
Re: Kafka Connect - HDFS or FileStream
Can you explain more about why you're writing a file with the data? Presumably, this is for another application to consume; could it not take the data from Kafka directly, whether with a native client or over the REST proxy? Oftentimes local files are unnecessary 'duck tape' for integration that can be done in a better way. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: > Hi > > I would like to consume a Topic and save the AVRO messages in local > directory in the AVRO file formats. As per Kafka Connect File Stream > documentation, it is not for production use. > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am not > sure if it can also write to the Local directory if we pass in the variable > hdfs.url the URL for local file system instead of HDFS path. > > Will this work or are there any other ready made options which can be used > for the same. > > Regards > > Vinay >
Kafka Connect - HDFS or FileStream
Hi I would like to consume a Topic and save the AVRO messages in local directory in the AVRO file formats. As per Kafka Connect File Stream documentation, it is not for production use. Other Option I am thinking to use Kafka Connect - HDFS Sink but I am not sure if it can also write to the Local directory if we pass in the variable hdfs.url the URL for local file system instead of HDFS path. Will this work or are there any other ready made options which can be used for the same. Regards Vinay
