Hi Dustin, I am looking for option 1.
Looking at Kafka Connect code, I guess we need to write converter code if not available. Thanks in advance. Regards Pari On 22 June 2016 at 18:50, Dustin Cote <dus...@confluent.io> wrote: > Hi Pari, > > Can you clarify which scenario you are looking to implement? > 1) plaintext Kafka data --> plaintext HDFS data readable by hive > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive > > Regards, > > > > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre < > pari.data...@gmail.com> wrote: > > > Thanks for your suggestions. I think if kafka connect provides the same > > functionality as flume and storm, why should we go for another > > infrastructure investment. > > > > Kafka Connect effectively copies data from Kafka topic to HDFS through > > connector. It supports avro as well as parquet, I am looking if we can > use > > it to load plain text data. > > > > Cheers > > Pari > > > > > > > > On 22 June 2016 at 12:34, Lohith Samaga M <lohith.sam...@mphasis.com> > > wrote: > > > > > Hi, > > > You can use Storm also, Here you have the option of rotating > the > > > file. You can also write to Hive directly. > > > > > > Best regards / Mit freundlichen Grüßen / Sincères salutations > > > M. Lohith Samaga > > > > > > > > > > > > > > > -----Original Message----- > > > From: Mudit Kumar [mailto:mudit.ku...@askme.in] > > > Sent: Wednesday, June 22, 2016 12.32 > > > To: users@kafka.apache.org; d...@kafka.apache.org > > > Subject: Re: Kafka HDFS Connector > > > > > > I think you can use flume also. > > > > > > Thanks, > > > Mudit > > > > > > > > > > > > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pari.data...@gmail.com> > > > wrote: > > > > > > >Anybody have any idea on this? > > > > > > > >Thanks > > > >Pari > > > > > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre < > pari.data...@gmail.com> > > > >wrote: > > > > > > > >> Hello All, > > > >> > > > >> I have data coming from sensors into kafka cluster in text format > > > >> delimited by comma. > > > >> > > > >> How to offload this data to Hive periodically from Kafka. I guess, > > > >> Kafka Connect should solve my problem but when I checked > > > >> documentation, examples have only avro formatted data. Can you > please > > > >> provide some knowledge on this. > > > >> > > > >> Many Thanks > > > >> Pari > > > >> > > > > > > Information transmitted by this e-mail is proprietary to Mphasis, its > > > associated companies and/ or its customers and is intended > > > for use only by the individual or entity to which it is addressed, and > > may > > > contain information that is privileged, confidential or > > > exempt from disclosure under applicable law. If you are not the > intended > > > recipient or it appears that this mail has been forwarded > > > to you without proper authority, you are notified that any use or > > > dissemination of this information in any manner is strictly > > > prohibited. In such cases, please notify us immediately at > > > mailmas...@mphasis.com and delete this mail from your records. > > > > > > > > > -- > Dustin Cote > confluent.io >
