Hi Pari, Can you clarify which scenario you are looking to implement? 1) plaintext Kafka data --> plaintext HDFS data readable by hive 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive
Regards, On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre < pari.data...@gmail.com> wrote: > Thanks for your suggestions. I think if kafka connect provides the same > functionality as flume and storm, why should we go for another > infrastructure investment. > > Kafka Connect effectively copies data from Kafka topic to HDFS through > connector. It supports avro as well as parquet, I am looking if we can use > it to load plain text data. > > Cheers > Pari > > > > On 22 June 2016 at 12:34, Lohith Samaga M <lohith.sam...@mphasis.com> > wrote: > > > Hi, > > You can use Storm also, Here you have the option of rotating the > > file. You can also write to Hive directly. > > > > Best regards / Mit freundlichen Grüßen / Sincères salutations > > M. Lohith Samaga > > > > > > > > > > -----Original Message----- > > From: Mudit Kumar [mailto:mudit.ku...@askme.in] > > Sent: Wednesday, June 22, 2016 12.32 > > To: users@kafka.apache.org; d...@kafka.apache.org > > Subject: Re: Kafka HDFS Connector > > > > I think you can use flume also. > > > > Thanks, > > Mudit > > > > > > > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" <pari.data...@gmail.com> > > wrote: > > > > >Anybody have any idea on this? > > > > > >Thanks > > >Pari > > > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre <pari.data...@gmail.com> > > >wrote: > > > > > >> Hello All, > > >> > > >> I have data coming from sensors into kafka cluster in text format > > >> delimited by comma. > > >> > > >> How to offload this data to Hive periodically from Kafka. I guess, > > >> Kafka Connect should solve my problem but when I checked > > >> documentation, examples have only avro formatted data. Can you please > > >> provide some knowledge on this. > > >> > > >> Many Thanks > > >> Pari > > >> > > > > Information transmitted by this e-mail is proprietary to Mphasis, its > > associated companies and/ or its customers and is intended > > for use only by the individual or entity to which it is addressed, and > may > > contain information that is privileged, confidential or > > exempt from disclosure under applicable law. If you are not the intended > > recipient or it appears that this mail has been forwarded > > to you without proper authority, you are notified that any use or > > dissemination of this information in any manner is strictly > > prohibited. In such cases, please notify us immediately at > > mailmas...@mphasis.com and delete this mail from your records. > > > -- Dustin Cote confluent.io
