Re: Pointer to stream data from Kafka to HDFS

[Neha Narkhede]

Kevin,

In the forthcoming 0.9 release, the community is adding a new ingest/export

feature to help with importing and exporting data to and from Kafka. There
is a Kafka-HDFS connector under active development. If you are interested
in knowing more about it, I'd consider pinging the
confluent-platf...@googlegroups.com mailing list.

Thanks,
Neha

On Fri, Nov 6, 2015 at 5:52 AM, Damian Guy  wrote:

> Hi, Assuming you are using Avro then you might want to take a look at:
> https://github.com/confluentinc/camus
>
> If not you can still adopt the pattern used to get data into HDFS. We use
> something similar at my place of employment.
>
> Thanks,
> Damian
>
> On 6 November 2015 at 08:46, Woodbine, Kevin  >
> wrote:
>
> > Dear Sirs,
> >
> > I'm working on a development project for my company that is looking at
> the
> > possibility of live streaming transaction data from our Oracle databases
> > into Hadoop HDFS.   Doing some research I've been looking into leveraging
> > the power of Kafka to form part of the solution and need some a pointer
> > which I hope you can help me with.
> >
> > We use Goldengate to stream data into a Kafka topic and can see this data
> > when running to command:
> >
> > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
> > --from-beginning
> >
> > My question is how would I get Kafka to write this data from the topic
> > into a HDFS directory?
> >
> > Any help you can provide will be gratefully received.
> >
> > Many thanks
> >
> > Regards
> >
> > Kevin Woodbine
> > * (CUCI) 01784 736039
> > * 07990 563 530
> > kevin.woodbi...@centrica.com
> >
> >
> > _
> > The information contained in or attached to this email is intended only
> > for the use of the individual or entity to which it is addressed. If you
> > are not the intended recipient, or a person responsible for delivering it
> > to the intended recipient, you are not authorised to and must not
> disclose,
> > copy, distribute, or retain this message or any part of it. It may
> contain
> > information which is confidential and/or covered by legal professional or
> > other privilege (or other rules or laws with similar effect in
> > jurisdictions outside England and Wales).
> >
> > The views expressed in this email are not necessarily the views of
> > Centrica plc, and the company, its directors, officers or employees make
> no
> > representation or accept any liability for its accuracy or completeness
> > unless expressly stated to the contrary.
> >
> > PH Jones is a trading name of British Gas Social Housing Limited.
> British
> > Gas Social Housing Limited (company no: 01026007), British Gas Trading
> > Limited (company no: 03078711), British Gas Services Limited (company no:
> > 3141243), British Gas Insurance Limited (company no: 06608316), British
> Gas
> > New Heating Limited (company no: 06723244), British Gas Services
> > (Commercial) Limited (company no: 07385984) and Centrica Energy (Trading)
> > Limited (company no: 02877397) are all wholly owned subsidiaries of
> > Centrica plc (company no: 3033654). Each company is registered in England
> > and Wales with a registered office at Millstream, Maidenhead Road,
> Windsor,
> > Berkshire SL4 5GD.
> >
> > British Gas Insurance Limited is authorised by the Prudential Regulation
> > Authority and regulated by the Financial Conduct Authority and the
> > Prudential Regulation Authority. British Gas Services Limited and
> Centrica
> > Energy (Trading) Limited are authorised and regulated by the Financial
> > Conduct Authority. British Gas Trading Limited is an appointed
> > representative of British Gas Services Limited which is authorised and
> > regulated by the Financial Conduct Authority.
>



-- 
Thanks,
Neha

Re: Pointer to stream data from Kafka to HDFS

[Damian Guy]

Hi, Assuming you are using Avro then you might want to take a look at:
https://github.com/confluentinc/camus

If not you can still adopt the pattern used to get data into HDFS. We use
something similar at my place of employment.

Thanks,
Damian

On 6 November 2015 at 08:46, Woodbine, Kevin 
wrote:

> Dear Sirs,
>
> I'm working on a development project for my company that is looking at the
> possibility of live streaming transaction data from our Oracle databases
> into Hadoop HDFS.   Doing some research I've been looking into leveraging
> the power of Kafka to form part of the solution and need some a pointer
> which I hope you can help me with.
>
> We use Goldengate to stream data into a Kafka topic and can see this data
> when running to command:
>
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
> --from-beginning
>
> My question is how would I get Kafka to write this data from the topic
> into a HDFS directory?
>
> Any help you can provide will be gratefully received.
>
> Many thanks
>
> Regards
>
> Kevin Woodbine
> * (CUCI) 01784 736039
> * 07990 563 530
> kevin.woodbi...@centrica.com
>
>
> _
> The information contained in or attached to this email is intended only
> for the use of the individual or entity to which it is addressed. If you
> are not the intended recipient, or a person responsible for delivering it
> to the intended recipient, you are not authorised to and must not disclose,
> copy, distribute, or retain this message or any part of it. It may contain
> information which is confidential and/or covered by legal professional or
> other privilege (or other rules or laws with similar effect in
> jurisdictions outside England and Wales).
>
> The views expressed in this email are not necessarily the views of
> Centrica plc, and the company, its directors, officers or employees make no
> representation or accept any liability for its accuracy or completeness
> unless expressly stated to the contrary.
>
> PH Jones is a trading name of British Gas Social Housing Limited.  British
> Gas Social Housing Limited (company no: 01026007), British Gas Trading
> Limited (company no: 03078711), British Gas Services Limited (company no:
> 3141243), British Gas Insurance Limited (company no: 06608316), British Gas
> New Heating Limited (company no: 06723244), British Gas Services
> (Commercial) Limited (company no: 07385984) and Centrica Energy (Trading)
> Limited (company no: 02877397) are all wholly owned subsidiaries of
> Centrica plc (company no: 3033654). Each company is registered in England
> and Wales with a registered office at Millstream, Maidenhead Road, Windsor,
> Berkshire SL4 5GD.
>
> British Gas Insurance Limited is authorised by the Prudential Regulation
> Authority and regulated by the Financial Conduct Authority and the
> Prudential Regulation Authority. British Gas Services Limited and Centrica
> Energy (Trading) Limited are authorised and regulated by the Financial
> Conduct Authority. British Gas Trading Limited is an appointed
> representative of British Gas Services Limited which is authorised and
> regulated by the Financial Conduct Authority.

Pointer to stream data from Kafka to HDFS

[Woodbine, Kevin]

Dear Sirs,

I'm working on a development project for my company that is looking at the 
possibility of live streaming transaction data from our Oracle databases into 
Hadoop HDFS.   Doing some research I've been looking into leveraging the power 
of Kafka to form part of the solution and need some a pointer which I hope you 
can help me with.

We use Goldengate to stream data into a Kafka topic and can see this data when 
running to command:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test 
--from-beginning

My question is how would I get Kafka to write this data from the topic into a 
HDFS directory?

Any help you can provide will be gratefully received.

Many thanks

Regards

Kevin Woodbine
* (CUCI) 01784 736039
* 07990 563 530
kevin.woodbi...@centrica.com


_
The information contained in or attached to this email is intended only for the 
use of the individual or entity to which it is addressed. If you are not the 
intended recipient, or a person responsible for delivering it to the intended 
recipient, you are not authorised to and must not disclose, copy, distribute, 
or retain this message or any part of it. It may contain information which is 
confidential and/or covered by legal professional or other privilege (or other 
rules or laws with similar effect in jurisdictions outside England and Wales).

The views expressed in this email are not necessarily the views of Centrica 
plc, and the company, its directors, officers or employees make no 
representation or accept any liability for its accuracy or completeness unless 
expressly stated to the contrary.

PH Jones is a trading name of British Gas Social Housing Limited.  British Gas 
Social Housing Limited (company no: 01026007), British Gas Trading Limited 
(company no: 03078711), British Gas Services Limited (company no: 3141243), 
British Gas Insurance Limited (company no: 06608316), British Gas New Heating 
Limited (company no: 06723244), British Gas Services (Commercial) Limited 
(company no: 07385984) and Centrica Energy (Trading) Limited (company no: 
02877397) are all wholly owned subsidiaries of Centrica plc (company no: 
3033654). Each company is registered in England and Wales with a registered 
office at Millstream, Maidenhead Road, Windsor, Berkshire SL4 5GD.

British Gas Insurance Limited is authorised by the Prudential Regulation 
Authority and regulated by the Financial Conduct Authority and the Prudential 
Regulation Authority. British Gas Services Limited and Centrica Energy 
(Trading) Limited are authorised and regulated by the Financial Conduct 
Authority. British Gas Trading Limited is an appointed representative of 
British Gas Services Limited which is authorised and regulated by the Financial 
Conduct Authority.