: 1.6.0
Mysql version : 5.7.22-log
Binlog type. : Row-based logging
Thanks a lot
Ashwin Konale
Developer
Booking.com B.V.
Herengracht 597 Amsterdam 1017 CE Netherlands
Direct +31207242562
[image: Booking.com] <https://www.booking.com/>
The world's #1 accommodation site
43 languages, 1
Hello all,
I recently started using nifi to read binlogs and ingest to various
sources. I noticed that when the column type is unsigned, the underlaying
library sends signed version of the value. (Eg, column type is unsigned
tinyint, when the inserted value is 128, it sends -128). Its related to
h
I have a nifi workflow to read from multiple mysql binlogs and put it to
hdfs. I am using ChangeDataCapture as a source and PutHdfs as sink. I am
using MergeContent processor in between to chunk the messages together for
hdfs.
*[CDC (Primary node only. About 200 of this processor for each db)]
*->
Hi,
I am running 3 node nifi cluster (embedded zookeeper) with following flow.
It pulls the data from mysql binlogs using CaptureChangeMysql and sending
it to HDFS.
CDC[Primary node only] -> outputPort -> RPG
inputPort -> MergeContent -> PutHdfs
I am using RPG to distribute load among other nod
Hi I have the following flow which I am trying to increase the throughput.
I am runnign 3 node nifi cluster(v1.6)
CDC -> processGroupOutputPort -> externalOutputPort
RPG -> downstream
Flowfiles are always queued up between processGroupOutputPort to
externalOutputPort. I dont see any configuration
/09/25 15:29:32, ashwin konale wrote:
> Hi I have the following flow which I am trying to increase the throughput.
> I am runnign 3 node nifi cluster(v1.6)
>
> CDC -> processGroupOutputPort -> externalOutputPort
> RPG -> downstream
>
> Flowfiles are always queued up
Output Port
> to use up to 3 threads per node to provide the data, but each node is still
> only using 1 thread to pull the data.
> By configuring the Remote Ports, you can configure the RPG to use 3 threads
> to pull the data as well.
>
> Thanks
> -Mark
>
>
> > On S
rate. It is always
> recommended that an external ZooKeeper be used for any sort of production use.
>
> Thanks
> -Mark
>
> > On Sep 25, 2018, at 6:03 AM, ashwin konale wrote:
> >
> > Hi,
> >
> > I am running 3 node nifi cluster (embedded zookeeper
Output Port
> to use up to 3 threads per node to provide the data, but each node is still
> only using 1 thread to pull the data.
> By configuring the Remote Ports, you can configure the RPG to use 3 threads
> to pull the data as well.
>
> Thanks
> -Mark
>
>
> > On S
Hi,
We have a 3 node nifi cluster (With separate zookeper instances running in
the same machines) which pulls the data from mysql and write to hdfs. I am
frequently running into problems with cluster. Nodes keeps disconnecting
from each other, primary nodes keeps switching and sometimes it just goe
Hi,
I am experimenting with nifi for one of our usecases with plans of
extending it to various other data routing, ingestion usecases. Right now I
need to ingest data from mysql binlogs to hdfs/GCS. We have around 250
different schemas and about 3000 tables to read data from. Volume of the
data fl
to this problem.
On 2018/10/17 21:14:09, Mike Thomsen wrote:
> > may have to build some kind of tooling on top of it to monitor/provision>
> new processor for newly added schemas etc.>
>
> Could you elaborate on this part of your use case?>
>
> On Wed, Oct
his problem.
On 2018/10/17 21:14:09, Mike Thomsen wrote:
> > may have to build some kind of tooling on top of it to
monitor/provision>
> new processor for newly added schemas etc.>
>
> Could you elaborate on this part of your use case?>
>
> On Wed, Oct 17, 2018 at 2
Hi,
I have key-value pair of avro messages in kafka topic I want to consume
from. I can easily do modifications on message value using nifi
consumeKafkaRecord processor, but it doesnt show key of the message.
ConsumeKafka processor has kafka.key attribute but I am not sure how to
read its contents(
try>
> switching to the Kafka string serializer in your producer.>
>
> On Fri, Nov 23, 2018 at 10:12 AM ashwin konale >
> wrote:>
>
> > Hi,>
> > I have key-value pair of avro messages in kafka topic I want to consume>
> > from. I c
ferent kind of key serialization like json which can then>
> be parsed with expression language functions>
>
> A possible improvement we could make is to add some kind of>
> "avro-to-json" EL function, then from there use the EL jsonPath>
> function.>
&
16 matches
Mail list logo