Fwd: [CaptureChangeMysql] Cannot get real integer value when integer column set unsigned

: 1.6.0 Mysql version : 5.7.22-log Binlog type. : Row-based logging Thanks a lot Ashwin Konale Developer Booking.com B.V. Herengracht 597 Amsterdam 1017 CE Netherlands Direct +31207242562 [image: Booking.com] <https://www.booking.com/> The world's #1 accommodation site 43 languages, 1

[CaptureChangeMysql] Cannot get real value of primitive type when column type is unsigned.

Hello all, I recently started using nifi to read binlogs and ingest to various sources. I noticed that when the column type is unsigned, the underlaying library sends signed version of the value. (Eg, column type is unsigned tinyint, when the inserted value is 128, it sends -128). Its related to h

How to optimise use of MergeContent for large number of bins.

I have a nifi workflow to read from multiple mysql binlogs and put it to hdfs. I am using ChangeDataCapture as a source and PutHdfs as sink. I am using MergeContent processor in between to chunk the messages together for hdfs. *[CDC (Primary node only. About 200 of this processor for each db)] *->

Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

Hi, I am running 3 node nifi cluster (embedded zookeeper) with following flow. It pulls the data from mysql binlogs using CaptureChangeMysql and sending it to HDFS. CDC[Primary node only] -> outputPort -> RPG inputPort -> MergeContent -> PutHdfs I am using RPG to distribute load among other nod

Slow flowfile transfer from process group port to output port.

Hi I have the following flow which I am trying to increase the throughput. I am runnign 3 node nifi cluster(v1.6) CDC -> processGroupOutputPort -> externalOutputPort RPG -> downstream Flowfiles are always queued up between processGroupOutputPort to externalOutputPort. I dont see any configuration

Re: Slow flowfile transfer from process group port to output port.

/09/25 15:29:32, ashwin konale wrote: > Hi I have the following flow which I am trying to increase the throughput. > I am runnign 3 node nifi cluster(v1.6) > > CDC -> processGroupOutputPort -> externalOutputPort > RPG -> downstream > > Flowfiles are always queued up

Re: Slow flowfile transfer from process group port to output port.

Output Port > to use up to 3 threads per node to provide the data, but each node is still > only using 1 thread to pull the data. > By configuring the Remote Ports, you can configure the RPG to use 3 threads > to pull the data as well. > > Thanks > -Mark > > > > On S

Re: Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

rate. It is always > recommended that an external ZooKeeper be used for any sort of production use. > > Thanks > -Mark > > > On Sep 25, 2018, at 6:03 AM, ashwin konale wrote: > > > > Hi, > > > > I am running 3 node nifi cluster (embedded zookeeper

Re: Slow flowfile transfer from process group port to output port.

Output Port > to use up to 3 threads per node to provide the data, but each node is still > only using 1 thread to pull the data. > By configuring the Remote Ports, you can configure the RPG to use 3 threads > to pull the data as well. > > Thanks > -Mark > > > > On S

Fixing unstable nifi cluster.

Hi, We have a 3 node nifi cluster (With separate zookeper instances running in the same machines) which pulls the data from mysql and write to hdfs. I am frequently running into problems with cluster. Nodes keeps disconnecting from each other, primary nodes keeps switching and sometimes it just goe

Scaling source processors in nifi horizontally.

Hi, I am experimenting with nifi for one of our usecases with plans of extending it to various other data routing, ingestion usecases. Right now I need to ingest data from mysql binlogs to hdfs/GCS. We have around 250 different schemas and about 3000 tables to read data from. Volume of the data fl

Re: Scaling source processors in nifi horizontally.

to this problem. On 2018/10/17 21:14:09, Mike Thomsen wrote: > > may have to build some kind of tooling on top of it to monitor/provision> > new processor for newly added schemas etc.> > > Could you elaborate on this part of your use case?> > > On Wed, Oct

Re: Scaling source processors in nifi horizontally.

his problem. On 2018/10/17 21:14:09, Mike Thomsen wrote: > > may have to build some kind of tooling on top of it to monitor/provision> > new processor for newly added schemas etc.> > > Could you elaborate on this part of your use case?> > > On Wed, Oct 17, 2018 at 2

Reading avro encoded message key from kafka

Hi, I have key-value pair of avro messages in kafka topic I want to consume from. I can easily do modifications on message value using nifi consumeKafkaRecord processor, but it doesnt show key of the message. ConsumeKafka processor has kafka.key attribute but I am not sure how to read its contents(

Re: Reading avro encoded message key from kafka

try> > switching to the Kafka string serializer in your producer.> > > On Fri, Nov 23, 2018 at 10:12 AM ashwin konale > > wrote:> > > > Hi,> > > I have key-value pair of avro messages in kafka topic I want to consume> > > from. I c

Re: Reading avro encoded message key from kafka

ferent kind of key serialization like json which can then> > be parsed with expression language functions> > > A possible improvement we could make is to add some kind of> > "avro-to-json" EL function, then from there use the EL jsonPath> > function.> &