Re: Redis - Sorted Sets

[Eric Chaves]

Hi John,

Here at work we also needed to use redis in a flow to store and retrieve
real-time events temporary. We haven't found any "Put/Fetch Redis"
processors and the RedisDistributedMapCache didn't fit out needs precisely
because our flow required more control over the key namespace and also to
use the native redis data types (lists and hashmaps).

We ended wroting a custom groovy ScriptedProcessor using the Jedis library
and it was actually pretty easy. The Nifi ExecuteScript cookbook is your
best friend to start.

I can't share the actual processor we wrote due to some internal logic it
has, but if you need some guidance send me a private message and may be
able to help you out.

Cheers,


Em sex, 13 de set de 2019 às 16:41, John Fak  escreveu:

> Has anyone used nifi to move data to Redis ?
> What processors etc ?
> Ideally want to use a sorted set in redis.
>

Redis - Sorted Sets

[John Fak]

Has anyone used nifi to move data to Redis ?
What processors etc ?
Ideally want to use a sorted set in redis.

RE: HandleHttpRequest - Failed to receive content from HTTP Request

[William Gosse]

I created the following Jira issue: 
https://issues.apache.org/jira/browse/NIFI-6661

Here’s some more info on this issue:
Also I only see this happening with one of my user who may be ealing with 
network latency.

I have not been able to recreate the issue myself with this exact excetion 
message. However the fact the it causes HandleHttpRequest processor to stop 
functioning seems serious to me.
The only time that ever happened to me is when it to long to get to the 
HandleHttpResponse. I currently have the Request Expiration property for my 
StandardHttpContextMap set to 10 minutes. If the this value is exceeded the 
HandleHttpRequest hangs up. In this specific issue that time out also was 
reached but it seems that the flow never got beyond the HandleHttpRequest.

From: William Gosse 
Sent: Tuesday, September 10, 2019 5:14 PM
To: users@nifi.apache.org
Subject: RE: HandleHttpRequest - Failed to receive content from HTTP Request

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.]

[WARNING: Potential spoofed mail]
The sender of this email used an @kodakalaris.com email address, but the email 
did not come from a known Kodak Alaris email system. This email might not be 
legitimate. Delete this email unless it was sent from a known, trusted source.
To avoid this notice if this email was sent from a known trusted source please 
submit an IT_KAre request to have this sender added as a valid email sender.

Proofpoint rule module.access.rule.anti_spoofing_rule Message ID 2uv82r5u4t-1 
2uv82r5u4t
I’m only seeing it with one user and I’m not sure how they are doing it.

From: Pierre Villard 
mailto:pierre.villard...@gmail.com>>
Sent: Tuesday, September 10, 2019 8:32 AM
To: users@nifi.apache.org
Subject: Re: HandleHttpRequest - Failed to receive content from HTTP Request

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.]

I'm AFK for the moment but could you file a JIRA with some steps to reproduce 
the issue? Happy to have a look once I get back at a computer.

Le mar. 10 sept. 2019 à 08:09, William Gosse 
mailto:william.go...@aifoundry.com>> a écrit :
It’s 1.9.2

From: Pierre Villard 
mailto:pierre.villard...@gmail.com>>
Sent: Monday, September 9, 2019 7:43 PM
To: users@nifi.apache.org
Subject: Re: HandleHttpRequest - Failed to receive content from HTTP Request

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.]

Hey William,

Which version of NiFi are you using?

Pierre

Le lun. 9 sept. 2019 à 16:20, William Gosse 
mailto:william.go...@aifoundry.com>> a écrit :
I have had couple occurrence of the following exception occurring on an 
HandleHttpRequest pprocessor:
nifi-app_2019-09-06_12.0.log:2019-09-06 12:24:48,132 ERROR [Timer-Driven 
Process Thread-3] o.a.n.p.standard.HandleHttpRequest 
HandleHttpRequest[id=6ceef915-4430-30fa-09d2-b12bb2142172] Failed to receive 
content from HTTP Request from 108.26.163.22 due to java.io.IOException: 
java.util.concurrent.TimeoutException: Idle timeout expired: 61/60 ms: 
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout 
expired: 61/60 ms

When it occurs the HandleHttpRequest processor stops excepting requests and I 
have to restart nifi in order to recover.

Is there anything I can do to better handle this exception?

Re: How to lock mysql a table for concurrent reads

[Eric Chaves]

Hi folks, found my error. I fixed my query to use an inner join with a
select for update instead of a subselect and configured the PutSQL to use a
batch count of 1.

Thanks!

Em qui, 12 de set de 2019 às 19:10, Eric Chaves  escreveu:

> Hi folks,
>
> I'm having a hard time to figure out how to do an update inside an atomic
> transaction with NiFi.
>
> In MySQL I use a table to control what is kind of a "record counter"
> distributed among files. The table looks like this:
>
> *create table file_records (*
>
> * file_id BIGINT NOT NULL AUTO_INCREMENT NOT NULL PRIMARY KEY,*
> * file_name VARCHAR(255) NOT NULL,*
>
>
> * first_id BIGINT NOT NULL DEFAULT 0, last_id BIGINT NOT NULL DEFAULT 0,*
> *)*
>
> The idea is that when a file arrives, given that I know how many lines it
> has I'll allocate a range of ids (first_id, last_id) equal to the amount of
> lines in advance, allowing me to process multiple files in parallel without
> the risk of having record ids collisions (the records/line will be inserted
> in another database that does not have auto increment fields).
>
> For this to work I planned to use an update query like this in an acid
> transaction with full table lock:
>
>
> *update tbl_name*
>
>
> *set first_id = (select max(last_id) + 1),last_id = (select max(last_id) +
> ${record_count})where campaign_id = ${sql.generated.key};*
>
> To test this concept I've tried multiple variations of a PutSQL
> configuration processor with 4 concurrent task neither with success (I'll
> expect to have records ).
>
> My last attempt was to setup a sequence of PutSQL doing a LOCK TABLE
> tbl_name; / UPDATE  / UNLOCK TABLE but still haven't got it working.
> When a processor tries to access the locked table it generates an error
> instead of waiting for the lock to be released.
>
> Has anyone ever done something similar in NiFi? What would be the proper
> way to perform each update inside an atomic transaction using a full table
> lock ?
>
> Thanks in advance?
>

Issue with empty files in content repository

[Malthe]

Trying to figure out why a `MergeContent` processor was producing a
linearly rising amount of content which wasn't reaped correctly (the
retention policies would not be upheld and disk space would fall to
zero), we realized that some flow files in the queue pointed to
content which didn't exist on disk. The file in the content repository
was zero bytes.

How might this have happened and if it happens, shouldn't processors
somehow be able to recover from it?

What seems to happen is that the flow file goes right back into the
queue where it will of course fail again. Further, a simple grep seems
to show that references to the empty content file id appears in many
other files in the content repository. This seems to suggest that all
this content can't be reaped because there it is still being
referenced somehow and thus isn't applicable for archival and/or
deletion.

Thanks for any ideas.