ClassNotFound LogSupport using ExtractEmailHeaders in v1.10.0

[Contacto Control Cobros]

Hi all,

I am using Apache Nifi 1.10, running with Java 11 and I had an error using
the "ExtractEmailHeaders" processor:


*WARN [Timer-Driven Process Thread-7]
o.a.n.controller.tasks.ConnectableTask Administratively Yielding
ExtractEmailHeaders[id=aabe0e09-016f-1000-02c0-2bb7e4f1c5b8] due to
uncaught Exception: java.lang.NoClassDefFoundError:
com/sun/activation/registries/LogSupportjava.lang.NoClassDefFoundError:
com/sun/activation/registries/LogSupport*

This seems to happen because in the */nifi-1.10.0/lib/java11* folder there
is no JAR file that has the class that was removed in Java 11 (
https://blog.codefx.org/java/java-11-migration-guide/#Removal-Of-Java-EE-Modules
).

I had to remove the file "javax.activation-api-1.2.0.jar" and add the
javax.activation-1.2.0.jar (which contains the com.sun.activation package).
For now, that seems to solve the problem.

Re: jolt array name to attribute?

[Matt Burgess]

Try the following chain spec:

[
  {
"operation": "shift",
"spec": {
  "*": {
"$": "type",
"*": {
  "*": "&"
}
  }
}
  }
]

This will "flatten" any key with an array value into a "type" field
with the name of the array key, and all its entries as top-level
fields.

Regards,
Matt


On Wed, Jan 15, 2020 at 2:52 PM l vic  wrote:
>
> i have input json such as the following example:
> {
>  "Address": [
> {
>  "Street": "Washington St",
>  "ZIP": "03420",
>  "unit": "1"
> }
> ]
> }
> How can i use Jolt transform to add array name as attribute value to
> array members as following result:
> {
> "type": "Address",
> "Street": "Washington St",
>  "ZIP": "03420",
>  "unit": "1"
> }
> Thank you,

Re: Holiday scheduling

[Mark Payne]

A drools engine may be a good solution. But if you wanted to lookup against a 
database or something like that, you could look at the LookupAttribute 
processor. It's configured with a LookupService, and there are several 
different services that you can use to lookup the value. Including 
SimpleDatabaseLookupService. Or a scripted service, so you could use Groovy or 
what have you to make a restful call or something like that to determine if 
it's a holiday.

> On Jan 15, 2020, at 1:52 PM, Andrew Grande  wrote:
> 
> Maybe a good fit for a drools engine rule set? I remember there was a 
> community processor.
> 
> Andrew
> 
> On Wed, Jan 15, 2020, 10:40 AM Dave Andrews  > wrote:
> Hello community, 
> Has anyone come up with a clean internal to NiFi solution to changing 
> workflow based on date/time?  For example, a routeonAttribute that will route 
> files differently for the upcoming weekend and Monday holiday?  I have 
> written different routing strategies like:
> 
> ${filename:toUpper():startsWith('XYZ')
> :and(${now():toNumber():format('MM'):equals('01')
> :and(${now():toNumber():format('dd'):equals('20')
> :and(${now():toNumber():format('HH'):lt('17')
> :and(${now():toNumber():format(''):equals('2020')
> 
>   But I'm curious if someone came up with a more elegant which is more 
> maintainable - I don't want to maintain a calendar.  
> 
> Should I query a shared database, or is there an online calendar I could 
> invoke http against?   or??
> 
> thanks --da

jolt array name to attribute?

[l vic]

i have input json such as the following example:
{
 "Address": [
{
 "Street": "Washington St",
 "ZIP": "03420",
 "unit": "1"
}
]
}
How can i use Jolt transform to add array name as attribute value to
array members as following result:
{
*"type": "Address"*,
"Street": "Washington St",
 "ZIP": "03420",
 "unit": "1"
}
Thank you,

Re: Holiday scheduling

[Andrew Grande]

Maybe a good fit for a drools engine rule set? I remember there was a
community processor.

Andrew

On Wed, Jan 15, 2020, 10:40 AM Dave Andrews 
wrote:

> Hello community,
> Has anyone come up with a clean internal to NiFi solution to changing
> workflow based on date/time?  For example, a routeonAttribute that will
> route files differently for the upcoming weekend and Monday holiday?  I
> have written different routing strategies like:
>
> ${filename:toUpper():startsWith('XYZ')
> :and(${now():toNumber():format('MM'):equals('01')
> :and(${now():toNumber():format('dd'):equals('20')
> :and(${now():toNumber():format('HH'):lt('17')
> :and(${now():toNumber():format(''):equals('2020')
>
>   But I'm curious if someone came up with a more elegant which is more
> maintainable - I don't want to maintain a calendar.
>
> Should I query a shared database, or is there an online calendar I could
> invoke http against?   or??
>
> thanks --da
>

Holiday scheduling

[Dave Andrews]

Hello community,
Has anyone come up with a clean internal to NiFi solution to changing
workflow based on date/time?  For example, a routeonAttribute that will
route files differently for the upcoming weekend and Monday holiday?  I
have written different routing strategies like:

${filename:toUpper():startsWith('XYZ')
:and(${now():toNumber():format('MM'):equals('01')
:and(${now():toNumber():format('dd'):equals('20')
:and(${now():toNumber():format('HH'):lt('17')
:and(${now():toNumber():format(''):equals('2020')

  But I'm curious if someone came up with a more elegant which is more
maintainable - I don't want to maintain a calendar.

Should I query a shared database, or is there an online calendar I could
invoke http against?   or??

thanks --da

RE: DistributedMapCacheService - automated creation

[William Gosse]

That was a very helpful suggestion. Here's what I came up with which seems to 
work:
#!/bin/bash
set -x

NIFI_ENCOMPASS_URL=http://xxx.xxx.xx.xxx:16080/nifi-api


ROOT_PROCESS=$(curl $NIFI_ENCOMPASS_URL/flow/process-groups/root)
ROOT_ID=`echo $ROOT_PROCESS | jq --raw-output '.processGroupFlow.id'`
echo "Nifi Encompass Connector Root ID = $ROOT_ID"

response=$(curl 
$NIFI_ENCOMPASS_URL/flow/process-groups/$ROOT_ID/controller-services)
controllerServices=`echo $response | jq --raw-output '.controllerServices'`
if [[ $controllerServices == '[]' ]]; then
echo "No existing controllerServices"
createService=true
else
if [[ $controllerServices == *"DistributedMapCacheServer"* ]]; then
echo "DistributedMapCacheServer exists"
else
echo "DistributedMapCacheServer does not exists"
createService=true
fi
fi

if [ "$createService" = true ]; then
clientId=$(curl $NIFI_ENCOMPASS_URL/flow/client-id)
curl -d 
'{"revision":{"clientId":"{$clientId}","version":0},"disconnectedNodeAcknowledged":false,"component":{"type":"org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer","bundle":{"group":"org.apache.nifi","artifact":"nifi-distributed-cache-services-nar","version":"1.10.0"}}}'
 -H "Content-Type: application/json" -X POST 
$NIFI_ENCOMPASS_URL/process-groups/$ROOT_ID/controller-services
fi

-Original Message-
From: Bryan Bende  
Sent: Wednesday, January 15, 2020 9:34 AM
To: users@nifi.apache.org
Subject: Re: DistributedMapCacheService - automated creation

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.] 

Anything you can do from the UI goes through the REST API. You can open Chrome 
Dev Tools and see what calls are made to the server when you create the 
controller service from the UI.

On Wed, Jan 15, 2020 at 9:25 AM William Gosse  
wrote:
>
> Well we will be deploy Nifi in a docker container and need to configure 
> everything at the time of the container’s creation and startup.
>
>
>
> To be specific I’m using DistributedMapCacheService with a DetectDuplicate 
> process to track whether a flowfile with a same attribute value had been 
> already created in the last 10 minutes. If so abandon the new flowfile.
>
>
>
> I guess the thin I liked about the DistributedMapCacheService is that it 
> already exists in NiFi, no additional installation required.
>
>
>
>
>
> From: Mike Thomsen 
> Sent: Tuesday, January 14, 2020 7:42 PM
> To: users@nifi.apache.org
> Subject: Re: DistributedMapCacheService - automated creation
>
>
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not 
> click links or open attachments unless you recognize the sender and 
> know the content is safe.]
>
> 
>
> What is your use case? Because this sounds vaguely like something that you 
> could implement implicitly with the HBase version by just creating one big 
> cache table with an eye for not overlapping column qualifiers and possibly 
> liberal use of HBase TTLs.
>
>
>
> On Tue, Jan 14, 2020 at 6:04 PM William Gosse  
> wrote:
>
> Is there any way to automate the creation of a 
> DistributedMapCacheService
>
>
>
> I was hoping I could do this via a template but it doesn’t to get exported 
> the way the  DistributedMapCacheClientService does.
>
>
>
> The client got created just fine but there’s no service.
>
>
>
> I don’t see away to do this with restful api either.
>
>
>
> Any help would be greatly appreciated.
>
>
>
>

Re: code page conversion

[Mike Thomsen]

Looking on Google, I found a number of people having problems with Java
converting back and forth between UTF-8 and that Hebrew ISO code page. For
example:

https://bugs.openjdk.java.net/browse/JDK-8195686

Depending on what JRE you're using, you might need to look very carefully
at how you're declaring the code page in NiFi to make sure that the JRE
itself is not having issues with mapping your input to a known character
set. I saw a PR on JavaMail also that was rejected because it manually
mapped some Hebrew code pages issues that folks felt were better fit into
the JRE.

‪On Wed, Jan 15, 2020 at 7:32 AM ‫סמדג'ה גיא‬‎ 
wrote:‬

> Thanks for answering
>
> The data looks readable all the way, until it enters the oracle DB
>
> Other tools like IBM DataStage handles this kind of data correctly, so I
> guess the problem relays on the NIFI side
>
> As I wrote , oracle is configure with ISO8859-8 code page ,
>
> so I need to do some kind of code page translation before oracle – I tried
> it with ConvertCharacterSet
> 
>  with
> no luck yet
>
>
>
>
>
>
>
>
>
>
>
> *From:* Mike Thomsen 
> *Sent:* Wednesday, January 15, 2020 2:05 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: code page conversion
>
>
>
> I'd dump a few of those records to disk with PutFile or view them with the
> content viewer in the NiFi flow. If you see problems within NiFi, you can
> add a new parameter for this in conf/bootstrap.conf:
>
>
>
> -Dfile.encoding=UTF-8
>
>
>
> If it looks good, you need to look at how Oracle is configured because
> it's probably set up to convert to that iso character set instead of
> keeping it as UTF-8
>
>
>
> ‪On Wed, Jan 15, 2020 at 1:24 AM ‫סמדג'ה גיא‎ 
> wrote:
>
> Hi all
>
> I have a code page conversion issue
>
> I have a flow that reads records from mongoDB and inserts the result into
> oracle
>
> The mongo encoding is utf-8 , and the oracle is iso8859-8
>
> Some special characters are not shown correctly in oracle – shows ???
>
> I tried using the convertcharacterset function before inserting into
> oracle, but with no luck (same ???)
>
> And ideas?
>
>
>
>
>
> Thanks in advanced
>
>
>
> *Guy Smadga*
>
> *ETL infrastructure team leader*
>
> *Israel discount bank*
>
>
>
>
>
>
>
> הודעה זו וכל מסמך הנלווה לה (להלן: "המידע"), הינו שירות דוא"ל מבנק
> דיסקונט. המידע מיועד אך ורק לנמען הרשום בה ולא לשום אדם או גוף זולתו
> ועשויים להכיל מידע סודי וחסוי. יש לעדכן מיידית אודות כל שינוי בכתובת הדואר
> האלקטרוני ו/או במספר טלפון הנייד, באתר הבנק, באפליקציה, בסניפך או בטלבנק
> דיסקונט 6111*. ניתן לבטל את השירות בכל עת, מכל סיבה שהיא.
>
> אם אינך הנמען הרשום בהודעה, אנא צור קשר עם טלבנק דיסקונט 6111*, כמו כן הנך
> מתבקש להימנע מלעשות כל שימוש, העתקה, פרסום, הפצה, גילוי או העברה של המידע
> או כל חלק ממנו, ולהשמיד את המידע וכל ההעתקים שלו.
>
> This email was scanned by Votiro
>
> הודעה זו וכל מסמך הנלווה לה (להלן: "המידע"), הינו שירות דוא"ל מבנק
> דיסקונט. המידע מיועד אך ורק לנמען הרשום בה ולא לשום אדם או גוף זולתו
> ועשויים להכיל מידע סודי וחסוי. יש לעדכן מיידית אודות כל שינוי בכתובת הדואר
> האלקטרוני ו/או במספר טלפון הנייד, באתר הבנק, באפליקציה, בסניפך או בטלבנק
> דיסקונט 6111*. ניתן לבטל את השירות בכל עת, מכל סיבה שהיא.
> אם אינך הנמען הרשום בהודעה, אנא צור קשר עם טלבנק דיסקונט 6111*, כמו כן הנך
> מתבקש להימנע מלעשות כל שימוש, העתקה, פרסום, הפצה, גילוי או העברה של המידע
> או כל חלק ממנו, ולהשמיד את המידע וכל ההעתקים שלו.
>

Re: DistributedMapCacheService - automated creation

[Bryan Bende]

Anything you can do from the UI goes through the REST API. You can
open Chrome Dev Tools and see what calls are made to the server when
you create the controller service from the UI.

On Wed, Jan 15, 2020 at 9:25 AM William Gosse
 wrote:
>
> Well we will be deploy Nifi in a docker container and need to configure 
> everything at the time of the container’s creation and startup.
>
>
>
> To be specific I’m using DistributedMapCacheService with a DetectDuplicate 
> process to track whether a flowfile with a same attribute value had been 
> already created in the last 10 minutes. If so abandon the new flowfile.
>
>
>
> I guess the thin I liked about the DistributedMapCacheService is that it 
> already exists in NiFi, no additional installation required.
>
>
>
>
>
> From: Mike Thomsen 
> Sent: Tuesday, January 14, 2020 7:42 PM
> To: users@nifi.apache.org
> Subject: Re: DistributedMapCacheService - automated creation
>
>
>
> [CAUTION: This email originated from outside of Kodak Alaris. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.]
>
> 
>
> What is your use case? Because this sounds vaguely like something that you 
> could implement implicitly with the HBase version by just creating one big 
> cache table with an eye for not overlapping column qualifiers and possibly 
> liberal use of HBase TTLs.
>
>
>
> On Tue, Jan 14, 2020 at 6:04 PM William Gosse  
> wrote:
>
> Is there any way to automate the creation of a DistributedMapCacheService
>
>
>
> I was hoping I could do this via a template but it doesn’t to get exported 
> the way the  DistributedMapCacheClientService does.
>
>
>
> The client got created just fine but there’s no service.
>
>
>
> I don’t see away to do this with restful api either.
>
>
>
> Any help would be greatly appreciated.
>
>
>
>

Merge and transform JSON

[Tomislav Novosel]

Hi Nifi Community,

I have situation where I need to merge JSON content from multiple flowfiles
into one JSON content (single flowfile).

JSON in every flowfile looks like this:

{
  "analyses": "prep_array",
  "args": "prep_array",
  "scriptId": "142",
  "libIds": "141",
  "job_name": "my_demo_job",
  "project_id": "23",
  "configId": "5",
  "file_id": "150",
  "analysis_name": "my_analysis_name",
  "source_id": "12",
  "customer_id": "16"
}

and every flowfile has only one filed different from the others, that is
file_id. I want to merge it all in one JSON but to get this structure:

{
  "analyses": "prep_array",
  "args": "prep_array",
  "scriptId": "142",
  "libIds": "141",
  "job_name": "my_demo_job",
  "project_id": "23",
  "configId": "5",
  "file_id": [
150,
151,
152,
153,
154
  ],
  "analysis_name": "my_analysis_name",
  "source_id": "12",
  "customer_id": "16"
}

What is the best way to do it? It sounds to me a little bit tricky, but I
hope I'm wrong.

Please can someone give me advice or some guidelines?

Thanks in advance,
Tom

RE: DistributedMapCacheService - automated creation

[William Gosse]

Well we will be deploy Nifi in a docker container and need to configure 
everything at the time of the container’s creation and startup.

To be specific I’m using DistributedMapCacheService with a DetectDuplicate 
process to track whether a flowfile with a same attribute value had been 
already created in the last 10 minutes. If so abandon the new flowfile.

I guess the thin I liked about the DistributedMapCacheService is that it 
already exists in NiFi, no additional installation required.


From: Mike Thomsen 
Sent: Tuesday, January 14, 2020 7:42 PM
To: users@nifi.apache.org
Subject: Re: DistributedMapCacheService - automated creation

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.]

What is your use case? Because this sounds vaguely like something that you 
could implement implicitly with the HBase version by just creating one big 
cache table with an eye for not overlapping column qualifiers and possibly 
liberal use of HBase TTLs.

On Tue, Jan 14, 2020 at 6:04 PM William Gosse 
mailto:william.go...@aifoundry.com>> wrote:
Is there any way to automate the creation of a DistributedMapCacheService

I was hoping I could do this via a template but it doesn’t to get exported the 
way the  DistributedMapCacheClientService does.

The client got created just fine but there’s no service.

I don’t see away to do this with restful api either.

Any help would be greatly appreciated.

RE: code page conversion

[סמדג'ה גיא]

Thanks for answering
The data looks readable all the way, until it enters the oracle DB
Other tools like IBM DataStage handles this kind of data correctly, so I guess 
the problem relays on the NIFI side
As I wrote , oracle is configure with ISO8859-8 code page ,
so I need to do some kind of code page translation before oracle – I tried it 
with ConvertCharacterSet 

  with no luck yet





From: Mike Thomsen 
Sent: Wednesday, January 15, 2020 2:05 PM
To: users@nifi.apache.org
Subject: Re: code page conversion

I'd dump a few of those records to disk with PutFile or view them with the 
content viewer in the NiFi flow. If you see problems within NiFi, you can add a 
new parameter for this in conf/bootstrap.conf:

-Dfile.encoding=UTF-8

If it looks good, you need to look at how Oracle is configured because it's 
probably set up to convert to that iso character set instead of keeping it as 
UTF-8

‪On Wed, Jan 15, 2020 at 1:24 AM ‫סמדג'ה גיא‎ 
mailto:guy.sma...@dbank.co.il>> wrote:
Hi all
I have a code page conversion issue
I have a flow that reads records from mongoDB and inserts the result into oracle
The mongo encoding is utf-8 , and the oracle is iso8859-8
Some special characters are not shown correctly in oracle – shows ???
I tried using the convertcharacterset function before inserting into oracle, 
but with no luck (same ???)
And ideas?


Thanks in advanced

Guy Smadga
ETL infrastructure team leader
Israel discount bank



הודעה זו וכל מסמך הנלווה לה (להלן: "המידע"), הינו שירות דוא"ל מבנק דיסקונט. 
המידע מיועד אך ורק לנמען הרשום בה ולא לשום אדם או גוף זולתו ועשויים להכיל מידע 
סודי וחסוי. יש לעדכן מיידית אודות כל שינוי בכתובת הדואר האלקטרוני ו/או במספר 
טלפון הנייד, באתר הבנק, באפליקציה, בסניפך או בטלבנק דיסקונט 6111*. ניתן לבטל את 
השירות בכל עת, מכל סיבה שהיא.
אם אינך הנמען הרשום בהודעה, אנא צור קשר עם טלבנק דיסקונט 6111*, כמו כן הנך 
מתבקש להימנע מלעשות כל שימוש, העתקה, פרסום, הפצה, גילוי או העברה של המידע או כל 
חלק ממנו, ולהשמיד את המידע וכל ההעתקים שלו.
This email was scanned by Votiro


הודעה זו וכל מסמך הנלווה לה (להלן: "המידע"), הינו שירות דוא"ל מבנק דיסקונט. 
המידע מיועד אך ורק לנמען הרשום בה ולא לשום אדם או גוף זולתו ועשויים להכיל מידע 
סודי וחסוי. יש לעדכן מיידית אודות כל שינוי בכתובת הדואר האלקטרוני ו/או במספר 
טלפון הנייד, באתר הבנק, באפליקציה, בסניפך או בטלבנק דיסקונט 6111*. ניתן לבטל את 
השירות בכל עת, מכל סיבה שהיא. 

אם אינך הנמען הרשום בהודעה, אנא צור קשר עם טלבנק דיסקונט 6111*, כמו כן הנך 
מתבקש להימנע מלעשות כל שימוש, העתקה, פרסום, הפצה, גילוי או העברה של המידע או כל 
חלק ממנו, ולהשמיד את המידע וכל ההעתקים שלו.

Re: code page conversion

[Mike Thomsen]

I'd dump a few of those records to disk with PutFile or view them with the
content viewer in the NiFi flow. If you see problems within NiFi, you can
add a new parameter for this in conf/bootstrap.conf:

-Dfile.encoding=UTF-8

If it looks good, you need to look at how Oracle is configured because it's
probably set up to convert to that iso character set instead of keeping it
as UTF-8

‪On Wed, Jan 15, 2020 at 1:24 AM ‫סמדג'ה גיא‬‎ 
wrote:‬

> Hi all
>
> I have a code page conversion issue
>
> I have a flow that reads records from mongoDB and inserts the result into
> oracle
>
> The mongo encoding is utf-8 , and the oracle is iso8859-8
>
> Some special characters are not shown correctly in oracle – shows ???
>
> I tried using the convertcharacterset function before inserting into
> oracle, but with no luck (same ???)
>
> And ideas?
>
>
>
>
>
> Thanks in advanced
>
>
>
> *Guy Smadga*
>
> *ETL infrastructure team leader*
>
> *Israel discount bank*
>
>
>
>
>
> הודעה זו וכל מסמך הנלווה לה (להלן: "המידע"), הינו שירות דוא"ל מבנק
> דיסקונט. המידע מיועד אך ורק לנמען הרשום בה ולא לשום אדם או גוף זולתו
> ועשויים להכיל מידע סודי וחסוי. יש לעדכן מיידית אודות כל שינוי בכתובת הדואר
> האלקטרוני ו/או במספר טלפון הנייד, באתר הבנק, באפליקציה, בסניפך או בטלבנק
> דיסקונט 6111*. ניתן לבטל את השירות בכל עת, מכל סיבה שהיא.
> אם אינך הנמען הרשום בהודעה, אנא צור קשר עם טלבנק דיסקונט 6111*, כמו כן הנך
> מתבקש להימנע מלעשות כל שימוש, העתקה, פרסום, הפצה, גילוי או העברה של המידע
> או כל חלק ממנו, ולהשמיד את המידע וכל ההעתקים שלו.
>