Rolling restart of Nifi

[Jorge Machado]

Hi Guys, 

we are deploying some nar files on the cluster via an automated process. 
What is the best way to restart a node without having problems with the 
flowFile.gz at start that does not match the master ? 
I looked at the api and it would be nice to have a method to disconnect a node 
from the cluster via hostname.

is this possible ? 

Thanks 

Jorge Machado

Is this possible to create table from JSON?

[prabhu Mahendran]

Hi All,

I can able to get avro schema from json.

It could be look like below.

{
"type" : "array",
"items" : {
"type" : "record",
"name" : "AvroSchema",
"fields" : [
{
"name" : "Name",
"type" : "String",
"doc" : "Type inferred from 'S'"
},
{ "name" : "Age",
"type" : "int", "doc" : "Type inferred from '23'"
}
}
}

With help of that, i should extract fields(name & type) and create table in
DB from that schema.

Can any one suggest me the best way to create table in SQL Server from NiFi
based on avro schema?

Re: Create nested records

[Koji Kawamura]

Hi Charlie,

Thanks for sharing the template.
Following configurations for UpdateRecord did the flat to nested mapping:

- Replacement Value Strategy: Record Path Value
- Dynamic property: "/phone" = "/"

It maps the flat record into /phone child record.
Fields those are not included in the outgoing schema will be discarded.
But it would work only if nested fields have the same name.

Example result:
[ {
  "person" : "john",
  "gender" : "m",
  "phone" : {
"phone1" : "123-456-7890",
"phone1type" : "mobile",
"phone2" : "234-567-8901",
"phone2type" : "home"
  }
} ]

If the mapping gets more complicated than that, then I'd go with
ConvertRecord -> JoltTransformJSON.

Hope this helps,
Koji

On Sat, Feb 10, 2018 at 1:31 AM, Charlie Frasure
 wrote:
> Hi,
>
> I'm including a template here that mimics the problem I'm having.  I
> generate a one record csv and try to convert it to a JSON file with a nested
> section for phones.  The phone section of the JSON file converts as null
> instead of having the phone attributes.
> If we can resolve this problem, I would also like to map phone1 and phone2
> attributes to phone[0] ... phone[n].
>
> Can this be completed with the record writers?
>
> On Thu, Feb 8, 2018 at 10:25 PM, Charlie Frasure 
> wrote:
>>
>> Hi,
>>
>> I'm having trouble taking a flat record and building an avro schema that
>> lets me nest it.
>> I found this example schema, but after using a ConvertRecord or
>> UpdateRecord, I receive the equivalent of [{"parent" = null}]
>>
>> {"type": "record", "name": "CustomerInput", "namespace":
>> "org.apache.example", "fields": [
>> {"name": "id", "type": "string"},
>> {"name": "companyName", "type": ["null", "string"], "default":
>> null},
>> {"name": "revenue", "type": ["null", "string"], "default": null},
>> {"name" : "parent", "type" : [ "null",
>> {"type" : "record", "name" : "parent", "fields" : [
>> {"name" : "name", "type" : ["null", "string"], "default" :
>> null},
>> {"name" : "id", "type" : "string"}
>> ] }
>> ], "default" : null }
>> ]}
>>
>> This example found online is meant to union a person schema with an
>> address schema, creating an "Address" type, but this resulted in a "Could
>> not parse incoming data" error.
>>
>> [
>>   {"type": "record", "name": "Address", "fields": [
>>   {"name": "streetaddress", "type": "string"},
>>   {"name": "city", "type": "string"}
>>   ]},
>>   {"type": "record", "name": "person", "fields": [
>>   {"name": "firstname","type": "string"},
>>   {"name": "lastname", "type": "string"},
>>   {"name": "address", "type": "Address"}
>>   ]}
>> ]
>>
>> I would also like to be able to include multiples of these sub-records as
>> well, such that I could have 0 to n addresses with distinct address types
>> associated to one person.
>> Is this possible with the record processors, or do I need to focus on Jolt
>> / ConvertAvro?
>>
>> Thanks,
>> Charlie
>>
>>
>

Re: JoltJsonTransform question

[Matt Burgess]

Austin,

This one works for your sample data but doesn't extend to more fields
unless you keep repeating the pattern in the spec:

[
  {
"operation": "shift",
"spec": {
  "@Place1": "TestArray[0].Place",
  "@Holder1": "TestArray[0].Holder",
  "@Place2": "TestArray[1].Place",
  "@Holder2": "TestArray[1].Holder"
}
  }
]

Basically I don't know of a way to get the "1" from "Place1" into a
index that can be used in the target.

Regards,
Matt

On Tue, Feb 13, 2018 at 5:32 PM, Austin Duncan  wrote:
> All, I am trying to transfrom a json. I want the final to look like:
> "TestArray": [{
> "Place": "Test1",
> "Holder": "Test2"
> },
> {
> "Place": "Test3",
> "Holder": "Test4"
> }
> My source Json is flat with these specs:
>  "Place1" : "Test1",
>   "Holder1" : "Test2",
>   "Place2" : "Test3",
>   "Holder2" : "Test4"
>
> With this transform:
>
> [
>   {
> "operation": "shift",
> "spec": {
>   "@Place1": "TestArray[].Place",
>   "@Holder1": "TestArray[].Holder",
>   "@Place2": "TestArray[].Place",
>   "@Holder2": "TestArray[].Holder"
> }
>   }
> ]
>
> I get the following result:
>
> {
>   "TestArray" : [ {
> "Payer" : "Test1"
>   }, {
> "PolicyNumber" : "Test2"
>   }, {
> "Payer" : "Test3"
>   }, {
> "PolicyNumber" : "Test4"
>   } ]
> }
>
> Does anyone have any insight as to how to fix my problem? I am at a loss
> here and Am not quite sure where to go.
>

JoltJsonTransform question

[Austin Duncan]

All, I am trying to transfrom a json. I want the final to look like:
"TestArray": [{
"Place": "Test1",
"Holder": "Test2"
},
{
"Place": "Test3",
"Holder": "Test4"
}
My source Json is flat with these specs:
 "Place1" : "Test1",
  "Holder1" : "Test2",
  "Place2" : "Test3",
  "Holder2" : "Test4"

With this transform:

[
  {
"operation": "shift",
"spec": {
  "@Place1": "TestArray[].Place",
  "@Holder1": "TestArray[].Holder",
  "@Place2": "TestArray[].Place",
  "@Holder2": "TestArray[].Holder"
}
  }
]

I get the following result:

{
  "TestArray" : [ {
"Payer" : "Test1"
  }, {
"PolicyNumber" : "Test2"
  }, {
"Payer" : "Test3"
  }, {
"PolicyNumber" : "Test4"
  } ]
}

Does anyone have any insight as to how to fix my problem? I am at a loss
here and Am not quite sure where to go.

Re: minifi secure connection

[Marc]

Arne,
  I took a break from the issue and came back and tried installing a
different version of openssl on top of the distro. When doing so it linked
properly and I'm able to send data through a secure Socket. Now that I have
a solution, I will move this discussion to the ticket.

  As a result of my testing, I will make updates to the bootstrap script
and build instructions to instruct users to install libssl1.0 on Debian
Stretch ( and perhaps Raspbian ). Any comments on the ticket will be
appreciated: https://issues.apache.org/jira/browse/MINIFICPP-400 . I will
have a fix once I finish testing across a few platforms.

  Thanks,
  Marc

On Tue, Feb 13, 2018 at 9:55 AM, Marc P.  wrote:

> Arne,
>
> Thanks for the info. I'm running the same environment with the same
> warnings produced -- and segfault -- aso I'll get back to you once I've
> identified the issue.
>
> TL;DR: Created a ticket ( https://issues.apache.org/
> jira/browse/MINIFICPP-400. ) to help insulate users from this more.
>
> More explanation:
>
>
>   In regards to the different versions, there were a number of tickets on
> Debian's bug report logs regarding the curl ABI. An example [1] was solved
> by changing linked versions of libraries.
>
>   The gist is that the SSL_CTX struct changes. I've validated with gdb
> that the struct is being passed in on U16 ( and works ) AND Debian stretch
> ( does not work ). The structures are slightly different. I'm going to dive
> deeper into this. I've created [2] https://issues.apache.org/
> jira/browse/MINIFICPP-400.
>
>[2] has more recent conversation as the issue still exists. This will
> require an soname change hence why they've likely been discussing this for
> two years. I think our job will be to insulate users, so [3] will be our
> efforts to do so. Thanks for identifying this. I'll continue to investigate
> this to address it seamlessly through either our bootstrap script (
> bootstrap.sh in the root ) or within CMAKE. Obviously we don't want to
> alienate Debian users.
>
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844018
> [2] https://issues.apache.org/jira/browse/MINIFICPP-400.
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858398
>
> On Tue, Feb 13, 2018 at 8:28 AM, Arne Oslebo 
> wrote:
>
>> Marc,
>>
>> I'm running it on Debian Stretch. libssl version might be the problem. I
>> see that both libssl1.0.2 and libssl1.1 are installed. I took another look
>> at the build log and found this:
>> /usr/bin/ld: warning: libssl.so.1.0.2, needed by
>> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
>> may conflict with libssl.so.1.1
>> /usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
>> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
>> may conflict with libcrypto.so.1.1
>>
>> I'll see if can remove one of the versions to avoid possible conflict.
>>
>> The full backtrace log is attached and thanks for looking into this,
>>
>> Arne
>>
>>
>>
>> On 13/02/2018 12:55, Marc wrote:
>>
>> Arne,
>>Sorry for your troubles! Can you give me some insight into your distro?
>>
>>I've been unable to replicate the issue on OSX, u16, and arch...but
>> all are running a different version of OpenSSL. What version of OpenSSL are
>> you running and on what distro? I'll be happy to try it to track this down.
>>
>>Additionally, do you have the log file you can pass on? The reason I
>> ask is that the line specified in gdb is a log statement with usage of
>> constructs that are inherent to libstdc++, so they won't cause a memory
>> error. The log file should help identify this and give me insight into what
>> occurred just before the error.
>>
>>Thanks,
>>Marc
>>
>> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo 
>> wrote:
>>
>>> Hello Mark,
>>>
>>> thanks for the update. I tried running master from github but
>>> unfortunately I now get a segmentation fault:
>>>
>>> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
>>> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> (gdb) bt full
>>> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
>>> sl.so.1.1
>>> No symbol table info available.
>>> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
>>> sl.so.1.1
>>> No symbol table info available.
>>> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> No symbol table info available.
>>> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>> No symbol table info available.
>>> #4  0x55ef91cb in org::apache::nifi::minifi::con
>>> trollers::SSLContextService::configure_ssl_context
>>> (this=0x569948b0, ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
>>> /extensions/http-curl/../../libminifi/include/controllers/SS
>>> LContextService.h:165
>>> retp = 1
>>> #5  0x55ef9be4 in org::apache::nifi::minifi::uti
>>> ls::HTTPClient::configur

Re: LDAP provider not recognizing the u/p combination

[Kevin Doran]

Hi Mike,

 

I don’t know enough about Active Directory and LDAP in general to answer your 
question off the type of my help, but I’m familiar with how the NiFi LDAP 
client is configured using the fields you’ve mentioned, so I may be able to 
help you figure it out.

 

I think you’re on the right track, but you may need to use the User Identity 
Attribute as well.

 

It would be helpful for me if I could try to reproduce the environment you are 
working in. As I don’t know the details of the Active Directory structure, 
would you be able to provide an example snippet of the directory in LDIF format 
[1] [2]? Please scrub any sensitive information (actual names or password 
hashes) before sending, I just need a better sense of the structure of the 
directory, not the value of fields themselves.

 

If that’s not possible for you, just let me know and I can try to follow up 
without those details as soon as I get a chance to dig into the specifics of AD 
a bit more.

 

Thanks,

Kevin

 

[1] https://support.microsoft.com/en-us/help/555636 

[2] https://docs.oracle.com/cd/A97630_01/network.920/a96579/comtools.htm#631224 

 

From: Mike Thomsen 
Reply-To: 
Date: Tuesday, February 13, 2018 at 11:18
To: 
Subject: LDAP provider not recognizing the u/p combination

 

We're using AD, and I have verified that we can actually pull the users and 
groups by logging in as the initial admin and checking the users. It shows the 
users and the LDAP groups we assigned. Looks fine there.

 

When a user goes to login with their domain account, it says invalid username 
and password.

 

So if their domain account is like this:

 

LOCALSITE\john.smith

 

I expect them to be able to put "john.smith" in the username field.

 

These are the search settings:

 

Search Filter: (CN={0})

Identity Strategy: USE_USERNAME

 

Based on the documentation, I would expect that that would take the username 
and password, put the username into the CN attribute of the search filter and 
filter on the search base (exact copy of the one that is working in the 
user/group search configuration).

 

Any suggestions on what might be wrong and/or how to debug this?

 

Thanks,

 

Mike

LDAP provider not recognizing the u/p combination

[Mike Thomsen]

We're using AD, and I have verified that we can actually pull the users and
groups by logging in as the initial admin and checking the users. It shows
the users and the LDAP groups we assigned. Looks fine there.

When a user goes to login with their domain account, it says invalid
username and password.

So if their domain account is like this:

LOCALSITE\john.smith

I expect them to be able to put "john.smith" in the username field.

These are the search settings:

Search Filter: (CN={0})
Identity Strategy: USE_USERNAME

Based on the documentation, I would expect that that would take the
username and password, put the username into the CN attribute of the search
filter and filter on the search base (exact copy of the one that is working
in the user/group search configuration).

Any suggestions on what might be wrong and/or how to debug this?

Thanks,

Mike

Re: minifi secure connection

[Marc P.]

Arne,

Thanks for the info. I'm running the same environment with the same
warnings produced -- and segfault -- aso I'll get back to you once I've
identified the issue.

TL;DR: Created a ticket (
https://issues.apache.org/jira/browse/MINIFICPP-400. ) to help insulate
users from this more.

More explanation:


  In regards to the different versions, there were a number of tickets on
Debian's bug report logs regarding the curl ABI. An example [1] was solved
by changing linked versions of libraries.

  The gist is that the SSL_CTX struct changes. I've validated with gdb that
the struct is being passed in on U16 ( and works ) AND Debian stretch (
does not work ). The structures are slightly different. I'm going to dive
deeper into this. I've created [2]
https://issues.apache.org/jira/browse/MINIFICPP-400.

   [2] has more recent conversation as the issue still exists. This will
require an soname change hence why they've likely been discussing this for
two years. I think our job will be to insulate users, so [3] will be our
efforts to do so. Thanks for identifying this. I'll continue to investigate
this to address it seamlessly through either our bootstrap script (
bootstrap.sh in the root ) or within CMAKE. Obviously we don't want to
alienate Debian users.


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844018
[2] https://issues.apache.org/jira/browse/MINIFICPP-400.
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858398

On Tue, Feb 13, 2018 at 8:28 AM, Arne Oslebo  wrote:

> Marc,
>
> I'm running it on Debian Stretch. libssl version might be the problem. I
> see that both libssl1.0.2 and libssl1.1 are installed. I took another look
> at the build log and found this:
> /usr/bin/ld: warning: libssl.so.1.0.2, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may
> conflict with libssl.so.1.1
> /usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may
> conflict with libcrypto.so.1.1
>
> I'll see if can remove one of the versions to avoid possible conflict.
>
> The full backtrace log is attached and thanks for looking into this,
>
> Arne
>
>
>
> On 13/02/2018 12:55, Marc wrote:
>
> Arne,
>Sorry for your troubles! Can you give me some insight into your distro?
>
>I've been unable to replicate the issue on OSX, u16, and arch...but all
> are running a different version of OpenSSL. What version of OpenSSL are you
> running and on what distro? I'll be happy to try it to track this down.
>
>Additionally, do you have the log file you can pass on? The reason I
> ask is that the line specified in gdb is a log statement with usage of
> constructs that are inherent to libstdc++, so they won't cause a memory
> error. The log file should help identify this and give me insight into what
> occurred just before the error.
>
>Thanks,
>Marc
>
> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo 
> wrote:
>
>> Hello Mark,
>>
>> thanks for the update. I tried running master from github but
>> unfortunately I now get a segmentation fault:
>>
>> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
>> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> (gdb) bt full
>> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
>> sl.so.1.1
>> No symbol table info available.
>> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
>> sl.so.1.1
>> No symbol table info available.
>> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> No symbol table info available.
>> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> No symbol table info available.
>> #4  0x55ef91cb in org::apache::nifi::minifi::con
>> trollers::SSLContextService::configure_ssl_context (this=0x569948b0,
>> ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
>> /extensions/http-curl/../../libminifi/include/controllers/SS
>> LContextService.h:165
>> retp = 1
>> #5  0x55ef9be4 in org::apache::nifi::minifi::uti
>> ls::HTTPClient::configure_ssl_context (curl=0x56a149e0,
>> ctx=0x56a28430, param=0x569948b0) at /usr/local/src/nifi-minifi-cpp
>> /extensions/http-curl/client/HTTPClient.h:177
>> ssl_context_service = 0x569948b0
>>
>> Any idea what the problem might be?
>>
>> My full config.yml:
>>
>> Flow Controller:
>>   name: MiNiFi Flow
>> Controller Services:
>> - name: SSLServiceName
>>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>>   class: SSLContextService
>>   Properties:
>>   Client Certificate: /opt/minifi/conf/client.pem
>>   Private Key: /opt/minifi/conf/client.key
>>   Passphrase: secret
>>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
>> Processors:
>> - id: cecb1868-9e5a-3e6c--
>>   name: TailFile
>>   class: org.apache.nifi.processors.standard.TailFile
>>   max concurrent tasks: 1
>>   

Re: minifi secure connection

[Arne Oslebo]

Marc,

I'm running it on Debian Stretch. libssl version might be the problem. I
see that both libssl1.0.2 and libssl1.1 are installed. I took another
look at the build log and found this:
/usr/bin/ld: warning: libssl.so.1.0.2, needed by
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
may conflict with libssl.so.1.1
/usr/bin/ld: warning: libcrypto.so.1.0.2, needed by
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so,
may conflict with libcrypto.so.1.1

I'll see if can remove one of the versions to avoid possible conflict.

The full backtrace log is attached and thanks for looking into this,

Arne


On 13/02/2018 12:55, Marc wrote:
> Arne,
>    Sorry for your troubles! Can you give me some insight into your distro?
>
>    I've been unable to replicate the issue on OSX, u16, and arch...but
> all are running a different version of OpenSSL. What version of
> OpenSSL are you running and on what distro? I'll be happy to try it to
> track this down. 
>
>    Additionally, do you have the log file you can pass on? The reason
> I ask is that the line specified in gdb is a log statement with usage
> of constructs that are inherent to libstdc++, so they won't cause a
> memory error. The log file should help identify this and give me
> insight into what occurred just before the error.
>
>    Thanks,
>    Marc
>
> On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo  > wrote:
>
> Hello Mark,
>
> thanks for the update. I tried running master from github but
> unfortunately I now get a segmentation fault:
>
> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
> 0x7777420a in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> (gdb) bt full
> #0  0x7777420a in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #1  0x77799681 in ?? () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #4  0x55ef91cb in
> 
> org::apache::nifi::minifi::controllers::SSLContextService::configure_ssl_context
> (this=0x569948b0, ctx=0x56a28430) at
> 
> /usr/local/src/nifi-minifi-cpp/extensions/http-curl/../../libminifi/include/controllers/SSLContextService.h:165
>     retp = 1
> #5  0x55ef9be4 in
> org::apache::nifi::minifi::utils::HTTPClient::configure_ssl_context
> (curl=0x56a149e0, ctx=0x56a28430, param=0x569948b0) at
> 
> /usr/local/src/nifi-minifi-cpp/extensions/http-curl/client/HTTPClient.h:177
>     ssl_context_service = 0x569948b0
>
> Any idea what the problem might be?
>
> My full config.yml:
>
> Flow Controller:
>   name: MiNiFi Flow
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/client.key
>   Passphrase: secret
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Processors:
> - id: cecb1868-9e5a-3e6c--
>   name: TailFile
>   class: org.apache.nifi.processors.standard.TailFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
>     File Location: Local
>     File to Tail: /tmp/test.log
>     Initial Start Position: Beginning of File
>     Rolling Filename Pattern:
>     tail-base-directory:
>     tail-mode: Single file
>     tailfile-lookup-frequency: 10 minutes
>     tailfile-maximum-age: 24 hours
>     tailfile-recursive-lookup: 'false'
> Connections:
> - id: 76ad4bc4-6557-3e23--
>   name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
>   source id: cecb1868-9e5a-3e6c--
>   source relationship names:
>   - success
>   destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
>   max work queue size: 1
>   max work queue data size: 1 GB
>   flowfile expiration: 0 sec
>   queue prioritizer class: ''
> Remote Processing Groups:
> - id: 3a25e1a3-c1b2-3e78--
>   name: ''
>   url: https://w.x.y.z:8443/nifi
>   comment: ''
>   timeout: 30 sec
>   yield period: 10 sec
>   transport protocol: RAW
>   proxy host: ''
>   proxy port: ''
>   proxy user: ''
>   proxy password: ''

Re: minifi secure connection

[Marc]

Arne,
   Sorry for your troubles! Can you give me some insight into your distro?

   I've been unable to replicate the issue on OSX, u16, and arch...but all
are running a different version of OpenSSL. What version of OpenSSL are you
running and on what distro? I'll be happy to try it to track this down.

   Additionally, do you have the log file you can pass on? The reason I ask
is that the line specified in gdb is a log statement with usage of
constructs that are inherent to libstdc++, so they won't cause a memory
error. The log file should help identify this and give me insight into what
occurred just before the error.

   Thanks,
   Marc

On Tue, Feb 13, 2018 at 6:17 AM, Arne Oslebo  wrote:

> Hello Mark,
>
> thanks for the update. I tried running master from github but
> unfortunately I now get a segmentation fault:
>
> Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
> 0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> (gdb) bt full
> #0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libs
> sl.so.1.1
> No symbol table info available.
> #1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libs
> sl.so.1.1
> No symbol table info available.
> #2  0x7777f2f6 in SSL_CTX_use_certificate () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> No symbol table info available.
> #4  0x55ef91cb in org::apache::nifi::minifi::con
> trollers::SSLContextService::configure_ssl_context (this=0x569948b0,
> ctx=0x56a28430) at /usr/local/src/nifi-minifi-cpp
> /extensions/http-curl/../../libminifi/include/controllers/
> SSLContextService.h:165
> retp = 1
> #5  0x55ef9be4 in org::apache::nifi::minifi::uti
> ls::HTTPClient::configure_ssl_context (curl=0x56a149e0,
> ctx=0x56a28430, param=0x569948b0) at /usr/local/src/nifi-minifi-cpp
> /extensions/http-curl/client/HTTPClient.h:177
> ssl_context_service = 0x569948b0
>
> Any idea what the problem might be?
>
> My full config.yml:
>
> Flow Controller:
>   name: MiNiFi Flow
> Controller Services:
> - name: SSLServiceName
>   id: 2438e3c8-015a-1000-79ca-83af40ec1974
>   class: SSLContextService
>   Properties:
>   Client Certificate: /opt/minifi/conf/client.pem
>   Private Key: /opt/minifi/conf/client.key
>   Passphrase: secret
>   CA Certificate: /opt/minifi/conf/nifi-cert.pem
> Processors:
> - id: cecb1868-9e5a-3e6c--
>   name: TailFile
>   class: org.apache.nifi.processors.standard.TailFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 0 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list: []
>   Properties:
> File Location: Local
> File to Tail: /tmp/test.log
> Initial Start Position: Beginning of File
> Rolling Filename Pattern:
> tail-base-directory:
> tail-mode: Single file
> tailfile-lookup-frequency: 10 minutes
> tailfile-maximum-age: 24 hours
> tailfile-recursive-lookup: 'false'
> Connections:
> - id: 76ad4bc4-6557-3e23--
>   name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
>   source id: cecb1868-9e5a-3e6c--
>   source relationship names:
>   - success
>   destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
>   max work queue size: 1
>   max work queue data size: 1 GB
>   flowfile expiration: 0 sec
>   queue prioritizer class: ''
> Remote Processing Groups:
> - id: 3a25e1a3-c1b2-3e78--
>   name: ''
>   url: https://w.x.y.z:8443/nifi
>   comment: ''
>   timeout: 30 sec
>   yield period: 10 sec
>   transport protocol: RAW
>   proxy host: ''
>   proxy port: ''
>   proxy user: ''
>   proxy password: ''
>   local network interface: ''
>   Input Ports:
>   - id: 56ae5abc-0161-1000-aa9e-c340d726e043
> name: Minifi
> comment: ''
> max concurrent tasks: 1
> use compression: false
> Properties:
>   Port: 10443
>   SSL Context Service: SSLServiceName
>   Host Name: w.x.y.z
>   Output Ports: []
> Provenance Reporting:
>
>
>
> On 09/02/2018 20:18, Marc wrote:
>
> Arne,
>   I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263 to
> address these issues.
>
> On Fri, Feb 9, 2018 at 8:38 AM, Marc  wrote:
>
>> Arne,
>>Evidently the HTTPClient relies on an SSL Context Service. Try the
>> following configuration in the config.yml file, where you define the
>> context service and reference it from the RPG. Let me know if that works
>> for you!
>>
>>   Additionally, I think you pointed out an inconsistency where we can
>> improve the configuration and documentation. I've created
>> https://issues.apache.org/jira/browse/MINIFICPP-396 and will take care
>> of that ASAP. Thanks
>>   very much for identifying this!
>>
>> Remote Processing Groups:

Re: minifi secure connection

[Arne Oslebo]

Hello Mark,

thanks for the update. I tried running master from github but
unfortunately I now get a segmentation fault:

Thread 1 "minifi" received signal SIGSEGV, Segmentation fault.
0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
(gdb) bt full
#0  0x7777420a in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#1  0x77799681 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#2  0x7777f2f6 in SSL_CTX_use_certificate () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#3  0x7777f6c0 in SSL_CTX_use_certificate_file () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#4  0x55ef91cb in
org::apache::nifi::minifi::controllers::SSLContextService::configure_ssl_context
(this=0x569948b0, ctx=0x56a28430) at
/usr/local/src/nifi-minifi-cpp/extensions/http-curl/../../libminifi/include/controllers/SSLContextService.h:165
    retp = 1
#5  0x55ef9be4 in
org::apache::nifi::minifi::utils::HTTPClient::configure_ssl_context
(curl=0x56a149e0, ctx=0x56a28430, param=0x569948b0) at
/usr/local/src/nifi-minifi-cpp/extensions/http-curl/client/HTTPClient.h:177
    ssl_context_service = 0x569948b0

Any idea what the problem might be?

My full config.yml:

Flow Controller:
  name: MiNiFi Flow
Controller Services:
- name: SSLServiceName
  id: 2438e3c8-015a-1000-79ca-83af40ec1974
  class: SSLContextService
  Properties:
  Client Certificate: /opt/minifi/conf/client.pem
  Private Key: /opt/minifi/conf/client.key
  Passphrase: secret
  CA Certificate: /opt/minifi/conf/nifi-cert.pem
Processors:
- id: cecb1868-9e5a-3e6c--
  name: TailFile
  class: org.apache.nifi.processors.standard.TailFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    File Location: Local
    File to Tail: /tmp/test.log
    Initial Start Position: Beginning of File
    Rolling Filename Pattern:
    tail-base-directory:
    tail-mode: Single file
    tailfile-lookup-frequency: 10 minutes
    tailfile-maximum-age: 24 hours
    tailfile-recursive-lookup: 'false'
Connections:
- id: 76ad4bc4-6557-3e23--
  name: TailFile/success/56ae5abc-0161-1000-aa9e-c340d726e043
  source id: cecb1868-9e5a-3e6c--
  source relationship names:
  - success
  destination id: 56ae5abc-0161-1000-aa9e-c340d726e043
  max work queue size: 1
  max work queue data size: 1 GB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Processing Groups:
- id: 3a25e1a3-c1b2-3e78--
  name: ''
  url: https://w.x.y.z:8443/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  transport protocol: RAW
  proxy host: ''
  proxy port: ''
  proxy user: ''
  proxy password: ''
  local network interface: ''
  Input Ports:
  - id: 56ae5abc-0161-1000-aa9e-c340d726e043
    name: Minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false
    Properties:
  Port: 10443
  SSL Context Service: SSLServiceName
  Host Name: w.x.y.z
  Output Ports: []
Provenance Reporting:


On 09/02/2018 20:18, Marc wrote:
> Arne,
>   I submitted a PR https://github.com/apache/nifi-minifi-cpp/pull/263
> to address these issues. 
>
> On Fri, Feb 9, 2018 at 8:38 AM, Marc  > wrote:
>
> Arne,
>    Evidently the HTTPClient relies on an SSL Context Service. Try
> the following configuration in the config.yml file, where you
> define the context service and reference it from the RPG. Let me
> know if that works for you!
>
>   Additionally, I think you pointed out an inconsistency where we
> can improve the configuration and documentation. I've
> created https://issues.apache.org/jira/browse/MINIFICPP-396
>  and will
> take care of that ASAP. Thanks 
>   very much for identifying this!
>
> Remote Processing Groups:
>     - name: NiFi Flow
>       id: 2438e3c8-015a-1000-79ca-83af40ec1998
>       url: https://127.0.0.1:8383/nifi
>       timeout: 30 secs
>       yield period: 5 sec
>       Input Ports:
>           - id: 2438e3c8-015a-1000-79ca-83af40ec1999
>             name: fromnifi
>             max concurrent tasks: 1
>             Properties:
>                 Port: 10443
>                 SSL Context Service: SSLMe
>                 Host Name: 127.0.0.1
>       Output Ports:
>           - id: ac82e521-015c-1000-2b21-41279516e19a
>             name: tominifi
>             max concurrent tasks: 2
>             Properties:
>                 Port: 10443
>                 SSL Context Service: SSLMe
>                 H