date:20160523

Re: ReplaceText Infinite loop?

2016-05-23 Thread Stéphane Maarek

Opened here: https://issues.apache.org/jira/browse/NIFI-1915

On Tue, May 24, 2016 at 11:11 AM Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Stephane
>
> This definitely needs to be looked at. Would you mind raising JIRA for it?
>
> Thanks
> Oleg
>
> On May 23, 2016, at 20:08, Stéphane Maarek 
> wrote:
>
> An easy way to reproduce is to take the csv to json template here:
>
> https://cwiki.apache.org/confluence/download/attachments/57904847/CsvToJSON.xml?version=1&modificationDate=1442927496000&api=v2
>
>
> In the first ReplaceText, replace a,b,c,d by a$a,b,c,d (it simulates data
> that may or may not contain a $ sign)
> Launch the flow, you'll see the errors / warning / infinite loop
>
>
>
> On Tue, May 24, 2016 at 9:53 AM Stéphane Maarek 
> wrote:
>
> Hi
>>
>> I'm extracting text from the following sentence using a regex:
>>
>> dummy$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
>>
>> The placing the content into a json (using replacetext).
>>
>> I think ReplaceText is in an infinite loop due to the $ sign in my first 
>> field.
>> How should I handle these cases? Isn't NiFi supposed to escape any character 
>> in an attribute?
>>
>> Thanks!
>>
>>
>> 
>>
>> 
>>
>> Stephane
>>
>

Re: Put SQL to connect to Teradata

2016-05-23 Thread Anuj Handa

Hi Jeremy,

I figured out that maven was looking for Jar files and not able to find
them so i had install the JAR files. there were few typos and names
incorrect which were causing problems. I was finally able to create the
Uber JAR but still getting class not found errors in NIFI.

Here's the copy of the JAR file opened in winrar and both JDBC and TDGSS
are in it com/teradata.


[image: Inline image 1]

[image: Inline image 2]
my updated POM File

http://maven.apache.org/POM/4.0.0"; xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance";
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
  4.0.0
  com.anuj
  teradata-uber
  jar
  1.0-SNAPSHOT
  teradata-uber
  http://maven.apache.org




com.anuj.teradata
tdgssconfig
15.00.00.33


com.anuj.teradata
terajdbc4
15.00.00.33



  


org.apache.maven.plugins
maven-shade-plugin
2.4.3


true


*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA






package

shade









Anuj

On Mon, May 23, 2016 at 8:29 AM, Jeremy Dyer  wrote:

> Anuj - Your close. Maven relies on "repositories' to find 3rd party jars
> that are needed as part of an application. Since the Teradata jars do not
> exist in your local repository you will need to install them there. You can
> do that by running these 2 commands and then attempting the "mvn clean
> install package" again. Note you need to place the "path" with the actual
> location of the jar on your local machine.
>
> mvn install:install-file -Dfile= -DgroupId=com.anuj.teradata
> -DartifactId=tdgsconfig -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar
> mvn install:install-file -Dfile= -DgroupId=com.anuj.teradata
> -DartifactId= terajdbc4 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar
>
>
> On Sat, May 21, 2016 at 11:03 PM, Anuj Handa  wrote:
>
>> hi Jeremy,
>>
>> I have no idea on how to use maven but i gave it a try and i am getting
>> the error
>>
>> [ERROR] Failed to execute goal on project teradata-uber: Could not
>> resolve dependencies for project com.anuj:teradata-uber:jar:1.0-SNAPSHOT:
>> The following artifacts could not be resolved:
>> com.anuj.teradata:tdgsconfig:jar
>>
>> i am not sure what location the teradata jar files should be and i tried
>> multiple locations.
>> MVN command
>> c:\apache-maven-3.3.9\teradata-uber> mvn archetype:generate
>> -DgroupId=com.anuj -DartifactId=teradata-uber
>> -DarchetypeArtifactId=maven-archetype-quickstart
>>  -DinteractiveMode=false
>>
>> my pom file
>> 
>> http://maven.apache.org/POM/4.0.0";
>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>> 4.0.0
>>
>> com.anuj
>> teradata-uber
>> 1.0-SNAPSHOT
>>
>> 
>>
>> 
>> com.anuj.teradata
>> tdgsconfig
>> 1.0.0-SNAPSHOT
>> 
>> 
>> com.anuj.teradata
>> terajdbc4
>> 1.0.0-SNAPSHOT
>> 
>>
>> 
>>
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-shade-plugin
>> 2.4.3
>> 
>>
>> true
>> 
>> 
>> *:*
>> 
>> META-INF/*.SF
>> META-INF/*.DSA
>> META-INF/*.RSA
>> 
>> 
>> 
>> 
>> 
>> 
>> package
>> 
>> shade
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> 
>>
>> On Thu, May 19, 2016 at 10:40 PM, Jeremy Dyer  wrote:
>>
>>> Anuj - I would recommend using the Maven shade plugin to create and Uber
>>> jar containing both tdfssconfig.jar and terajdbc4.jar. I used this method a
>>> couple of days ago successfully after I was encountering the same issue you
>>> were having. I wish I could just share the Uber jar with you but due to
>>> Teradata's license model I am unable to do that. Here is the Maven pom.xml
>>> that I used to create the Uber jar however. You will nee

Re: Spark & NiFi question

2016-05-23 Thread Bryan Bende

Conrad,

I think the error message is mis-leading a little bit, it says...

"Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires
Secure Site-to-Site communications, but this instance is not configured for
secure communications"

That statement is saying that your NiFi cluster is configured for secure
site-to-site (which you proved from the debug logs), but that "this
instance" which is actually your Spark streaming job, is not configured for
secure communication.
The reason it thinks your Spark streaming job is not configured for secure
communication is because of the bug I mentioned in the previous email,
where it will never create the SSLContext.

The error message was originally written in the context of two NiFi
instances talking to each other, so it makes more sense in that context.
Perhaps it should be changed to... "this site-to-site client is not
configured for secure communication".

-Bryan


On Mon, May 23, 2016 at 11:04 AM, Conrad Crampton <
conrad.cramp...@secdata.com> wrote:

> Hi,
> I don’t know if I’m hitting some bug here but something doesn’t make sense.
> With ssl debug on I get the following
> NiFi Receiver, READ: TLSv1.2 Application Data, length = 1648
> Padded plaintext after DECRYPTION:  len = 1648
> : 65 A2 B8 34 DF 20 6B 95   56 88 97 16 7A EC 8F E3  e..4. k.V...z...
> 0010: 48 54 54 50 2F 31 2E 31   20 32 30 30 20 4F 4B 0D  HTTP/1.1 200 OK.
> 0020: 0A 44 61 74 65 3A 20 4D   6F 6E 2C 20 32 33 20 4D  .Date: Mon, 23 M
> 0030: 61 79 20 32 30 31 36 20   31 34 3A 34 39 3A 33 39  ay 2016 14:49:39
> 0040: 20 47 4D 54 0D 0A 53 65   72 76 65 72 3A 20 4A 65   GMT..Server: Je
> 0050: 74 74 79 28 39 2E 32 2E   31 31 2E 76 32 30 31 35  tty(9.2.11.v2015
> 0060: 30 35 32 39 29 0D 0A 43   61 63 68 65 2D 43 6F 6E  0529)..Cache-Con
> 0070: 74 72 6F 6C 3A 20 70 72   69 76 61 74 65 2C 20 6E  trol: private, n
> 0080: 6F 2D 63 61 63 68 65 2C   20 6E 6F 2D 73 74 6F 72  o-cache, no-stor
> 0090: 65 2C 20 6E 6F 2D 74 72   61 6E 73 66 6F 72 6D 0D  e, no-transform.
> 00A0: 0A 56 61 72 79 3A 20 41   63 63 65 70 74 2D 45 6E  .Vary: Accept-En
> 00B0: 63 6F 64 69 6E 67 2C 20   55 73 65 72 2D 41 67 65  coding, User-Age
> 00C0: 6E 74 0D 0A 44 61 74 65   3A 20 4D 6F 6E 2C 20 32  nt..Date: Mon, 2
> 00D0: 33 20 4D 61 79 20 32 30   31 36 20 31 34 3A 34 39  3 May 2016 14:49
> 00E0: 3A 33 39 20 47 4D 54 0D   0A 43 6F 6E 74 65 6E 74  :39 GMT..Content
> 00F0: 2D 54 79 70 65 3A 20 61   70 70 6C 69 63 61 74 69  -Type: applicati
> 0100: 6F 6E 2F 6A 73 6F 6E 0D   0A 56 61 72 79 3A 20 41  on/json..Vary: A
> 0110: 63 63 65 70 74 2D 45 6E   63 6F 64 69 6E 67 2C 20  ccept-Encoding,
> 0120: 55 73 65 72 2D 41 67 65   6E 74 0D 0A 43 6F 6E 74  User-Agent..Cont
> 0130: 65 6E 74 2D 4C 65 6E 67   74 68 3A 20 31 32 38 35  ent-Length: 1285
> 0140: 0D 0A 0D 0A 7B 22 72 65   76 69 73 69 6F 6E 22 3A  ."revision":
> 0150: 7B 22 63 6C 69 65 6E 74   49 64 22 3A 22 39 34 38  ."clientId":"948
> 0160: 66 62 34 31 33 2D 65 39   37 64 2D 34 32 37 65 2D  fb413-e97d-427e-
> 0170: 61 34 38 36 2D 31 31 63   39 65 37 31 63 63 62 62  a486-11c9e71ccbb
> 0180: 32 22 7D 2C 22 63 6F 6E   74 72 6F 6C 6C 65 72 22  2".,"controller"
> 0190: 3A 7B 22 69 64 22 3A 22   31 38 63 38 39 64 32 33  :."id":"18c89d23
> 01A0: 2D 61 35 31 65 2D 34 35   35 38 2D 62 30 31 61 2D  -a51e-4558-b01a-
> 01B0: 33 66 36 30 64 66 31 31   63 39 61 64 22 2C 22 6E  3f60df11c9ad","n
> 01C0: 61 6D 65 22 3A 22 4E 69   46 69 20 46 6C 6F 77 22  ame":"NiFi Flow"
> 01D0: 2C 22 63 6F 6D 6D 65 6E   74 73 22 3A 22 22 2C 22  ,"comments":"","
> 01E0: 72 75 6E 6E 69 6E 67 43   6F 75 6E 74 22 3A 31 36  runningCount":16
> 01F0: 34 2C 22 73 74 6F 70 70   65 64 43 6F 75 6E 74 22  4,"stoppedCount"
> 0200: 3A 34 33 2C 22 69 6E 76   61 6C 69 64 43 6F 75 6E  :43,"invalidCoun
> 0210: 74 22 3A 31 2C 22 64 69   73 61 62 6C 65 64 43 6F  t":1,"disabledCo
> 0220: 75 6E 74 22 3A 30 2C 22   69 6E 70 75 74 50 6F 72  unt":0,"inputPor
> 0230: 74 43 6F 75 6E 74 22 3A   37 2C 22 6F 75 74 70 75  tCount":7,"outpu
> 0240: 74 50 6F 72 74 43 6F 75   6E 74 22 3A 31 2C 22 72  tPortCount":1,"r
> 0250: 65 6D 6F 74 65 53 69 74   65 4C 69 73 74 65 6E 69  emoteSiteListeni
> 0260: 6E 67 50 6F 72 74 22 3A   39 38 37 30 2C 22 73 69  ngPort":9870,"si
> 0270: 74 65 54 6F 53 69 74 65   53 65 63 75 72 65 22 3A  teToSiteSecure":
> 0280: 74 72 75 65 2C 22 69 6E   73 74 61 6E 63 65 49 64  true,"instanceId
> 0290: 22 3A 22 30 35 38 30 63   35 31 38 2D 39 62 63 37  ":"0580c518-9bc7
> 02A0: 2D 34 37 38 33 2D 39 32   34 38 2D 35 38 30 61 36  -4783-9248-580a6
> 02B0: 37 34 65 34 33 35 62 22   2C 22 69 6E 70 75 74 50  74e435b","inputP
> 02C0: 6F 72 74 73 22 3A 5B 7B   22 69 64 22 3A 22 33 32  orts":[."id":"32
> 02D0: 37 30 39 33 31 66 2D 64   61 38 35 2D 34 63 34 65  70931f-da85-4c4e
> 02E0: 2D 62 61 65 36 2D 38 63   36 32 37 62 30 39 62 37  -bae6-8c627b09b7
> 02F0: 32 66 22 2C 22 6E 61 6D   65 22 3A 22 48 44 46 53  2f","name":"HDFS
> 0300: 49 6E 63 6F 6D 69 6E 67   22 2C 22 63 6F 6D 6D 65  Incoming","comme
> 0310: 6E 74

Re: Spark & NiFi question

2016-05-23 Thread Conrad Crampton

Hi,
I don’t know if I’m hitting some bug here but something doesn’t make sense.
With ssl debug on I get the following
NiFi Receiver, READ: TLSv1.2 Application Data, length = 1648
Padded plaintext after DECRYPTION:  len = 1648
: 65 A2 B8 34 DF 20 6B 95   56 88 97 16 7A EC 8F E3  e..4. k.V...z...
0010: 48 54 54 50 2F 31 2E 31   20 32 30 30 20 4F 4B 0D  HTTP/1.1 200 OK.
0020: 0A 44 61 74 65 3A 20 4D   6F 6E 2C 20 32 33 20 4D  .Date: Mon, 23 M
0030: 61 79 20 32 30 31 36 20   31 34 3A 34 39 3A 33 39  ay 2016 14:49:39
0040: 20 47 4D 54 0D 0A 53 65   72 76 65 72 3A 20 4A 65   GMT..Server: Je
0050: 74 74 79 28 39 2E 32 2E   31 31 2E 76 32 30 31 35  tty(9.2.11.v2015
0060: 30 35 32 39 29 0D 0A 43   61 63 68 65 2D 43 6F 6E  0529)..Cache-Con
0070: 74 72 6F 6C 3A 20 70 72   69 76 61 74 65 2C 20 6E  trol: private, n
0080: 6F 2D 63 61 63 68 65 2C   20 6E 6F 2D 73 74 6F 72  o-cache, no-stor
0090: 65 2C 20 6E 6F 2D 74 72   61 6E 73 66 6F 72 6D 0D  e, no-transform.
00A0: 0A 56 61 72 79 3A 20 41   63 63 65 70 74 2D 45 6E  .Vary: Accept-En
00B0: 63 6F 64 69 6E 67 2C 20   55 73 65 72 2D 41 67 65  coding, User-Age
00C0: 6E 74 0D 0A 44 61 74 65   3A 20 4D 6F 6E 2C 20 32  nt..Date: Mon, 2
00D0: 33 20 4D 61 79 20 32 30   31 36 20 31 34 3A 34 39  3 May 2016 14:49
00E0: 3A 33 39 20 47 4D 54 0D   0A 43 6F 6E 74 65 6E 74  :39 GMT..Content
00F0: 2D 54 79 70 65 3A 20 61   70 70 6C 69 63 61 74 69  -Type: applicati
0100: 6F 6E 2F 6A 73 6F 6E 0D   0A 56 61 72 79 3A 20 41  on/json..Vary: A
0110: 63 63 65 70 74 2D 45 6E   63 6F 64 69 6E 67 2C 20  ccept-Encoding,
0120: 55 73 65 72 2D 41 67 65   6E 74 0D 0A 43 6F 6E 74  User-Agent..Cont
0130: 65 6E 74 2D 4C 65 6E 67   74 68 3A 20 31 32 38 35  ent-Length: 1285
0140: 0D 0A 0D 0A 7B 22 72 65   76 69 73 69 6F 6E 22 3A  ."revision":
0150: 7B 22 63 6C 69 65 6E 74   49 64 22 3A 22 39 34 38  ."clientId":"948
0160: 66 62 34 31 33 2D 65 39   37 64 2D 34 32 37 65 2D  fb413-e97d-427e-
0170: 61 34 38 36 2D 31 31 63   39 65 37 31 63 63 62 62  a486-11c9e71ccbb
0180: 32 22 7D 2C 22 63 6F 6E   74 72 6F 6C 6C 65 72 22  2".,"controller"
0190: 3A 7B 22 69 64 22 3A 22   31 38 63 38 39 64 32 33  :."id":"18c89d23
01A0: 2D 61 35 31 65 2D 34 35   35 38 2D 62 30 31 61 2D  -a51e-4558-b01a-
01B0: 33 66 36 30 64 66 31 31   63 39 61 64 22 2C 22 6E  3f60df11c9ad","n
01C0: 61 6D 65 22 3A 22 4E 69   46 69 20 46 6C 6F 77 22  ame":"NiFi Flow"
01D0: 2C 22 63 6F 6D 6D 65 6E   74 73 22 3A 22 22 2C 22  ,"comments":"","
01E0: 72 75 6E 6E 69 6E 67 43   6F 75 6E 74 22 3A 31 36  runningCount":16
01F0: 34 2C 22 73 74 6F 70 70   65 64 43 6F 75 6E 74 22  4,"stoppedCount"
0200: 3A 34 33 2C 22 69 6E 76   61 6C 69 64 43 6F 75 6E  :43,"invalidCoun
0210: 74 22 3A 31 2C 22 64 69   73 61 62 6C 65 64 43 6F  t":1,"disabledCo
0220: 75 6E 74 22 3A 30 2C 22   69 6E 70 75 74 50 6F 72  unt":0,"inputPor
0230: 74 43 6F 75 6E 74 22 3A   37 2C 22 6F 75 74 70 75  tCount":7,"outpu
0240: 74 50 6F 72 74 43 6F 75   6E 74 22 3A 31 2C 22 72  tPortCount":1,"r
0250: 65 6D 6F 74 65 53 69 74   65 4C 69 73 74 65 6E 69  emoteSiteListeni
0260: 6E 67 50 6F 72 74 22 3A   39 38 37 30 2C 22 73 69  ngPort":9870,"si
0270: 74 65 54 6F 53 69 74 65   53 65 63 75 72 65 22 3A  teToSiteSecure":
0280: 74 72 75 65 2C 22 69 6E   73 74 61 6E 63 65 49 64  true,"instanceId
0290: 22 3A 22 30 35 38 30 63   35 31 38 2D 39 62 63 37  ":"0580c518-9bc7
02A0: 2D 34 37 38 33 2D 39 32   34 38 2D 35 38 30 61 36  -4783-9248-580a6
02B0: 37 34 65 34 33 35 62 22   2C 22 69 6E 70 75 74 50  74e435b","inputP
02C0: 6F 72 74 73 22 3A 5B 7B   22 69 64 22 3A 22 33 32  orts":[."id":"32
02D0: 37 30 39 33 31 66 2D 64   61 38 35 2D 34 63 34 65  70931f-da85-4c4e
02E0: 2D 62 61 65 36 2D 38 63   36 32 37 62 30 39 62 37  -bae6-8c627b09b7
02F0: 32 66 22 2C 22 6E 61 6D   65 22 3A 22 48 44 46 53  2f","name":"HDFS
0300: 49 6E 63 6F 6D 69 6E 67   22 2C 22 63 6F 6D 6D 65  Incoming","comme
0310: 6E 74 73 22 3A 22 22 2C   22 73 74 61 74 65 22 3A  nts":"","state":
0320: 22 53 54 4F 50 50 45 44   22 7D 2C 7B 22 69 64 22  "STOPPED".,."id"
0330: 3A 22 30 39 33 30 63 62   32 63 2D 37 61 38 33 2D  :"0930cb2c-7a83-
0340: 34 38 36 64 2D 62 62 61   65 2D 38 62 33 30 31 32  486d-bbae-8b3012
0350: 64 36 31 39 66 37 22 2C   22 6E 61 6D 65 22 3A 22  d619f7","name":"
0360: 50 6F 72 74 20 39 30 39   38 20 49 6E 63 6F 6D 69  Port 9098 Incomi
0370: 6E 67 20 53 79 73 6C 6F   67 73 22 2C 22 63 6F 6D  ng Syslogs","com
0380: 6D 65 6E 74 73 22 3A 22   22 2C 22 73 74 61 74 65  ments":"","state
0390: 22 3A 22 52 55 4E 4E 49   4E 47 22 7D 2C 7B 22 69  ":"RUNNING".,."i
03A0: 64 22 3A 22 31 34 62 64   32 66 66 35 2D 38 38 36  d":"14bd2ff5-886
03B0: 61 2D 34 61 32 39 2D 62   39 39 61 2D 38 64 34 34  a-4a29-b99a-8d44
03C0: 65 66 37 38 66 30 31 30   22 2C 22 6E 61 6D 65 22  ef78f010","name"
03D0: 3A 22 48 44 46 53 57 65   62 73 65 6E 73 65 53 65  :"HDFSWebsenseSe
03E0: 63 75 72 69 74 79 22 2C   22 63 6F 6D 6D 65 6E 74  curity","comment
03F0: 73 22 3A 22 22 2C 22 73   74 61 74 65 22 3A 22 53  s":"","state":"S
0400: 54 4F 50 50 45 44 22 7D   2C 7B

Re: Spark & NiFi question

2016-05-23 Thread Bryan Bende

Conrad,

Unfortunately I think this is a result of the issue you discovered with the
SSLContext not getting created from the properties on the
SiteToSiteClientBuilder...

Whats happening is the spark side is hitting this:

if (siteToSiteSecure) {
if (sslContext == null) {
throw new IOException("Unable to communicate with " + hostname +
":" + port
+ " because it requires Secure Site-to-Site communications,
but this instance is not configured for secure communications");
}

And siteToSiteSecure is true, but the sslContext is null so it can never
get past this point. I submitted a pull request on Friday that should
address the issue [1].

Once we get this merged in you could possibly build the source to get the
fixed SiteToSiteClient code, otherwise you could wait for the 0.7.0 release
to happen.

-Bryan

[1] https://github.com/apache/nifi/pull/457

On Mon, May 23, 2016 at 5:39 AM, Conrad Crampton <
conrad.cramp...@secdata.com> wrote:

> Hi,
> An update to this but still not working
> I have now set keystore and truststore as system properties, and included
> these as part of the SiteToSiteClientConfig building. I have used a cert
> that I have for one of the servers in my cluster as I know they can
> communicate over ssl with NCM as my 6 node cluster works over ssl and has
> remote ports working (as I read from syslog on a primary server then
> distribute to other via remote ports as suggested somewhere else) .
> When I try now to connect to output port via Spark, I get a
> "EndpointConnectionPool[Cluster URL=
> https://yarn-cm1.mis-cds.local:9090/nifi/] Unable to refresh Remote
> Group's peers due to java.io.IOException: Unable to communicate with
> yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site
> communications, but this instance is not configured for secure
> communications"
> Exception even though I know Secure Site-to-Site communication is working
> (9870 being the port set up for remote s2s comms in nifi.properties), so I
> am now really confused!!
>
> Does the port that I wish to read from need to be set up with remote
> process group (conceptually I’m struggling with how to do this for an
> output port), or is it is sufficient to be ‘just an output port’?
>
> I have this working when connecting to an unsecured (http) instance of
> NiFi running on my laptop with Spark and a standard output port. Does it
> make a difference that my production cluster is a cluster and therefore
> needs setting up differently?
>
> So many questions but I’m stuck now so any suggestions welcome.
> Thanks
> Conrad
>
> From: Conrad Crampton 
> Reply-To: "users@nifi.apache.org" 
> Date: Friday, 20 May 2016 at 09:16
> To: "users@nifi.apache.org" 
> Subject: SPOOFED: Re: Spark & NiFi question
>
> Thanks for the pointers Bryan, however wrt your first suggestion. I tried
> without setting SSL properties on System properties and get an unable to
> find ssl path error – this gets resolved by doing as I have done (but of
> course this may be a red herring). I initially tried setting on site
> builder but got the same error as below – it appears to make no difference
> as to what is logged in the nifi-users.log if I include SSL props on site
> builder or not, I get the same error viz:
>
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> ( L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759]
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
> ( L=Maidstone, ST=Kent, C=GB>) GET
> https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
> 2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to
> verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data
> Europe Ltd, L=Maidstone, ST=Kent, C=GB
>
> I am using self signed certs if that makes a difference (but these work
> fine on across the cluster). I am not seeing my spark user appear in the
> list of users to grant access.
>
> I have turned on debug for ssl to see if that is throwing up anything but
> nothing appears obvious – here is the snipet that I would expect errors to
> be shown from that log.
>
> ... no IV derived for this protocol
> %% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
> NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
> NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
> *** Finished
> verify_data:  { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data:  { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
> ***
> NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
> NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240
>
> I don’t really know enough about certificates and how c

Re: Scheduling approach

2016-05-23 Thread Ravisankar Mani

 Hi Aldrin Piri,

Thank you for your response.

yes you are correct . i just mentioned the sample Cron scheduling (i.e for
every  sec). we may also have possibilities to schedule some processors on
every Saturday 12:00 Pm. on that case we have to go for Cron type
scheduling right ?

As you said that Cron scheduling will not start process at 0 sec. thats
fine for me.

Is there is any other way to force start the process irrespective of
scheduling in NIfI using REST API ?

Regards

Ravi

On Fri, May 20, 2016 at 7:52 AM, Aldrin Piri  wrote:

> Hi Ravi,
>
> If my understanding is correct, you would like the processor to run
> immediately and then every 5 minutes after?  If so, this could be
> accomplished with the Timer Driven scheduling strategy and specifying a Run
> Schedule of 5 minutes.
>
> Cron will get you a specific time enabling you to either run every 5th
> minute (*/5 * * ...) or an explicit listing of minutes that might start at
> a non-zero number of the hour such as (2,7,12,17,22... * * ...).
>
> If your needs are a bit different and not met by these approaches, some
> additional details will help us track down the right path.
>
>
>
> On Fri, May 20, 2016 at 7:09 AM, Ravisankar Mani 
> wrote:
>
>> Hi All,
>>
>>
>> I have used the *CRON driven* type scheduling in processor in order to
>> repeat the process for every 5 mins. But after starting the processor i
>> need to wait for 5 mins to get the result form the processor. is it
>> possible to run the process while starting Processor  ?.  after that
>> initial run process will schedule to every 5 mins. is it possible ?
>>
>>
>>
>> Regards,
>>
>> Ravisankar
>>
>
>

Re: Use Case...Please help

2016-05-23 Thread Conrad Crampton

Hi,
A bit hard to tell with the information you have given, but it appears to me 
that you are trying to fetch a file from the file system and transfer to HDFS 
but you are referring to a file that isn’t actually available on the machine 
NiFi is running on. The ‘Input Directory Location’ attribute (according to the 
docs) states this is used to preserve state (I assume of the files being read 
in the directory) either locally or across the cluster – it isn’t there to 
indicate that the file is remote (to the Nifi server).
If the assumptions I am making are true, then you can’t use ListFile processor 
to get a file from a server that isn’t the local nit one.
You will need to get the files in a different way (FTP etc.
Conrad

From: "Tripathi, Shiv Deepak" 
mailto:shiv.deepak.tripa...@philips.com>>
Reply-To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Monday, 23 May 2016 at 11:28
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: RE: Use Case...Please help

Can anybody help me please. I am new and got stucked.


Thanks,
Deepak
From: Tripathi, Shiv Deepak [mailto:shiv.deepak.tripa...@philips.com]
Sent: Sunday, May 22, 2016 11:10 PM
To: users@nifi.apache.org
Subject: RE: Use Case...Please help

Hi Mark,

In order to implement apache nifi.

I downloaded hortonworks sandbox and installed apache nifi on that. Its working 
fine in below scenario.

Scenario 1: My input directory is in local file system on HDP(screenshot name 
“listfilelocaldir”) and output is on HDFS file system.

For all processor in dataflow please see screenshot – “HDP sandbox local to 
HDFS”

Scenario 2: could you please tell me which processor and in what order I need 
to use if I want to send file from 
\\btc7n001\Ongoing-MR\MRI\Deepak  
(password enable network drive mapped to my machine) to HDP cluster created in 
VMplayer.

Its not recognizing the input directory at all. Please see the screenshot 
name-“Usecaseinputdir.jpeg”

Please help me.

Thanks,
Deepak


From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Monday, May 16, 2016 6:19 PM
To: users@nifi.apache.org
Subject: Re: Use Case...Please help

Deepak,

Yes, you should be able to do so.

Thanks
-Mark

On May 16, 2016, at 8:44 AM, Tripathi, Shiv Deepak 
mailto:shiv.deepak.tripa...@philips.com>> 
wrote:

Thanks a lot Mark.

Looking forward to try it out.

If I understood correctly than I can drop log copying script and staging 
machine and can directly pull the logs from repository.

Please confirm.

Thanks,
Deepak

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Monday, May 16, 2016 5:06 PM
To: users@nifi.apache.org
Subject: Re: Use Case...Please help

Deepak,

Thanks for providing such a detailed description of your use case. I think NiFi 
would be an excellent
tool to help you out here!

As I mentioned before, you would typically use ListFile -> FetchFile to pull 
the data in. Clearly, here,
though, you want to be more selective about what you pull in. You can 
accomplish this by using a
RouteOnAttribute processor. So you'd have something like: ListFile -> 
RouteOnAttribute -> FetchFile.
The RouteOnAttribute processor is very powerful and allows you to configure how 
to route each piece
of data based on whatever attributes are available. The ListFile Processor adds 
the following attributes
to each piece of data that it pulls in:

filename (name of file)
path (relative path of file)
absolute.path (absolute directory of file)
fs.owner (owner of the file)
fs.group (group that the file belongs to)
fs.lastModified (last modified date)
fs.length (file length)
fs.permissions (file permissions, such as rw-rw-r--)

From these, you can make all sorts of routing decisions, based on name, 
timestamp, etc. You can choose
to terminate data that does not meet your criteria.

When you use FetchFile, you have the option of deleting the source file, moving 
it elsewhere, or leaving
it as-is. So you wouldn't need to delete it if you don't want to. This is 
possible because ListFile keeps track
of what has been 'listed'. So it won't ingest duplicate data, but it will pick 
up new files (if any existing
file is modified, it will pick up the new version of the file.)

You can then use UnpackContent if you want to unzip the data, or you can leave 
it zipped. After the FetchFile,
you can also use a RouteOnAttribute processor to separate out the XML from the 
log files and put those to
different directories in HDFS.

Does this sound like it will provide you all that you need?

Thanks
-Mark



On May 16, 2016, at 3:06 AM, Tripathi, Shiv Deepak 
mailto:shiv.deepak.tripa...@philips.com>> 
wrote:

Hi Mark,

I am very happy to see the detailed reply. I am very thankful to you. So 
explaining more about my use case below.

1-  Screenshot Name--> “Stagingdirectory_copiedfiles”

Log copying script will copy the log in the

Re: Use Case...Please help

2016-05-23 Thread Simon Elliston Ball

Hi Deepak,

It looks like your flow is following the right kind of direction, so I suspect 
there’s something about the path that isn’t working out. One solution would be 
to use a mapped drive on your machine, which makes it a little simpler, 
however, it would be nice if we could get it working with the unc path as well. 
Are you getting any validation messages on the ListFile processor, either on 
the bulletin board in Nifi () or in the nifi-app.log file? 

Note that you will have to be connected to the drive to ensure you have 
credentials, or have you nifi user be able to connect to that drive with its 
windows credentials. There isn’t currently a means to provide authentication 
per share in the processor, but nifi should inherit the credential context of 
whichever user is running the nifi process. 

Hope that helps, 
Simon


> On 22 May 2016, at 18:40, Tripathi, Shiv Deepak 
>  wrote:
> 
> Hi Mark,
>  
> In order to implement apache nifi.
>  
> I downloaded hortonworks sandbox and installed apache nifi on that. Its 
> working fine in below scenario.
>  
> Scenario 1: My input directory is in local file system on HDP(screenshot name 
> “listfilelocaldir”) and output is on HDFS file system.
>  
> For all processor in dataflow please see screenshot – “HDP sandbox local to 
> HDFS”
>  
> Scenario 2: could you please tell me which processor and in what order I need 
> to use if I want to send file from\\btc7n001\Ongoing-MR\MRI\Deepak 
>   (password enable network drive 
> mapped to my machine) to HDP cluster created in VMplayer.
>  
> Its not recognizing the input directory at all. Please see the screenshot 
> name-“Usecaseinputdir.jpeg”
>  
> Please help me.
>  
> Thanks,
> Deepak
>  
>  
> From: Mark Payne [mailto:marka...@hotmail.com ] 
> Sent: Monday, May 16, 2016 6:19 PM
> To: users@nifi.apache.org 
> Subject: Re: Use Case...Please help
>  
> Deepak,
>  
> Yes, you should be able to do so.
>  
> Thanks
> -Mark
>  
> On May 16, 2016, at 8:44 AM, Tripathi, Shiv Deepak 
> mailto:shiv.deepak.tripa...@philips.com>> 
> wrote:
>  
> Thanks a lot Mark.
>  
> Looking forward to try it out.
>  
> If I understood correctly than I can drop log copying script and staging 
> machine and can directly pull the logs from repository.
>  
> Please confirm.
>  
> Thanks,
> Deepak
>  
> From: Mark Payne [mailto:marka...@hotmail.com ] 
> Sent: Monday, May 16, 2016 5:06 PM
> To: users@nifi.apache.org 
> Subject: Re: Use Case...Please help
>  
> Deepak,
>  
> Thanks for providing such a detailed description of your use case. I think 
> NiFi would be an excellent
> tool to help you out here!
>  
> As I mentioned before, you would typically use ListFile -> FetchFile to pull 
> the data in. Clearly, here,
> though, you want to be more selective about what you pull in. You can 
> accomplish this by using a
> RouteOnAttribute processor. So you'd have something like: ListFile -> 
> RouteOnAttribute -> FetchFile.
> The RouteOnAttribute processor is very powerful and allows you to configure 
> how to route each piece
> of data based on whatever attributes are available. The ListFile Processor 
> adds the following attributes
> to each piece of data that it pulls in:
>  
> filename (name of file)
> path (relative path of file)
> absolute.path (absolute directory of file)
> fs.owner (owner of the file)
> fs.group (group that the file belongs to)
> fs.lastModified (last modified date)
> fs.length (file length)
> fs.permissions (file permissions, such as rw-rw-r--)
>  
> From these, you can make all sorts of routing decisions, based on name, 
> timestamp, etc. You can choose
> to terminate data that does not meet your criteria.
>  
> When you use FetchFile, you have the option of deleting the source file, 
> moving it elsewhere, or leaving
> it as-is. So you wouldn't need to delete it if you don't want to. This is 
> possible because ListFile keeps track
> of what has been 'listed'. So it won't ingest duplicate data, but it will 
> pick up new files (if any existing
> file is modified, it will pick up the new version of the file.)
>  
> You can then use UnpackContent if you want to unzip the data, or you can 
> leave it zipped. After the FetchFile,
> you can also use a RouteOnAttribute processor to separate out the XML from 
> the log files and put those to
> different directories in HDFS.
>  
> Does this sound like it will provide you all that you need?
>  
> Thanks
> -Mark
>  
>  
>  
> On May 16, 2016, at 3:06 AM, Tripathi, Shiv Deepak 
> mailto:shiv.deepak.tripa...@philips.com>> 
> wrote:
>  
> Hi Mark,
>  
> I am very happy to see the detailed reply. I am very thankful to you. So 
> explaining more about my use case below.
>  
> 1-  Screenshot Nameà “Stagingdirectory_copiedfiles”
>  
> Log copying script will copy the log in the  staging directory which is in my 
> case “D:\PaiValidation” and will maintain mu

Re: Put SQL to connect to Teradata

2016-05-23 Thread Jeremy Dyer

Anuj - Your close. Maven relies on "repositories' to find 3rd party jars
that are needed as part of an application. Since the Teradata jars do not
exist in your local repository you will need to install them there. You can
do that by running these 2 commands and then attempting the "mvn clean
install package" again. Note you need to place the "path" with the actual
location of the jar on your local machine.

mvn install:install-file -Dfile= -DgroupId=com.anuj.teradata
-DartifactId=tdgsconfig -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar
mvn install:install-file -Dfile= -DgroupId=com.anuj.teradata
-DartifactId= terajdbc4 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar


On Sat, May 21, 2016 at 11:03 PM, Anuj Handa  wrote:

> hi Jeremy,
>
> I have no idea on how to use maven but i gave it a try and i am getting
> the error
>
> [ERROR] Failed to execute goal on project teradata-uber: Could not resolve
> dependencies for project com.anuj:teradata-uber:jar:1.0-SNAPSHOT: The
> following artifacts could not be resolved: com.anuj.teradata:tdgsconfig:jar
>
> i am not sure what location the teradata jar files should be and i tried
> multiple locations.
> MVN command
> c:\apache-maven-3.3.9\teradata-uber> mvn archetype:generate
> -DgroupId=com.anuj -DartifactId=teradata-uber
> -DarchetypeArtifactId=maven-archetype-quickstart
>  -DinteractiveMode=false
>
> my pom file
> 
> http://maven.apache.org/POM/4.0.0";
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> 4.0.0
>
> com.anuj
> teradata-uber
> 1.0-SNAPSHOT
>
> 
>
> 
> com.anuj.teradata
> tdgsconfig
> 1.0.0-SNAPSHOT
> 
> 
> com.anuj.teradata
> terajdbc4
> 1.0.0-SNAPSHOT
> 
>
> 
>
> 
> 
> 
> org.apache.maven.plugins
> maven-shade-plugin
> 2.4.3
> 
>
> true
> 
> 
> *:*
> 
> META-INF/*.SF
> META-INF/*.DSA
> META-INF/*.RSA
> 
> 
> 
> 
> 
> 
> package
> 
> shade
> 
> 
> 
> 
> 
> 
>
> 
>
> On Thu, May 19, 2016 at 10:40 PM, Jeremy Dyer  wrote:
>
>> Anuj - I would recommend using the Maven shade plugin to create and Uber
>> jar containing both tdfssconfig.jar and terajdbc4.jar. I used this method a
>> couple of days ago successfully after I was encountering the same issue you
>> were having. I wish I could just share the Uber jar with you but due to
>> Teradata's license model I am unable to do that. Here is the Maven pom.xml
>> that I used to create the Uber jar however. You will need to change the
>> groupId, artifactId, and version to match what you actually have installed
>> in your local Maven repository.
>>
>> 
>> http://maven.apache.org/POM/4.0.0";
>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
>> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
>> 4.0.0
>>
>> com.jeremydyer
>> teradata-uber
>> 1.0-SNAPSHOT
>>
>> 
>>
>> 
>> com.jeremydyer.teradata
>> tdgsconfig
>> 1.0.0-SNAPSHOT
>> 
>> 
>> com.jeremydyer.teradata
>> terajdbc4
>> 1.0.0-SNAPSHOT
>> 
>>
>> 
>>
>> 
>> 
>> 
>> org.apache.maven.plugins
>> maven-shade-plugin
>> 2.4.3
>> 
>> 
>> true
>> 
>> 
>> *:*
>> 
>> META-INF/*.SF
>> META-INF/*.DSA
>> META-INF/*.RSA
>> 
>> 
>> 
>> 
>> 
>> 
>> package
>> 
>> shade
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> 
>>
>>
>> On Thu, May 19, 2016 at 9:05 PM, Anuj Handa  wrote:
>>
>>> Hi  folks,
>>>
>>> I am using putsql on nifi-0.6.1 to connect to teradata and getting the
>>> below error
>>>
>>> 2016-05-19 20:06:33,230 ERROR [Timer-Driven Process Thread-4]
>>> o.apache.nifi.processors.standard.PutSQL
>>> java.l

RE: Use Case...Please help

2016-05-23 Thread Tripathi, Shiv Deepak

Can anybody help me please. I am new and got stucked.

Thanks,
Deepak
From: Tripathi, Shiv Deepak [mailto:shiv.deepak.tripa...@philips.com]
Sent: Sunday, May 22, 2016 11:10 PM
To: users@nifi.apache.org
Subject: RE: Use Case...Please help

Hi Mark,

In order to implement apache nifi.

I downloaded hortonworks sandbox and installed apache nifi on that. Its working 
fine in below scenario.

Scenario 1: My input directory is in local file system on HDP(screenshot name 
“listfilelocaldir”) and output is on HDFS file system.

For all processor in dataflow please see screenshot – “HDP sandbox local to 
HDFS”

Scenario 2: could you please tell me which processor and in what order I need 
to use if I want to send file from 
\\btc7n001\Ongoing-MR\MRI\Deepak  
(password enable network drive mapped to my machine) to HDP cluster created in 
VMplayer.

Its not recognizing the input directory at all. Please see the screenshot 
name-“Usecaseinputdir.jpeg”

Please help me.

Thanks,
Deepak

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Monday, May 16, 2016 6:19 PM
To: users@nifi.apache.org
Subject: Re: Use Case...Please help

Deepak,

Yes, you should be able to do so.

Thanks
-Mark

On May 16, 2016, at 8:44 AM, Tripathi, Shiv Deepak 
mailto:shiv.deepak.tripa...@philips.com>> 
wrote:

Thanks a lot Mark.

Looking forward to try it out.

If I understood correctly than I can drop log copying script and staging 
machine and can directly pull the logs from repository.

Please confirm.

Thanks,
Deepak

From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Monday, May 16, 2016 5:06 PM
To: users@nifi.apache.org
Subject: Re: Use Case...Please help

Deepak,

Thanks for providing such a detailed description of your use case. I think NiFi 
would be an excellent
tool to help you out here!

As I mentioned before, you would typically use ListFile -> FetchFile to pull 
the data in. Clearly, here,
though, you want to be more selective about what you pull in. You can 
accomplish this by using a
RouteOnAttribute processor. So you'd have something like: ListFile -> 
RouteOnAttribute -> FetchFile.
The RouteOnAttribute processor is very powerful and allows you to configure how 
to route each piece
of data based on whatever attributes are available. The ListFile Processor adds 
the following attributes
to each piece of data that it pulls in:

filename (name of file)
path (relative path of file)
absolute.path (absolute directory of file)
fs.owner (owner of the file)
fs.group (group that the file belongs to)
fs.lastModified (last modified date)
fs.length (file length)
fs.permissions (file permissions, such as rw-rw-r--)

From these, you can make all sorts of routing decisions, based on name, 
timestamp, etc. You can choose
to terminate data that does not meet your criteria.

When you use FetchFile, you have the option of deleting the source file, moving 
it elsewhere, or leaving
it as-is. So you wouldn't need to delete it if you don't want to. This is 
possible because ListFile keeps track
of what has been 'listed'. So it won't ingest duplicate data, but it will pick 
up new files (if any existing
file is modified, it will pick up the new version of the file.)

You can then use UnpackContent if you want to unzip the data, or you can leave 
it zipped. After the FetchFile,
you can also use a RouteOnAttribute processor to separate out the XML from the 
log files and put those to
different directories in HDFS.

Does this sound like it will provide you all that you need?

Thanks
-Mark

On May 16, 2016, at 3:06 AM, Tripathi, Shiv Deepak 
mailto:shiv.deepak.tripa...@philips.com>> 
wrote:

Hi Mark,

I am very happy to see the detailed reply. I am very thankful to you. So 
explaining more about my use case below.

1-  Screenshot Name--> “Stagingdirectory_copiedfiles”

Log copying script will copy the log in the  staging directory which is in my 
case “D:\PaiValidation” and will maintain multiple folders. These folders are 
nothing but device serial no. Every serial no will have multiple log files and 
xml files as on  Each day one new log files used to come in this directory, as 
you can see.

In log copy script we defined that how many days logs we want. So lets say we 
passed 360 , so it will copy logs from last 360 days and as it is continuously 
running so after 10 days of when you passed configuration very first time it 
will have logs 360(last 360 days from the time when you passed this parameter 
to script) +10 days+ growing day by day= 370++

And after pushing the files to cluster we are renaming or creating dummy files 
with 0 byte as you can see in screenshot

Also we are passing one more parameter which specifies the device serial no of 
which we want logs not from all devices.

2-  The source repository
Screenshot Name --> “Repository files”

This is actual repository from where we are taking the logs and copying it to 
staging director

Re: Spark & NiFi question

2016-05-23 Thread Conrad Crampton

Hi,
An update to this but still not working
I have now set keystore and truststore as system properties, and included these 
as part of the SiteToSiteClientConfig building. I have used a cert that I have 
for one of the servers in my cluster as I know they can communicate over ssl 
with NCM as my 6 node cluster works over ssl and has remote ports working (as I 
read from syslog on a primary server then distribute to other via remote ports 
as suggested somewhere else) .
When I try now to connect to output port via Spark, I get a
"EndpointConnectionPool[Cluster URL=https://yarn-cm1.mis-cds.local:9090/nifi/] 
Unable to refresh Remote Group's peers due to java.io.IOException: Unable to 
communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure 
Site-to-Site communications, but this instance is not configured for secure 
communications"
Exception even though I know Secure Site-to-Site communication is working (9870 
being the port set up for remote s2s comms in nifi.properties), so I am now 
really confused!!

Does the port that I wish to read from need to be set up with remote process 
group (conceptually I’m struggling with how to do this for an output port), or 
is it is sufficient to be ‘just an output port’?

I have this working when connecting to an unsecured (http) instance of NiFi 
running on my laptop with Spark and a standard output port. Does it make a 
difference that my production cluster is a cluster and therefore needs setting 
up differently?

So many questions but I’m stuck now so any suggestions welcome.
Thanks
Conrad

From: Conrad Crampton 
mailto:conrad.cramp...@secdata.com>>
Reply-To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Friday, 20 May 2016 at 09:16
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: SPOOFED: Re: Spark & NiFi question

Thanks for the pointers Bryan, however wrt your first suggestion. I tried 
without setting SSL properties on System properties and get an unable to find 
ssl path error – this gets resolved by doing as I have done (but of course this 
may be a red herring). I initially tried setting on site builder but got the 
same error as below – it appears to make no difference as to what is logged in 
the nifi-users.log if I include SSL props on site builder or not, I get the 
same error viz:

2016-05-20 08:59:47,082 INFO [NiFi Web Server-29590180] 
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for 
() GET 
https://yarn-cm1.m.:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,082 INFO [NiFi Web Server-29494759] 
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for 
() GET 
https://yarn-cm1.m.xxx:9090/nifi-api/controller (source ip: xx.xx.xx.1)
2016-05-20 08:59:47,083 INFO [NiFi Web Server-29590180] 
o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Unable to 
verify access for CN=spark-processor.m.xxx, OU=Development, O=Secure Data 
Europe Ltd, L=Maidstone, ST=Kent, C=GB

I am using self signed certs if that makes a difference (but these work fine on 
across the cluster). I am not seeing my spark user appear in the list of users 
to grant access.

I have turned on debug for ssl to see if that is throwing up anything but 
nothing appears obvious – here is the snipet that I would expect errors to be 
shown from that log.

... no IV derived for this protocol
%% Server resumed [Session-4, TLS_RSA_WITH_AES_128_CBC_SHA256]
NiFi Receiver, READ: TLSv1.2 Change Cipher Spec, length = 1
NiFi Receiver, READ: TLSv1.2 Handshake, length = 80
*** Finished
verify_data:  { 109, 126, 134, 14, 33, 110, 224, 83, 198, 116, 54, 228 }
***
NiFi Receiver, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data:  { 83, 120, 49, 158, 181, 136, 127, 219, 30, 194, 58, 167 }
***
NiFi Receiver, WRITE: TLSv1.2 Handshake, length = 80
NiFi Receiver, WRITE: TLSv1.2 Application Data, length = 240

I don’t really know enough about certificates and how client java apps would 
use them wrt to the host name/ ip address etc. of details is included in them. 
The nifi-user.log is showing access from a specific IP address which clearly 
doesn’t match the CN details in the cert. Just clutching at straws here!

Any other suggestions?

Thanks
Conrad

From: Bryan Bende mailto:bbe...@gmail.com>>
Reply-To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Thursday, 19 May 2016 at 17:08
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: Re: Spark & NiFi question

Hi Conrad,

I think there are a couple of things at play here...

One is that the SSL properties need to be set on the SiteToSiteClientBuilder, 
rather than through system properties. There should be methods to set the 
keystore and other values.

In a secured NiFi instance, the certificate you are authenticating with (the 
keystore used by the s2s client) would need to have an account in NiFi, a

Re: ReplaceText Infinite loop?

Re: Put SQL to connect to Teradata

Re: Spark & NiFi question

Re: Spark & NiFi question

Re: Spark & NiFi question

Re: Scheduling approach

Re: Use Case...Please help

Re: Use Case...Please help

Re: Put SQL to connect to Teradata

RE: Use Case...Please help

Re: Spark & NiFi question

11 matches

Site Navigation

Mail list logo

Footer information