published by PublishKafkaRecord_0_10 doesn't embed schema.

2018-03-25 Thread Milan Das
uot;,"type":"string","doc":"ISO 8601 timestamp of the eventl <TRUNCATED…..  >,{"name":"privileges","type":["null",{"type":"array","items":"string"}],"doc":"The list of privileges associated with a Privilege Escalation event","default":null},{"name":"subcode","type":["null","string"],"doc":"The error subcode for auth;��d��g%�z�SUCCESS6controller1.ad.interset.com�H22018-03-15T09:07:04-04:00usedid1Pz�� Regards, Milan Das

Re: published by PublishKafkaRecord_0_10 doesn't embed schema.

2018-03-26 Thread Milan Das
Hi Bryan, We are using NIFI 1.4.0. Can we backport this fix to NIFI 1.4? Thanks, Milan Das On 3/26/18, 11:26 AM, "Bryan Bende" wrote: Hello, What version of NiFi are you using? This should be fixed in 1.5.0: https://issues.apache.org/jira/browse

Decommissioning nifi node from cluster

2018-05-15 Thread Milan Das
to make sure all  in flight transactions are processed or not lost (Should be processed by other cluster member). Is the correct process to use “remove” using  node manager?  Does the process going to maintain all in-flight transactions ? Thanks, Milan Das Sr. System Architect email: m

Re: How to run NiFi on HTTPS

2018-05-23 Thread Milan Das
Hi Bajendra, TLS vs SSL is all depends on how you generate the SSL certificates. If the SSL certificate is TLS 1.2 then JVM SSL protocol will use TLS. Regards, Milan Sr. Architect-- Interset Inc On 5/23/18, 7:40 AM, "Brajendra Mishra" wrote: Hi Andy, Thanks a lot for you valuable

Re: How to run NiFi on HTTPS

2018-05-23 Thread Milan Das
eed) https://github.com/dmilan77/scripts/blob/master/ssl/createSSLCert.sh You can verify using nmap or oepnssl openssl s_client -connect hostname:443 -tls1_2 nmap --script ssl-enum-ciphers -p 443 hostname Regards, Milan Das On 5/23/18, 8:03 AM, "Brajendra Mishra" wrote: I

NIFI regex expression on RouteOnAttribute

2018-06-13 Thread Milan Das
Hello All, I am trying to use “RouteOnAttribute” on the following text attribute “/tmp/subdir/9/” with the expression   ${setabsolutepath:matches('\\/9|12\\/')} I also tried without escape sequence ${setabsolutepath:matches('\/9|12\/')} Everytime it is getting routed to unmatched. Here is

ListFile hung when it scan NFS mount or CFS mount

2018-06-14 Thread Milan Das
Hi All, Listfile processor in NIFI 1.4 environment is getting hung  when scanning directories in NFS mount or CFS mount. Wondering if it is a known issue and if there is a fix. Regards, Milan Das

NIFI Multiple Kerberos configuration

2018-06-21 Thread Milan Das
, Milan Das Milan Das Sr. System Architect email: m...@interset.com mobile: +1 678 216 5660 www.interset.com

Re: NIFI Multiple Kerberos configuration

2018-06-22 Thread Milan Das
The problem is krb5.conf. There are two different krb5.conf with two different kdc server. Regards, Milan Das On 6/22/18, 2:04 AM, "Koji Kawamura" wrote: Hi Milan, I haven't tried myself, but since NiFi has Kerberos configuration per Processor instance, e

How to Add DFM user /group ?

2018-08-19 Thread Milan Das
” user by default can not add any NIFI component because it have only Read access to “Global policy to view the user interface” is read. Basically I want add a DFM user or provide DFM to “Initial Admin Identity”. I am using NIFI version 1.5 Appreciate any help. Thanks, Milan Das

ListFile is slow when it scan NFS mount

2018-09-20 Thread Milan Das
conjured on ListS3 ? Thanks, Milan Das

NIFI single node in cluster mode

2018-10-12 Thread Milan Das
. Thanks, Milan Das

Re: NIFI single node in cluster mode

2018-10-13 Thread Milan Das
CuratorFrameworkImpl backgroundOperationsLoop exiting 2018-10-13 13:33:53,726 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. Thanks, Milan Das On 10/12/18, 6:26 PM, "Bryan Bende&qu

Re: NIFI single node in cluster mode

2018-10-14 Thread Milan Das
726/listing-requests (source ip: 172.30.1.235) 2018-10-14 15:03:21,633 INFO [NiFi Web Server-138] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for nifiadmin1@ Regards, Milan Das Milan Das Sr. System Architect email: m...@interset.com <https://www.linkedin.com/in/mi

Unable to List Queue

2018-10-15 Thread Milan Das
for nifiadmin1@ Thanks, Milan Das

Re: Unable to List Queue

2018-10-15 Thread Milan Das
Hi Bryan Thanks for your response. The user have all access including view the data at root processor level. It works when is.cluster is false. It doesn’t work when is.cluster is true. Thanks, Milan Das On 10/15/18, 2:56 PM, "Bryan Bende" wrote: The error message is saying

Re: Unable to List Queue

2018-10-15 Thread Milan Das
Hi Brian, Yes that was the problem. I didn’t know that cluster node identity also need to be added. After adding it worked. Thanks a lot. Thanks, Milan Das On 10/15/18, 5:44 PM, "Bryan Bende" wrote: Just to confirm, the cluster nodes are also granted access to "

Re: Load issues

2018-10-23 Thread Milan Das
Did you increase the no. of Concurrent Tasks and Run Duration on slow processors ? Thanks, Milan Das On 10/23/18, 8:26 PM, "Phil H" wrote: Hi team, My NiFi is struggling to process data (lots of back pressure in various queues) but it is only using a very small a

Controller Service not loading ERROR: The service APIs should not be bundled with the implementations.

2018-10-29 Thread Milan Das
] o.a.n.d.html.HtmlDocumentationWriter Could not link to com.interset.nifi.hbase.CDHHBase_ClientService because no bundles were found Thanks, Milan Das

Re: Hi

2018-10-30 Thread Milan Das
Did you try WinEventLogProcessor ? Thanks, Milan Das On 10/29/18, 11:17 PM, "find" <907611...@qq.com> wrote: Hi, Sorry to disturb you.I want to parse a .evtx(a window event log file) file using nifi(https://github.com/apache/nifi),but I just don't know how to

Re: Hi

2018-10-30 Thread Milan Das
I mean ParseEvtx processor. On 10/30/18, 7:37 AM, "Milan Das" wrote: Did you try WinEventLogProcessor ? Thanks, Milan Das On 10/29/18, 11:17 PM, "find" <907611...@qq.com> wrote: Hi, Sorry to disturb you.I want to parse

Secured NIFI (clustered) error on ListFile viewstate

2018-11-02 Thread Milan Das
I have a (Kerberos) secured NIFI running and it is connecting to secured SASL secured zookeeper.   Zookeeper-jaas is configured in bootstrap. Cluster starts clean Not sure, if there is something else need to be done. Configuratons are as below: conf/state-management.xml         z

Contributing to NIFI (Hazelcast Custom processor)

2018-11-04 Thread Milan Das
for added features ? Thanks, Milan Das

Re: Contributing to NIFI (Hazelcast Custom processor)

2018-11-04 Thread Milan Das
Thanks Mike. I have created https://issues.apache.org/jira/browse/NIFI-5786 I will create a PR with NIFI-5786 and request for review. Thanks, Milan Das On 11/4/18, 8:00 PM, "Mike Thomsen" wrote: Milan, Yeah, add at least a Jira ticket that describes what you did. Do

Re: Secured NIFI (clustered) error on ListFile viewstate

2018-11-08 Thread Milan Das
Hello Nifi team, Wondering if it is something more I need to do. Otherwise I am planning to log a defect in Jira Thanks, Milan From: Milan Das Date: Friday, November 2, 2018 at 10:08 AM To: "dev@nifi.apache.org" Subject: Secured NIFI (clustered) error on ListFile viewstate

Re: Secured NIFI (clustered) error on ListFile viewstate

2018-11-08 Thread Milan Das
Hi Andy, Thanks for your response. I have tested it with 1.4 , 1.6 and 1.7.1 . It is external zookeeper . It works when zookeeper is not secured. Zookeeper is up and running. Otherwise NIFI cluster state will not show running. Thanks, Milan Das On 11/8/18, 6:32 PM, "Andy LoPresto&qu

Re: Custom processors/controller services without Maven

2018-11-15 Thread Milan Das
Trust me, you are bringing trouble. I am not sure if there is any user have done that. You may practice the following and try building it the nar (zip on your own). https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/ Thanks, Milan Das On 11/15/18, 4:54 PM, "Arun

Re: Secured NIFI (clustered) error on ListFile viewstate

2018-12-14 Thread Milan Das
code CONNECTIONLOSS Thanks, Milan Das On 11/8/18, 6:41 PM, "Andy LoPresto" wrote: There was an issue with malformed replicated requests causing a timeout [1] in previous versions, and this was resolved in 1.8.0. Could you try against 1.8.0? This issue had nothing to do with ZK

HortonworksSchemaRegistry kerberos/TLS support

2018-12-17 Thread Milan Das
Hello Team, I do not see an option to use Kerberos and SSL Truststore in HortonworksSchemaRegistry ? wondering if there is a way to use kerberos/TLS with HortonworksSchemaRegistry ? Thanks, Milan Milan Das Sr. System Architect email: m...@interset.com mobile: +1 678 216 5660

Re: HortonworksSchemaRegistry kerberos/TLS support

2018-12-17 Thread Milan Das
feedback would certainly be appreciated if you can give it a try. [1] https://issues.apache.org/jira/browse/NIFI-5753 Thanks, Pierre Le lun. 17 déc. 2018 à 17:36, Milan Das a écrit : > Hello Team, > > I do not see an option to use Kerberos an

Set root level access policies using NiFi REST API

2018-12-18 Thread Milan Das
I am wondering if it is possible to set root level access policies using NiFi REST API. There is an unanswerd forum. https://community.hortonworks.com/answers/213913/post.html Thanks, Milan Das

Re: Set root level access policies using NiFi REST API

2018-12-27 Thread Milan Das
min-user/> Thanks, Milan Das > On Dec 18, 2018, at 4:50 PM, Andy LoPresto wrote: > > You can also use tools like the NiFi CLI [1] or community-provided tools like > NiPyAPI [2] to exercise the REST API via command-line actions rather than > having to execute individu

Re: NIFI install and setup

2019-01-04 Thread Milan Das
1. Are you trying to do standalone Single server or cluster instance ? If standalone just untar and start NIFI. If cluster more configuration steps are needed. 2. Do you want any security ? 3. If cluster ? Do you want to use embedded or external zookeeper Thanks, Milan > On Jan 4, 2019, at 4:49

Re: NIFI DBCP connection pool not working for oracle.

2019-02-01 Thread Milan Das
1. Did you configure the Oracle JDBC Driver in DBCP connection pool /Database driver name? 2. Does JDBC driver UrL have access to NIFI runAs user ? Thanks, Milan Das > On Feb 1, 2019, at 2:18 AM, vishwa wrote: > > Hi, > > I'm trying to make a Oracle connection pool vi

Controlling number of threads in distributed mode

2019-02-13 Thread Milan Das
Hello, Is there a way to control number of thread less than cluster member ? I have a 5 node NiFI cluster and I want to run 3 instances of the processor in distributed mode. Thanks, Milan Das

Re: Controlling number of threads in distributed mode

2019-02-13 Thread Milan Das
(primary node) will be overloaded with data. Thanks, Milan Das > On Feb 13, 2019, at 4:38 PM, Joe Witt wrote: > > Milan > > The only mechanism to limit whether a given component executes on a given > nifi node or not is whether it runs on all or whether it runs on primary &g

Please add me to nifi mailing list

2017-12-19 Thread Milan Das
Hello All, I would like to start contributing to NIFI project. Please add me and guide me to be a part of the team. My introduction is quick: I work with Interset we are heavily using NIFI. I have started with creating a bug NIFI-4715 and also I have the fix. Regards, Milan Das

NIFI-4715 : ListS3 list duplicate files when incoming file throughput to S3 is high

2017-12-22 Thread Milan Das
715 The fix I did already seems ok and working for us. long maxTimestamp = currentTimestamp; Wanted to check thought from other experts or of there is any other know fix . Regards, Milan Das Sr. System Architect email: m...@interset.com mobile: +1 678 216 5660 www.interset.com

Re: DBCP Connection Pooling using multiple OJDBC Drivers

2017-12-27 Thread Milan Das
Hi Nadeem, I think you can do very similar to Kafka implementation. For different version NIFI have two different processor for Kafka 10 and 11 “PublishKafka_0_10” and “PublishKafka_0_11” Regards, Milan Das Sr. System Architect email: m...@interset.com <https://www.linkedin.com/in/milan

Nifi "event driven" scheduling strategy Custom Processor

2017-12-27 Thread Milan Das
I am trying to build a custom processor with “Event Driven”. I added   “@EventDriven” annotation. It is showing EventDriven in the scheduling strategy. Question: how to send an event to this processor to intiate the flow? What are my options? Regards, Milan Das Sr. System Architect email

Re: Will you accept contributions in Scala?

2018-02-13 Thread Milan Das
I think we should not add blindly any language but should be open to add couple of language like Scala. In Bigdata world Scala/Python/Java are widely accepted. Thanks, Milan Das Interset On 2/13/18, 10:20 AM, "Weiss, Adam" wrote: I think it makes the most sense to me for us to

Hive backward compatibily

2018-03-05 Thread Milan Das
nifi/blob/rel/nifi-1.4.0/pom.xml#L105 Wondering if there is a way for backward compatibility. Regards, Milan Das Sr. System Architect email: m...@interset.com mobile: +1 678 216 5660 www.interset.com

Re: Hive backward compatibily

2018-03-06 Thread Milan Das
specific Hive versions. > > Regards, > Matt > > > On Mon, Mar 5, 2018 at 12:29 PM, Milan Das wrote: > > > Hello NIFI Dev, > > > > I am trying to use SelectHiveQL with Hive version hive-1.1.0 (it is part > > of the

HiveConnectionPool URL with trustStorePasswrd

2018-03-12 Thread Milan Das
set trustStorePassword in encrypted format ? Database ConnectionUrl: jdbc:hive2://ip-xxx-xx-x-xxx.ec2.internal:1/default;principal=hive/_h...@co.acme.com;ssl=true;sslTrustStore=/etc/hadoop/ssl/truststore.jks;trustStorePassword=password Regard, Milan Das Sr. System Architect email: m

Re: HiveConnectionPool URL with trustStorePasswrd

2018-03-14 Thread Milan Das
18-03-12 14:49 GMT+01:00 Milan Das : Hello folks, I am connecting to Hive environment with TLS security on. In order to do that need to send trustStorePasswrd in Hive2 URL . As the configuration is in controller services, not able to find a way to set the trustStorePassword in encrypted f

Finding Performance bottleneck issue

2018-03-16 Thread Milan Das
I have a custom processor, it works as expected. But I feel there is some performance measure need to be done. I see that my processor is actually queuing up  records at source. Is there a run a load  test and do performance measure for Custom Processor? Regards, Milan Das

Re: Finding Performance bottleneck issue

2018-03-16 Thread Milan Das
be used to correlate this event with the corresponding user logon event as well as to any other security audit events generated during this logon session. Regards, Milan Das On 3/16/18, 10:56 AM, "Mike Thomsen" wrote: Milan, Can you share some details about where

Re: Finding Performance bottleneck issue

2018-03-17 Thread Milan Das
nputString = CharStreams.toString(new InputStreamReader(in, "UTF-8"));” I gained 20 times performance. Regards, Milan Das On 3/16/18, 12:07 PM, "Mike Thomsen" wrote: That seems like a very reasonable use case. You said: > I see that my processor is actually queuing up re