[CANCEL] [VOTE] Release Apache NiFi MiNiFi 0.3.0

2017-12-13 Thread Aldrin Piri 
The integration testing definitely isn't just your environment and is
certainly from the change to alpine as a base image.  It looks like Koji
reported some issues around the Windows service installation/support which
I think is important to make sure we get right, so I will be cancelling
this vote/RC.

I've seen some other items arrive, so we can look to get all those
incorporated and generate an RC2.

Thanks for the findings!


On Wed, Dec 13, 2017 at 1:54 AM, Joe Percivall 
wrote:

> There are some really cool new features in this release and I'm really
> looking forward to seeing NiFi + MiNiFi + Registry all working together.
> Thanks everyone for their hard work!
>
> I was able to verify the hashes and the normal build + contrib check work
> on OSX 10.12.5. Also, a simple S2S flow communicating to NiFi 1.4.0 works
> as expected. Unfortunately, I am continually hitting MINIFI-416[1] when
> running the tests in minifi-integration-tests (either using the command
> line or IDE). I believe the cause of which is the switch to Alpine Linux as
> the base docker image. That said, I'd like to get verification that this
> isn't just a configuration issue on my end before voting "-1".
>
> Also, I was surprised we haven't upgraded past NiFi-1.2.0 yet. I went ahead
> and created a ticket for it[2] and put up a PR[3]. Not something that needs
> to be in the release, I just wanted to bring attention to it.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-416
> [2] https://issues.apache.org/jira/browse/MINIFI-417
> [3] https://github.com/apache/nifi-minifi/pull/103
>
> Joe
>
> On Sun, Dec 10, 2017 at 6:42 PM, Aldrin Piri  wrote:
>
> > Hello,
> >
> > I am pleased to call this vote for the source release of Apache NiFi
> > MiNiFi, minifi-0.3.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1112
> >
> > The Git tag is minifi-0.3.0-RC1
> > The Git commit ID is 930c3fb0112c043614158018d0d8f1ca31a4f1d8
> > * https://git-wip-us.apache.org/repos/asf?p=nifi-minifi.git;a=
> > commit;h=930c3fb0112c043614158018d0d8f1ca31a4f1d8
> > * https://github.com/apache/nifi-minifi/commit/930c3fb0112c043
> > 614158018d0d8f1ca31a4f1d8
> >
> > Checksums of minifi-0.3.0-source-release.zip:
> > MD5:  69f2e8b90af7a7bdec45684406c7b83d
> > SHA1:  bc0e8e1edb13f9f6ea69f19cfb70d18443b6604f
> > SHA256:  b11c6df7c82479d77c4c245cfb737fe15cb2bd4742c962b08375485bbbe2
> 894e
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/aldrin.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 18 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > ctId=12319921&version=12340598
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/MINIFI/Release+N
> > otes#ReleaseNotes-Version0.3.0
> >
> > The vote will be open for at 72 hours and close at 7:30PM EDT, 13
> December
> > 2017 [1].
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  Then please vote:
> >
> > [ ] +1 Release this package as minifi-0.3.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks!
> >
> > [1] You can determine this time for your local time zone at
> > https://s.apache.org/minifi-0.3.0-rc1-close
> >
> >
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Re: Unable to checkpoint FlowFile Repository (No space left on device)

2017-12-13 Thread 尹文才 
Hi Michael, the no space left on device occurred again and I checked the
inodes at the time and found it was indeed full, why would the inodes
become full and are there any solutions to get around this problem? Thanks.

/Ben

2017-12-13 13:36 GMT+08:00 尹文才 :

> Hi Michael, I checked the system available inodes by running df -i command
> and there're quite enough inodes in the system. I then removed all the
> files in all repository folders and restarted
> the system, I couldn't see the error again. I will continue to track the
> problem to see what's causing it, but it seems not relevant to the inode
> use-up reason you mentioned. Thanks.
>
> /Ben
>
> 2017-12-12 23:45 GMT+08:00 Michael Moser :
>
>> Greetings Ben,
>>
>> The "No space left on device" error can also be caused by running out of
>> inodes on your device.  You can check this with "df -i".
>>
>> -- Mike
>>
>>
>> On Tue, Dec 12, 2017 at 1:36 AM, 尹文才  wrote:
>>
>> > sorry that I forgot to mention the environment that caused this problem,
>> > I'm using the latest nifi 1.4.0 release and installed it on centos 7.
>> >
>> > 2017-12-12 14:35 GMT+08:00 尹文才 :
>> >
>> > > Hi guys, I'm running into a very weird problem, I wrote a processor
>> > > specifically to extract some data
>> > > and I found starting from yesterday it kept showing errors in the
>> log, as
>> > > below:
>> > >
>> > > 2017-12-12 14:01:04,661 INFO [pool-10-thread-1] o.a.n.c.r.
>> > WriteAheadFlowFileRepository
>> > > Initiating checkpoint of FlowFile Repository
>> > > 2017-12-12 14:01:04,676 ERROR [pool-10-thread-1] o.a.n.c.r.
>> > WriteAheadFlowFileRepository
>> > > Unable to checkpoint FlowFile Repository due to
>> > > java.io.FileNotFoundException: ../flowfile_repository/
>> > partition-5/96.journal
>> > > (No space left on device)
>> > > java.io.FileNotFoundException: ../flowfile_repository/
>> > partition-5/96.journal
>> > > (No space left on device)
>> > > at java.io.FileOutputStream.open0(Native Method)
>> > > at java.io.FileOutputStream.open(FileOutputStream.java:270)
>> > > at java.io.FileOutputStream.(FileOutputStream.java:213)
>> > > at java.io.FileOutputStream.(FileOutputStream.java:162)
>> > > at org.wali.MinimalLockingWriteAheadLog$Partition.rollover(
>> > > MinimalLockingWriteAheadLog.java:779)
>> > > at org.wali.MinimalLockingWriteAheadLog.checkpoint(
>> > > MinimalLockingWriteAheadLog.java:528)
>> > > at org.apache.nifi.controller.repository.
>> > > WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.
>> > > java:451)
>> > > at org.apache.nifi.controller.repository.
>> > > WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.
>> > java:423)
>> > > at java.util.concurrent.Executors$RunnableAdapter.
>> > > call(Executors.java:511)
>> > > at java.util.concurrent.FutureTask.runAndReset(
>> > > FutureTask.java:308)
>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
>> > > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
>> > > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > > ThreadPoolExecutor.java:1142)
>> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > > ThreadPoolExecutor.java:617)
>> > > at java.lang.Thread.run(Thread.java:745)
>> > >
>> > >
>> > > I noticed the log mentioned no space left on device and I went to
>> check
>> > > the available space and found 33G left. Does anyone know what could
>> > > possibly cause this and how to resolve this problem, thanks
>> > >
>> > > /Ben
>> > >
>> >
>>
>
>

Re: NiFI 1.4.0 UI can't be displayed in an IFrame?

2017-12-13 Thread Andy LoPresto 
Virgil,

This was intentionally introduced via NIFI-3907 [1] in Apache NiFi 1.3.0 as a 
mitigation for CVE-2017-7667 [2]. Prior to this change, a malicious site could 
have displayed the NiFi UI and introduced invisible overlays such that an 
unsuspecting user would perform actions like entering sensitive credentials 
into a malicious form field. See here [3] and here [4] for further information 
on Cross Frame Scripting / Clickjacking, as the attack is called.

If you have some kind of enterprise portal and have a legitimate need to 
display a NiFi UI within a frame that is not hosted on the same origin, you can 
resort to modifying the value provided to the response header in the filter 
here [5]. If you need this as an included feature in NiFi (for example, a 
configurable URI in nifi.properties), I suggest raising a Jira ticket, but I 
have to caution that it would be a low priority, as this actively weakens the 
security of the system and is not a common use case.

[1] https://issues.apache.org/jira/browse/NIFI-3907
[2] https://nifi.apache.org/security.html#CVE-2017-7667
[3] https://www.owasp.org/index.php/Cross_Frame_Scripting 

[4] http://msdn.microsoft.com/en-us/library/ms533028%28VS.85%29.aspx 

[5] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L1000
 



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Dec 12, 2017, at 10:15 AM, tanezavm  wrote:
> 
> Hi,
> 
> I tried to display NiFi 1.4.0 UI in an IFrame but it failed to load with
> error below:
> 
> Refused to display 'https://172.16.0.33:8443/nifi/' in a frame because it
> set 'X-Frame-Options' to 'sameorigin'.
> 
> Note: This setup works using NiFi 1.1.2.
> 
> Kindly advise.
> 
> 
> Thanks,
> Virgil
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Questions re: customizing a NiFi processor

2017-12-13 Thread Andy LoPresto 
Jim,

To be honest, I would recommend making those changes directly to the existing 
PublishAMQP processor, as unless I am missing something, that does not seem 
like a breaking change that would affect other users negatively. I’d suggest 
opening a Jira for the feature, cloning the GitHub repository (see the 
instructions in the Contributor Guide [1]), making the changes in your branch, 
building and testing, and then submitting a PR (or patch) for review and 
inclusion in the project. This will also require the least amount of code 
manipulation (example of a commit from Andre adding EL evaluation to a property 
descriptor [2]).

If you do not wish to do that, my next recommendation would be to clone the 
project and create a new processor in the same Maven module (and add it to the 
MANIFEST) alongside PublishAMQP (call it PublishAMQPWithEL or something) and 
extend the existing PublishAMQP class and override just the specific behavior 
you are interested in.

A possible, but in my opinion unnecessary, process would be to add a completely 
new NAR with the existing nifi-amqp-bundle/nifi-amqp-processors artifact listed 
as a dependency and perform your extension in your own NAR.

Hope this helps.

[1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide 

[2] 
https://github.com/apache/nifi/commit/84f1fb395963614a2ff78a615c92fc79a52e2942 



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Dec 2, 2017, at 3:43 PM, James McMahon  wrote:
> 
> Good evening. I am familiar with NiFi and have developed a number of
> workflows using out -of-the-box processors. I now need to develop a custom
> version of PublishAMQP that permits the use of attributes in expression
> language to set specs for the AMQP target.
> 
> I am not very familiar with Maven, but have done some initial research and
> hope I can bootstrap my effort following this link:
> https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html
> 
> Lacking any experience developing custom processors, I'd like to understand
> how I would establish in my maven project the source code for the existing
> PublishAMQP. I'd like to modify that proven code base to allow the use of
> expression language to set other configuration properties.
> 
> Can anyone help me with an example where source code for an existing
> processor is brought in from a github resource or the like to use as my
> starting point?
> 
> Thank you in advance for any assistance. -Jim



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Unable to checkpoint FlowFile Repository (No space left on device)

2017-12-13 Thread 尹文才 
One Strange thing one of our testers found was that we're using the default
12hours archive and 50% disk space configuration, he noticed that when nifi
removed the archived files inside the content_repository, he checked the
inodes ' count didn't go down, then he tried to remove some of the archived
files inside using the rm command, the inodes's count did go down.

/Ben

2017-12-14 9:49 GMT+08:00 尹文才 :

> Hi Michael, the no space left on device occurred again and I checked the
> inodes at the time and found it was indeed full, why would the inodes
> become full and are there any solutions to get around this problem? Thanks.
>
> /Ben
>
> 2017-12-13 13:36 GMT+08:00 尹文才 :
>
>> Hi Michael, I checked the system available inodes by running df -i
>> command and there're quite enough inodes in the system. I then removed all
>> the files in all repository folders and restarted
>> the system, I couldn't see the error again. I will continue to track the
>> problem to see what's causing it, but it seems not relevant to the inode
>> use-up reason you mentioned. Thanks.
>>
>> /Ben
>>
>> 2017-12-12 23:45 GMT+08:00 Michael Moser :
>>
>>> Greetings Ben,
>>>
>>> The "No space left on device" error can also be caused by running out of
>>> inodes on your device.  You can check this with "df -i".
>>>
>>> -- Mike
>>>
>>>
>>> On Tue, Dec 12, 2017 at 1:36 AM, 尹文才  wrote:
>>>
>>> > sorry that I forgot to mention the environment that caused this
>>> problem,
>>> > I'm using the latest nifi 1.4.0 release and installed it on centos 7.
>>> >
>>> > 2017-12-12 14:35 GMT+08:00 尹文才 :
>>> >
>>> > > Hi guys, I'm running into a very weird problem, I wrote a processor
>>> > > specifically to extract some data
>>> > > and I found starting from yesterday it kept showing errors in the
>>> log, as
>>> > > below:
>>> > >
>>> > > 2017-12-12 14:01:04,661 INFO [pool-10-thread-1] o.a.n.c.r.
>>> > WriteAheadFlowFileRepository
>>> > > Initiating checkpoint of FlowFile Repository
>>> > > 2017-12-12 14:01:04,676 ERROR [pool-10-thread-1] o.a.n.c.r.
>>> > WriteAheadFlowFileRepository
>>> > > Unable to checkpoint FlowFile Repository due to
>>> > > java.io.FileNotFoundException: ../flowfile_repository/
>>> > partition-5/96.journal
>>> > > (No space left on device)
>>> > > java.io.FileNotFoundException: ../flowfile_repository/
>>> > partition-5/96.journal
>>> > > (No space left on device)
>>> > > at java.io.FileOutputStream.open0(Native Method)
>>> > > at java.io.FileOutputStream.open(FileOutputStream.java:270)
>>> > > at java.io.FileOutputStream.>> >(FileOutputStream.java:213)
>>> > > at java.io.FileOutputStream.>> >(FileOutputStream.java:162)
>>> > > at org.wali.MinimalLockingWriteAheadLog$Partition.rollover(
>>> > > MinimalLockingWriteAheadLog.java:779)
>>> > > at org.wali.MinimalLockingWriteAheadLog.checkpoint(
>>> > > MinimalLockingWriteAheadLog.java:528)
>>> > > at org.apache.nifi.controller.repository.
>>> > > WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRe
>>> pository.
>>> > > java:451)
>>> > > at org.apache.nifi.controller.repository.
>>> > > WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.
>>> > java:423)
>>> > > at java.util.concurrent.Executors$RunnableAdapter.
>>> > > call(Executors.java:511)
>>> > > at java.util.concurrent.FutureTask.runAndReset(
>>> > > FutureTask.java:308)
>>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
>>> > > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
>>> > > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> > > ThreadPoolExecutor.java:1142)
>>> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> > > ThreadPoolExecutor.java:617)
>>> > > at java.lang.Thread.run(Thread.java:745)
>>> > >
>>> > >
>>> > > I noticed the log mentioned no space left on device and I went to
>>> check
>>> > > the available space and found 33G left. Does anyone know what could
>>> > > possibly cause this and how to resolve this problem, thanks
>>> > >
>>> > > /Ben
>>> > >
>>> >
>>>
>>
>>
>