Pat, As another datapoint, is your environment fully up to date with all the updates. Clicked one of the update notifications in the VM and haven't been able to recreate the problem since that happened.
Java and Maven retained the versions installed but my kernel was upgraded to: Linux nifi-virtual-machine 3.16.0-41-generic #57~14.04.1-Ubuntu SMP Thu Jun 18 18:01:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux I noticed prior I had consistently gotten CPU stuck/soft lockup notifications which also seemed to have resolved. Are you seeing any such thing in dmesg? On Fri, Jun 19, 2015 at 4:40 PM, Aldrin Piri <[email protected]> wrote: > Pat, > > I was able to recreate the first error you mentioned in a similar Ubuntu > environment as the one you presented. I have created an issue [1] to work > on it and will evaluate from there once that can be resolved. > > [1] https://issues.apache.org/jira/browse/NIFI-705 > > On Fri, Jun 19, 2015 at 12:54 PM, plj <[email protected]> wrote: > >> Howdy, >> >> I am using a different VM than I started with because I >> didn't have sudo permission on the other one. Below are the memory and >> Ubuntu release info. >> $ free -m >> total used free shared buffers >> cached >> Mem: 7985 4467 3518 1 415 3351 >> -/+ buffers/cache: 700 7285 >> Swap: 4094 0 4094 >> >> >> $ lsb_release -a >> No LSB modules are available. >> Distributor ID: Ubuntu >> Description: Ubuntu 14.04.2 LTS >> Release: 14.04 >> Codename: trusty >> >> I started using this VM when you suggested I follow the sys admin guide's >> settings. >> >> Thank you, >> >> Pat >> From: Aldrin Piri [via Apache NiFi (incubating) Developer List] [mailto: >> [email protected]] >> Sent: Friday, June 19, 2015 11:02 AM >> To: Jones, Patrick L. >> Subject: Re: mvn fail >> >> Pat, >> >> How much memory does the system have? Certainly perplexing that there are >> so many issues. >> >> I remember when you first wrote to the list you were using 12.04 LTS. I >> saw in a subsequent message that your lsb_release shows 14.04. Is the >> same >> "machine" (real or virtual) that just had updates/upgrades applied? Or is >> this a different instance? I would like to try to recreate the >> environment >> with what you have shown. >> >> Thanks! >> >> On Fri, Jun 19, 2015 at 8:29 AM, plj <[hidden >> email]</user/SendEmail.jtp?type=node&node=1972&i=0>> wrote: >> >> > Howdy, >> > >> > As suggested I ran "mvn clean install" and in another >> > session ran: >> > ps -hax -o user | grep whateverusernameyoubuildas | uniq -c >> > >> > I ran it lots of times. It gave answers like >> > $ ps -hax -o user | grep plj | uniq -c >> > 14 plj >> > >> > Then I ran it after I started getting the "Caused by: >> > java.lang.OutOfMemoryError: unable to create new native thread >> > $ ps -hax -o user | grep plj | uniq -c >> > -bash: fork: retry: Resource temporarily unavailable >> > -bash: fork: retry: Resource temporarily unavailable >> > -bash: fork: retry: Resource temporarily unavailable >> > -bash: fork: retry: Resource temporarily unavailable >> > -bash: fork: Resource temporarily unavailable >> > >> > I get the OutOfMemoryError many times and now it hangs (it used to >> finish >> > with errors) when it gets to: >> > >> > Running org.apache.nifi.processors.standard.TestListenUDP >> > Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec - >> in >> > org.apache.nifi.processors.standard.TestListenUDP >> > >> > My env is below. >> > >> > Thank you very much for the help >> > >> > Pat >> > --------------- >> > $ env >> > TERM=xterm >> > SHELL=/bin/bash >> > XDG_SESSION_COOKIE=xxx >> > SSH_CLIENT=xx.xxx.xx.xx xxxxx xx >> > SSH_TTY=/dev/pts/6 >> > http_proxy=http://xxx.xxx.org:80/ >> > USER=plj >> > >> > >> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: >> > ftp_proxy=ftp://xxx.xxx.org:80/ >> > MAVEN_OPTS=-Xms1024m -Xmx3076m -XX:MaxPermSize=256m >> > MAIL=/var/mail/plj >> > >> > >> PATH=/usr/lib/jvm/java-7-oracle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games >> > QT_QPA_PLATFORMTHEME=appmenu-qt5 >> > PWD=/home/plj/incubator-nifi/nifi >> > JAVA_HOME=/usr/lib/jvm/java-7-oracle >> > socks_proxy=socks://xxx.xxx.org:80/ >> > LANG=en_US.UTF-8 >> > https_proxy=https://xxx.xxx.org:80/ >> > KRB5CCNAME=FILE:/tmp/xxx >> > SHLVL=1 >> > HOME=/home/plj >> > LOGNAME=plj >> > SSH_CONNECTION=xxx.xxx.xx.xx xxxx xx.xx.xx.xx xx >> > LESSOPEN=| /usr/bin/lesspipe %s >> > DISPLAY=localhost:10.0 >> > LESSCLOSE=/usr/bin/lesspipe %s %s >> > _=/usr/bin/env >> > OLDPWD=/home/plj >> > >> > I also changed to a more recent java (doesn't seem to matter) >> > $ java -version >> > java version "1.7.0_80" >> > Java(TM) SE Runtime Environment (build 1.7.0_80-b15) >> > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) >> > >> > From: Joe Witt [via Apache NiFi (incubating) Developer List] [mailto: >> > [hidden email]</user/SendEmail.jtp?type=node&node=1972&i=1>] >> > Sent: Thursday, June 18, 2015 10:22 PM >> > To: Jones, Patrick L. >> > Subject: Re: mvn fail >> > >> > Pat >> > >> > You can ignore that error/warning when attempting to set the network >> > parameter. It is likely less relevant with newer kernels and it is >> > more of an optimization than a work/not work setting. >> > >> > As for the actual AssertionError it is likely a side effect of the >> > other error condition previously included which was >> > >> > "Caused by: java.lang.OutOfMemoryError: unable to create new native >> thread >> > at java.lang.Thread.start0(Native Method) >> > at java.lang.Thread.start(Thread.java:714)" >> > >> > This is the error that needs the attention. Your 'ulimit -a' output >> > shows a max of 10,000 user processes. This should be *way more* than >> > enough so there must be more to this story. While building nifi you >> > can run in another session the following >> > >> > ps -hax -o user | grep whateverusernameyoubuildas | uniq -c >> > >> > It should never get close to the max of 10000 you have indicated. How >> > much memory does this system have? Have you tried using a smaller >> > amount of memory for the build info with Maven? Consider perhaps >> > having >> > >> > MAVEN_OPTS=-Xms1024m -Xmx3076m -XX:MaxPermSize=256m >> > >> > Also can you please paste the results of typing 'env' here as well >> > (remove anything sensitive of course please). >> > >> > Thanks >> > Joe >> > >> > On Thu, Jun 18, 2015 at 1:39 PM, plj <[hidden >> > email]</user/SendEmail.jtp?type=node&node=1969&i=0>> wrote: >> > >> > > I'm still getting the errors when I run mvn clean install from the >> nifi >> > > directory. >> > > Before I said that I had done the various sys admin required. When I >> > went >> > > over it again I discovered that when I issued: >> > > $ sudo sysctl -w >> > net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait="1" >> > > I get >> > > sysctl: cannot stat >> > > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait: No >> such >> > > file or directory >> > > >> > > Note there is no directory /proc/sys/net/ipv4/netfilter and no file >> > > tcp_timeout_time_wait >> > > >> > > any thoughts? >> > > >> > > >> > > ------ error as reported previously -------- >> > > Running org.apache.nifi.processors.standard.TestHandleHttpRequest >> > > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.675 >> sec >> > > <<< FAILURE! - in >> > org.apache.nifi.processors.standard.TestHandleHttpRequest >> > > >> > >> testRequestAddedToService(org.apache.nifi.processors.standard.TestHandleHttpRequest) >> > > Time elapsed: 6.675 sec <<< FAILURE! >> > > java.lang.AssertionError: Could not invoke methods annotated with >> > > @OnScheduled annotation due to: >> > java.lang.reflect.InvocationTargetException >> > > at org.junit.Assert.fail(Assert.java:88) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:199) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:182) >> > > at >> > > >> > >> org.apache.nifi.processors.standard.TestHandleHttpRequest.testRequestAddedToService(TestHandleHttpRequest.java:100) >> > > >> > > Running org.apache.nifi.processors.standard.TestReplaceTextWithMapping >> > > Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 >> > sec - >> > > in org.apache.nifi.processors.standard.TestReplaceTextWithMapping >> > > Running org.apache.nifi.processors.standard.TestJmsConsumer >> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestJmsConsumer >> > > Running org.apache.nifi.processors.standard.TestSplitText >> > > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec >> - >> > in >> > > org.apache.nifi.processors.standard.TestSplitText >> > > Running org.apache.nifi.processors.standard.TestSplitXml >> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestSplitXml >> > > Running org.apache.nifi.processors.standard.TestBase64EncodeContent >> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestBase64EncodeContent >> > > Running org.apache.nifi.processors.standard.TestUnpackContent >> > > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestUnpackContent >> > > Running org.apache.nifi.processors.standard.TestExtractText >> > > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestExtractText >> > > Running org.apache.nifi.processors.standard.TestConvertCharacterSet >> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec >> - >> > in >> > > org.apache.nifi.processors.standard.TestConvertCharacterSet >> > > Running org.apache.nifi.processors.standard.TestSegmentContent >> > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 >> > sec - >> > > in org.apache.nifi.processors.standard.TestSegmentContent >> > > Running org.apache.nifi.processors.standard.TestCompressContent >> > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec >> - >> > in >> > > org.apache.nifi.processors.standard.TestCompressContent >> > > Running org.apache.nifi.processors.standard.TestMonitorActivity >> > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.063 >> > sec - >> > > in org.apache.nifi.processors.standard.TestMonitorActivity >> > > Running org.apache.nifi.processors.standard.TestExecuteStreamCommand >> > > Tests run: 7, Failures: 5, Errors: 1, Skipped: 1, Time elapsed: 2.723 >> sec >> > > <<< FAILURE! - in >> > > org.apache.nifi.processors.standard.TestExecuteStreamCommand >> > > >> > >> testExecuteIngestAndUpdateWithWorkingDir(org.apache.nifi.processors.standard.TestExecuteStreamCommand) >> > > Time elapsed: 0.115 sec <<< FAILURE! >> > > java.lang.AssertionError: java.lang.OutOfMemoryError: unable to create >> > new >> > > native thread >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:219) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:182) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:177) >> > > at >> > > >> > >> org.apache.nifi.processors.standard.TestExecuteStreamCommand.testExecuteIngestAndUpdateWithWorkingDir(TestExecuteStreamCommand.java:150) >> > > Caused by: java.lang.OutOfMemoryError: unable to create new native >> thread >> > > at java.lang.Thread.start0(Native Method) >> > > at java.lang.Thread.start(Thread.java:714) >> > > at >> > > >> > >> org.apache.nifi.processors.standard.ExecuteStreamCommand$StdInWriterCallback$1.process(ExecuteStreamCommand.java:376) >> > > at >> > > >> > >> org.apache.nifi.util.MockProcessSession.write(MockProcessSession.java:593) >> > > at >> > > >> org.apache.nifi.util.MockProcessSession.write(MockProcessSession.java:58) >> > > at >> > > >> > >> org.apache.nifi.processors.standard.ExecuteStreamCommand$StdInWriterCallback.process(ExecuteStreamCommand.java:357) >> > > at >> > > >> org.apache.nifi.util.MockProcessSession.read(MockProcessSession.java:411) >> > > at >> > > >> > >> org.apache.nifi.processors.standard.ExecuteStreamCommand.onTrigger(ExecuteStreamCommand.java:292) >> > > at >> > > >> > >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:269) >> > > at >> > > >> > >> org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:263) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > > at >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > > at >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:745) >> > > >> > > >> > > >> > > >> > > -- >> > > View this message in context: >> > >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/mvn-fail-tp1716p1962.html >> > > Sent from the Apache NiFi (incubating) Developer List mailing list >> > archive at Nabble.com. >> > >> > ________________________________ >> > If you reply to this email, your message will be added to the discussion >> > below: >> > >> > >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/mvn-fail-tp1716p1969.html >> > To unsubscribe from mvn fail, click here< >> > >. >> > NAML< >> > >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml >> > > >> > >> > >> > >> > >> > -- >> > View this message in context: >> > >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/mvn-fail-tp1716p1971.html >> > Sent from the Apache NiFi (incubating) Developer List mailing list >> archive >> > at Nabble.com. >> > >> >> ________________________________ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/mvn-fail-tp1716p1972.html >> To unsubscribe from mvn fail, click here< >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1716&code=cGxqQG1pdHJlLm9yZ3wxNzE2fDU1MTMzODA1 >> >. >> NAML< >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml >> > >> >> >> >> >> -- >> View this message in context: >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/mvn-fail-tp1716p1973.html >> Sent from the Apache NiFi (incubating) Developer List mailing list >> archive at Nabble.com. >> > >
