Hi all,
Manifold CF crashes.
Raising this issue again, after enhancing memory and otherwise, still
facing the same issue. TRied every possible solution If anybody can help ,
please suggest me some solution.
*Problem:-* Manifoldcf crashes while crawling data from an intranet URL
hinting out below error in logs:-
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.HashMap.resize(HashMap.java:704)
at java.util.HashMap.putVal(HashMap.java:629)
at java.util.HashMap.put(HashMap.java:612)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndEx
ecute(CacheManager.java:135)
at
org.apache.manifoldcf.agents.transformationconnection.TransformationC
onnectionManager.loadMultipleInternal(TransformationConnectionManager.java:265)
at
org.apache.manifoldcf.agents.transformationconnection.TransformationC
onnectionManager.loadMultiple(TransformationConnectionManager.java:246)
agents process could not start - shutting down
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:11
93)
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:15
77)
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:36
7)
at
org.apache.manifoldcf.core.database.Database.execute(Database.java:87
3)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(D
atabase.java:696)
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
[Thread-491] INFO org.eclipse.jetty.server.ServerConnector - Stopped
ServerConne
ctor@3a4621bd{HTTP/1.1}{0.0.0.0:8345}
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
[Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.
j.w.WebAppContext@6a57ae10{/mcf-api-service,file:/tmp/jetty-0.0.0.0-8345-mcf-api
-service.war-_mcf-api-service-any-2010273384908625074.dir/webapp/,UNAVAILABLE}{/
usr/share/manifoldcf/example/./../web/war/mcf-api-service.war}
[Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.
j.w.WebAppContext@51c693d{/mcf-authority-service,file:/tmp/jetty-0.0.0.0-8345-mc
f-authority-service.war-_mcf-authority-service-any-1431262612693374489.dir/webap
p/,UNAVAILABLE}{/usr/share/manifoldcf/example/./../web/war/mcf-authority-service
.war}
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
Exception in thread "PostgreSQL-JDBC-SharedTimer-1"
java.lang.OutOfMemoryError:
Java heap space
All configuration are:-
I have -Xms1024m ,-Xmx1024m memory allocated in
*start-options.env.unix, start-options.env.win file.*
Also Configuration:-
1) *For Crawler server *- 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
E5-2660v3 @ 2.60GHz and
2) *For Elasticsearch server* - 48GB and 16-Core Intel(R) Xeon(R) CPU
E5-2660 v3 @ 2.60GHz and i am using postgres as database.
3) Database as PostgreSQL
4) No of worker threads:-20.
5) manifold Version :-2.13
Can anybody please suggest some solution to get rid of this error and
crash, as whenever this istutaion comes, manifoldCF UI results me this:_
[image: image.png]
Any suggestion would be highly appreciated.
Thanks
Priya
On Fri, Aug 16, 2019 at 6:41 PM Priya Arora <pr...@smartshore.nl> wrote:
> Sure I will try to do that.. and will get back to you
>
> Sent from my iPhone
>
> > On 16-Aug-2019, at 6:23 PM, Michael Cizmar <mich...@michaelcizmar.com>
> wrote:
> >
> > Priya - Was this right?
> >
> > 48GB and 1-Core Intel(R) Xeon(R) CPU
> >
> > While not directly related to an out of memory issue, you should have
> more
> > cores allocated to ES.
> >
> >> On Fri, Aug 16, 2019 at 1:09 AM Priya Arora <pr...@smartshore.nl>
> wrote:
> >>
> >> *Existing Threads/connections configuration is :-*
> >>
> >> How many worker threads do you have? - 15 worker threads has been
> >> allocated(in properties.xml file).
> >> And the Tika Extractor connections -10 connections are defined.
> >>
> >> Is this suggested to reduce the number more.
> >> If not, what else can be a solution
> >>
> >> Thanks
> >> Priya
> >>
> >>
> >>
> >>> On Wed, Aug 14, 2019 at 5:32 PM Karl Wright <daddy...@gmail.com>
> wrote:
> >>>
> >>> How many worker threads do you have?
> >>> Even if each worker thread is constrained in memory, and they should
> be,
> >>> you can easily cause things to run out of memory by giving too many
> >> worker
> >>> threads. Another way to keep Tika's usage constrained would be to
> reduce
> >>> the number of Tika Extractor connections, because that effectively
> limits
> >>> the number of extractions that can be going on at the same time.
> >>>
> >>> Karl
> >>>
> >>>
> >>>> On Wed, Aug 14, 2019 at 7:23 AM Priya Arora <pr...@smartshore.nl>
> wrote:
> >>>>
> >>>> Yes , I am using Tika Extractor. And the version used for manifold is
> >>> 2.13.
> >>>> Also I am using postgres as database.
> >>>>
> >>>> I have 4 types of jobs
> >>>> One is accessing/re crawling data from a public site. Other three are
> >>>> accessing intranet site.
> >>>> Out of which two are giving me correct output-without any error and
> >> third
> >>>> one which is having data more than the other two , and giving me this
> >>>> error.
> >>>>
> >>>> Is there any possibility with site accessibility issue. Can you please
> >>>> suggest some solution
> >>>> Thanks and regards
> >>>> Priya
> >>>>
> >>>> On Wed, Aug 14, 2019 at 3:11 PM Karl Wright <daddy...@gmail.com>
> >> wrote:
> >>>>
> >>>>> I will need to know more. Do you have the tika extractor in your
> >>>>> pipeline? If so, what version of ManifoldCF are you using? Tika has
> >>> had
> >>>>> bugs related to memory consumption in the past; the out of memory
> >>>> exception
> >>>>> may be coming from it and therefore a stack trace is critical to
> >> have.
> >>>>>
> >>>>> Alternatively, you can upgrade to the latest version of MCF (2.13)
> >> and
> >>>> that
> >>>>> has a newer version of Tika without those problem. But you may need
> >> to
> >>>> get
> >>>>> the agents process more memory.
> >>>>>
> >>>>> Another possible cause is that you're using hsqldb in production.
> >>> HSQLDB
> >>>>> keeps all of its tables in memory. If you have a large crawl, you do
> >>> not
> >>>>> want to use HSQLDB.
> >>>>>
> >>>>> Thanks,
> >>>>> Karl
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 14, 2019 at 3:41 AM Priya Arora <pr...@smartshore.nl>
> >>> wrote:
> >>>>>
> >>>>>> Hi Karl,
> >>>>>>
> >>>>>> Manifold CF logs hints out me an error like :
> >>>>>> agents process ran out of memory - shutting down
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> Also I have -Xms1024m ,-Xmx1024m memory allocated in
> >>>>>> start-options.env.unix, start-options.env.win file.
> >>>>>> Also Configuration:-
> >>>>>> 1) For Crawler server - 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
> >>>> E5-2660
> >>>>>> v3 @ 2.60GHz and
> >>>>>>
> >>>>>> 2) For Elasticsearch server - 48GB and 1-Core Intel(R) Xeon(R) CPU
> >>>>> E5-2660
> >>>>>> v3 @ 2.60GHz and i am using postgres as database.
> >>>>>>
> >>>>>> Can you please help me out, what to do in this case.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Priya
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Aug 14, 2019 at 12:33 PM Karl Wright <daddy...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> The error occurs, I believe, as the result of basic connection
> >>>>> problems,
> >>>>>>> e.g. the connection is getting rejected. You can find more
> >>>> information
> >>>>>> in
> >>>>>>> the simple history, and in the manifoldcf log.
> >>>>>>>
> >>>>>>> I would like to know the underlying cause, since the connector
> >>> should
> >>>>> be
> >>>>>>> resilient against errors of this kind.
> >>>>>>>
> >>>>>>> Karl
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Aug 14, 2019, 1:46 AM Priya Arora <pr...@smartshore.nl>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Karl,
> >>>>>>>>
> >>>>>>>> I have an web Repository connector(Seeds:- an intranet Site).,
> >>> and
> >>>>> job
> >>>>>> i
> >>>>>>>> son Production server.
> >>>>>>>>
> >>>>>>>> When i ran job on PROD, the job stops itself 2 times with and
> >>>>>>> error:Error:
> >>>>>>>> Unexpected HTTP result code: -1: null.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Can you please provide me an idea, in which it happens so?
> >>>>>>>>
> >>>>>>>> Thanks and regards
> >>>>>>>> Priya Arora
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
