Hi Priya, the container you trying to interactivily executing a command with is no longer running. It is not possible to execute command with stopped containers.
The logger issues might be related to missing file system permissions. But thats a wild guess. Is there a "Caused by" part in the stacktrace of the AppenderLoggingException? Cheers, Markus Am 20.12.2019 um 11:48 schrieb Priya Arora: > Hi All, > > When i am trying to execute bash command inside manifoldcf container > getting error. > image.png > And when checking logs Sudo docker logs <CID> > 2019-12-19 18:09:05,848 Job start thread ERROR Unable to write to stream > logs/manifoldcf.log for appender MyFile > 2019-12-19 18:09:05,848 Seeding thread ERROR Unable to write to stream > logs/manifoldcf.log for appender MyFile > 2019-12-19 18:09:05,848 Job reset thread ERROR Unable to write to stream > logs/manifoldcf.log for appender MyFile > 2019-12-19 18:09:05,848 Job notification thread ERROR Unable to write to > stream logs/manifoldcf.log for appender MyFile > 2019-12-19 18:09:05,849 Seeding thread ERROR An exception occurred > processing Appender MyFile org > .apache.logging.log4j.core.appender.AppenderLoggingException: Error > flushing stream logs/manifoldcf.log > at > org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:159) > > Can any body suggest reason behind this error? > > Thanks > Priya > > On Fri, Dec 20, 2019 at 3:37 PM Priya Arora <[email protected] > <mailto:[email protected]>> wrote: > > Hi Markus, > > Many thanks for your reply!!. > > I tried this approach to reproduce the scenario in a different > environment, but the case where I listed the error above is when I > am crawling INTRANET sites which can be accessible over a remote > server. Also I have used Transformation connectors:-Allow Documents, > Tika Parser, Content Limiter( 10000000), Metadata Adjuster. > > When tried reproducing the error with Public sites of the same > domain and on a different server(DEV), it was successful, with no > error.Also there was no any postgres related error. > > Can it depends observer related configurations like Firewall etc, as > this case include some firewall,security related configurations. > > Thanks > Priya > > > > > On Fri, Dec 20, 2019 at 3:23 PM Markus Schuch <[email protected] > <mailto:[email protected]>> wrote: > > Hi Priya, > > in my experience, i would focus on the OutOfMemoryError (OOME). > 8 Gigs can be enough, but they don't have to. > > At first i would check if the jvm is really getting the desired heap > size. The dockered environment make that a little harder find > find out, > since you need to get access to the jvm metrics, e.g. via jmxremote. > Beeing able to monitor the jvm metrics helps you with > correlating the > errors with the heap and garbage collection activity. > > The errors you see on postgresql jdbc driver might be very > related to > the OOME. > > Some question i would ask myself: > > Do the problems repeatingly occur only when crawling this specific > content source or only with this specific output connection? Can you > reproduce it outside of docker in a controlled dev environment? > Or is it > a more general problem with your manifoldcf instance? > > May be there are some huge files beeing crawled in your content > source? > To you have any kind of transformations configured? (e.g. > content size > limit?) You should try to see in the job's history if there are any > patterns, like the error rises always after encountering the same > document xy. > > Cheers > Markus > > > > Am 20.12.2019 um 09:59 schrieb Priya Arora: > > Hi Markus , > > > > Heap size defined is 8GB. Manifoldcf start-options-unix file > Xmx etc > > parameters is defined to have memory 8192mb. > > > > It seems to be an issue with memory also, and also when > manifoldcf tries > > to communicate to Database. Do you explicitly define somewhere > > connection timer when to communicate to postgres. > > Postgres is installed as a part of docker image pull and then some > > changes in properties.xml(of manifoldcf) to connect to database. > > On the other hand Elastic search is also holding sufficient > memory and > > Manifoldcf is also provided with 8 cores CPU. > > > > Can you suggest some solution. > > > > Thanks > > Priya > > > > On Fri, Dec 20, 2019 at 2:23 PM Markus Schuch > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > Hi Priya, > > > > your manifoldcf JVM suffers from high garbage collection > pressure: > > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > What is your current heap size? > > Without knowing that, i suggest to increase the heap size. > (java > > -Xmx...) > > > > Cheers, > > Markus > > > > Am 20.12.2019 um 09:02 schrieb Priya Arora: > > > Hi All, > > > > > > I am facing below error while accessing Manifoldcf. > Requirement is to > > > crawl data from a website using Repository as "Web" and > Output > > connector > > > as "Elastic Search" > > > Manifoldcf is configured inside a docker container and also > > postgres is > > > used a docker container. > > > When launching manifold getting below error > > > image.png > > > > > > When checked logs:- > > > *1)sudo docker exec -it 0b872dfafc5c tail -1000 > > > /usr/share/manifoldcf/example/logs/manifoldcf.log* > > > FATAL 2019-12-20T06:06:13,176 (Stuffer thread) - Error > tossed: Timer > > > already cancelled. > > > java.lang.IllegalStateException: Timer already cancelled. > > > at java.util.Timer.sched(Timer.java:397) > ~[?:1.8.0_232] > > > at java.util.Timer.schedule(Timer.java:193) > ~[?:1.8.0_232] > > > at > > > > org.postgresql.jdbc.PgConnection.addTimerTask(PgConnection.java:1113) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > org.postgresql.jdbc.PgStatement.startTimer(PgStatement.java:887) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:427) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > > > > org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > > > > org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > org.postgresql.jdbc.PgConnection.isValid(PgConnection.java:1311) > > > ~[postgresql-42.1.3.jar:42.1.3] > > > at > > > > > > > org.apache.manifoldcf.core.jdbcpool.ConnectionPool.getConnection(ConnectionPool.java:92) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.ConnectionFactory.getConnectionWithRetries(ConnectionFactory.java:126) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.ConnectionFactory.getConnection(ConnectionFactory.java:75) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:797) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1457) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:204) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:837) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221) > > > ~[mcf-core.jar:?] > > > at > > > > > > > org.apache.manifoldcf.crawler.jobs.Jobs.getActiveJobConnections(Jobs.java:736) > > > ~[mcf-pull-agent.jar:?] > > > at > > > > > > > org.apache.manifoldcf.crawler.jobs.JobManager.getNextDocuments(JobManager.java:2869) > > > ~[mcf-pull-agent.jar:?] > > > at > > > > > > > org.apache.manifoldcf.crawler.system.StufferThread.run(StufferThread.java:186) > > > [mcf-pull-agent.jar:?] > > > *2)sudo docker logs <CID> --tail 1000* > > > Exception in thread "PostgreSQL-JDBC-SharedTimer-1" > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > at java.util.ArrayList.iterator(ArrayList.java:840) > > > at > > > > > > > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316) > > > at > java.net.InetAddress.getAllByName0(InetAddress.java:1277) > > > at > java.net.InetAddress.getAllByName(InetAddress.java:1193) > > > at > java.net.InetAddress.getAllByName(InetAddress.java:1127) > > > at > java.net.InetAddress.getByName(InetAddress.java:1077) > > > at > > java.net.InetSocketAddress.<init>(InetSocketAddress.java:220) > > > at > org.postgresql.core.PGStream.<init>(PGStream.java:66) > > > at > > > > > > > org.postgresql.core.QueryExecutorBase.sendQueryCancel(QueryExecutorBase.java:155) > > > at > > > > org.postgresql.jdbc.PgConnection.cancelQuery(PgConnection.java:971) > > > at > > org.postgresql.jdbc.PgStatement.cancel(PgStatement.java:812) > > > at > org.postgresql.jdbc.PgStatement$1.run(PgStatement.java:880) > > > at java.util.TimerThread.mainLoop(Timer.java:555) > > > at java.util.TimerThread.run(Timer.java:505) > > > 2019-12-19 18:09:05,848 Job start thread ERROR Unable to > write to > > stream > > > logs/manifoldcf.log for appender MyFile > > > 2019-12-19 18:09:05,848 Seeding thread ERROR Unable to > write to stream > > > logs/manifoldcf.log for appender MyFile > > > 2019-12-19 18:09:05,848 Job reset thread ERROR Unable to > write to > > stream > > > logs/manifoldcf.log for appender MyFile > > > 2019-12-19 18:09:05,848 Job notification thread ERROR > Unable to > > write to > > > stream logs/manifoldcf.log for appender MyFile > > > 2019-12-19 18:09:05,849 Seeding thread ERROR An > exception occurred > > > processing Appender MyFile > > > > org.apache.logging.log4j.core.appender.AppenderLoggingException: > Error > > > flushing stream logs/manifoldcf.log > > > at > > > > > > > org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:159). > > > > > > _Also tried the approach to clean up Database by > truncating all > > > manifoldcf related tables, but still getting this error._ > > > > > > Parameters defined in *postgresql conf *file is as > suggested :- and > > > "max_pred_per_locks_transctions" is set to value "256". > > > image.png > > >
