last exception: java.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception
Hello, in the past two weeks, I see the following error on HBase Thrift servers, we have total of about 10 Thrift servers and randomly get the following errors: 2017-02-28 10:45:56,541 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4087940, table=MBData, attempt=11/35 failed=2ops, last exception: ja va.io.IOException: Call to e26-node.fqdn.com/10.12.1.209:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutExcepti on: Call id=36009061, waitTime=180001, operationTimeout=18 expired. on e26-node.fqdn.com,60020,1487800633128, tracking started null, retrying after=10060ms, replay=2ops 2017-02-28 10:45:57,207 INFO org.apache.hadoop.hbase.client.AsyncProcess: #4084675, waiting for some tasks to finish. Expected max=0, tasksInPro gress=22 Restarting the Thrift server resolves the issue. Any ideas where should I be looking for? Thanks, Motty
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Thank you very much! Terry; issues is fixed. I bumped that up Java heap size to 512MB as well. Thanks, -Motty On 06/24/2015 09:08 AM, Terry Siu wrote: Hi Motty, On your EventServer config, check what the java heap size setting is. Ours was set to 128MB, which was much less than the default of 1GB. After bumping up our value to 512MB, the EventServer started up fine and no longer restarted. Did you have see any stderr logs for your EventServer? The only ones I saw were from stdout and they did not record any OOM. One of the Cloudera support folks said that I should see OOM errors in the stderr logs, which is one cause of the EventServer restart. Hope this helps. -Terry On Tuesday, June 23, 2015 at 10:07:29 AM UTC-7, motty cruz wrote: yes, I did upgrade to 5.4.1: *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736) I did not enable trace level as suggest! Please let me know if you find a solution, I had tried increasing memory to that service but no successful results yet. Thanks, -Motty On 06/23/2015 09:31 AM, Terry Siu wrote: Motty, Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your INFO message for the EventCatcherService and noticed that your Version number at the end says 5.3.1. On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote: Bit skeptical that enabling trace would fix this issue, but I did and no change. Event Server keeps restarting and no errors. Posted a message on the Cloudera Manager forum to see if anybody has any other ideas. On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote: Hi Motty, No, I haven't. The Event Server has been restarting constantly since I left it last night and the logs do not provide any errors on what might be wrong. Did you try enabling the trace level as described by Atul? I'll see if I can figure out where the setting is. This is getting annoying and I'm going to engage Cloudera Support for this. -Terry On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote: Hello Terry, did you find a solution to this problem? We're experiencing this issue, no solution thus far. Thanks, -Motty On 06/22/2015 03:12 PM, Terry Siu wrote: Hi guys, I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue with the Clouder Manager EventServer (unexpected exits). I tailed the event server log and saw not ERROR logs and it looked like the EventServer keeps restarting itself over and over again. The only notable thing I've seen in the log is WARN level message: 2015-06-22 15:04:10,885 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F ailed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], SEVERITY=[IMPORTANT], SERVI CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], SERVICE_TYPE=[MGMT], LOG_LEVEL= [WARN], HOSTS=[8K04.corp.pivotlink.com http://8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC E is not ROLE for input role type KMS-KMS., timestamp=1435010650675} Anyone else got this resolved/found a workaround? Thanks, -Terry On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote: Thanks for your reply Atul, I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67. I have not resolve the issue yet. do you mind sharing the steps you taken to enable trace. Thanks, Motty On 06/03/2015 06:18 PM, Atul Kulkarni wrote: We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Hello Terry, did you find a solution to this problem? We're experiencing this issue, no solution thus far. Thanks, -Motty On 06/22/2015 03:12 PM, Terry Siu wrote: Hi guys, I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue with the Clouder Manager EventServer (unexpected exits). I tailed the event server log and saw not ERROR logs and it looked like the EventServer keeps restarting itself over and over again. The only notable thing I've seen in the log is WARN level message: 2015-06-22 15:04:10,885 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F ailed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], SEVERITY=[IMPORTANT], SERVI CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], SERVICE_TYPE=[MGMT], LOG_LEVEL= [WARN], HOSTS=[8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC E is not ROLE for input role type KMS-KMS., timestamp=1435010650675} Anyone else got this resolved/found a workaround? Thanks, -Terry On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote: Thanks for your reply Atul, I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67. I have not resolve the issue yet. do you mind sharing the steps you taken to enable trace. Thanks, Motty On 06/03/2015 06:18 PM, Atul Kulkarni wrote: We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some intermediate data. As this is not a perfect or even sensible solution this may not work for you, we are still investigating the real reason but since we have not been able reproduce it - we are not sure how to debug further. Just thought of sharing this random act that we still can't believe turned out to be the solution. Let us know if this works for you too. Regards, Atul. On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote: Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings eventserver (cloudera1) http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status Process Status, Unexpected Exits 2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83) Any ideas? we're using Java 8 Thanks, Motty -- --- You received this message because you are subscribed to the Google Groups CDH Users group. To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+u...@cloudera.org javascript:. For more options, visit https://groups.google.com/a/cloudera.org/d/optout https://groups.google.com/a/cloudera.org/d/optout. -- --- You received this message because you are subscribed to the Google Groups CDH Users group. To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscr...@cloudera.org mailto:cdh-user+unsubscr...@cloudera.org. For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
yes, I did upgrade to 5.4.1: *Version*: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736) I did not enable trace level as suggest! Please let me know if you find a solution, I had tried increasing memory to that service but no successful results yet. Thanks, -Motty On 06/23/2015 09:31 AM, Terry Siu wrote: Motty, Did you upgrade your Cloudera Manager to 5.4.1? I was comparing your INFO message for the EventCatcherService and noticed that your Version number at the end says 5.3.1. On Tuesday, June 23, 2015 at 9:26:42 AM UTC-7, Terry Siu wrote: Bit skeptical that enabling trace would fix this issue, but I did and no change. Event Server keeps restarting and no errors. Posted a message on the Cloudera Manager forum to see if anybody has any other ideas. On Tuesday, June 23, 2015 at 9:01:08 AM UTC-7, Terry Siu wrote: Hi Motty, No, I haven't. The Event Server has been restarting constantly since I left it last night and the logs do not provide any errors on what might be wrong. Did you try enabling the trace level as described by Atul? I'll see if I can figure out where the setting is. This is getting annoying and I'm going to engage Cloudera Support for this. -Terry On Tuesday, June 23, 2015 at 7:26:10 AM UTC-7, motty cruz wrote: Hello Terry, did you find a solution to this problem? We're experiencing this issue, no solution thus far. Thanks, -Motty On 06/22/2015 03:12 PM, Terry Siu wrote: Hi guys, I just updated CDH from 5.3.3 to 5.4.2 and am seeing the same issue with the Clouder Manager EventServer (unexpected exits). I tailed the event server log and saw not ERROR logs and it looked like the EventServer keeps restarting itself over and over again. The only notable thing I've seen in the log is WARN level message: 2015-06-22 15:04:10,885 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: F ailed to publish event: SimpleEvent{attributes={ROLE_TYPE=[EVENTSERVER], CATEGORY=[LOG_MESS AGE], ROLE=[mgmt-EVENTSERVER-fd434ec7afa4142d2e2f91566bafb72b], SEVERITY=[IMPORTANT], SERVI CE=[mgmt], HOST_IDS=[10a65ece-add7-4512-82bf-9f6db5c6c7ec], SERVICE_TYPE=[MGMT], LOG_LEVEL= [WARN], HOSTS=[8K04.corp.pivotlink.com http://8K04.corp.pivotlink.com], EVENTCODE=[EV_LOG_EVENT]}, content=Category SERVIC E is not ROLE for input role type KMS-KMS., timestamp=1435010650675} Anyone else got this resolved/found a workaround? Thanks, -Terry On Thursday, June 4, 2015 at 7:24:16 AM UTC-7, motty cruz wrote: Thanks for your reply Atul, I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67. I have not resolve the issue yet. do you mind sharing the steps you taken to enable trace. Thanks, Motty On 06/03/2015 06:18 PM, Atul Kulkarni wrote: We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some intermediate data. As this is not a perfect or even sensible solution this may not work for you, we are still investigating the real reason but since we have not been able reproduce it - we are not sure how to debug further. Just thought of sharing this random act that we still can't believe turned out to be the solution. Let us know if this works for you too. Regards, Atul. On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote: Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings
Re: upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Thanks for your reply Atul, I believe it has to do with version of Java. We Also updated Java to version 8 previously we were using java version 1.7-67. I have not resolve the issue yet. do you mind sharing the steps you taken to enable trace. Thanks, Motty On 06/03/2015 06:18 PM, Atul Kulkarni wrote: We had the same issue (upgrade from 5.3.3 to 5.4.0) - while trying to debug I enabled trace and it started working correctly - it was very odd, in the mean time we did not care about the previous events data, hence we also whacked directory (after backing it up, of course!) where it stores some intermediate data. As this is not a perfect or even sensible solution this may not work for you, we are still investigating the real reason but since we have not been able reproduce it - we are not sure how to debug further. Just thought of sharing this random act that we still can't believe turned out to be the solution. Let us know if this works for you too. Regards, Atul. On Wednesday, June 3, 2015 at 10:30:13 AM UTC-7, motty cruz wrote: Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings eventserver (cloudera1) http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status Process Status, Unexpected Exits 2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83) Any ideas? we're using Java 8 Thanks, Motty -- --- You received this message because you are subscribed to the Google Groups CDH Users group. To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscr...@cloudera.org mailto:cdh-user+unsubscr...@cloudera.org. For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
upgrade CDH 5.3.1 to 5.4.1 - Event Server health issue
Hello, Just finished upgrading CDH from 5.3.1 to 5.4.1 using http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/install_upgrade_to_cdh54_parcels.html however Event health issues warnings eventserver (cloudera1) http://cloudera1.sscs.ad:7180/cmf/services/6/instances/28/status Process Status, Unexpected Exits 2015-02-24 14:18:06,012 INFO com.cloudera.cmf.eventcatcher.server.EventCatcherService: Starting EventCatcherService. JVM Args: [-XX:+UseConcMarkSweepGC, -XX:-CMSConcurrentMTEnabled, -XX:+UseParNewGC, -Dmgmt.log.file=mgmt-cmf-mgmt-EVENTSERVER-cloudera1.domain.ad.log.out, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Xms52428800, -Xmx52428800, -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh], Args: [], Version: 5.3.1 (#191 built by jenkins on 20150123-2020 git: b0377087cf605a686591e659eb14078923bc3c83) Any ideas? we're using Java 8 Thanks, Motty
Re: How to Backup and Restore various components of Hadoop ?
Hi Krish, did you get a response in regards to Backup and Restore? I would like to have the ability to backup and restore if necessary. At this moment we're replicating to another cluster, however I want to be able to restore in case a table is delete and replicated to the backup cluster. Thanks, Motty On 03/01/2015 10:09 PM, Krish Donald wrote: Hi, Wanted to know the backup and recovery strategy for your Hadoop cluster. How do you Backup and Restore various components of Hadoop ? Thanks Krish
upgrade to CDH5 from CDH4.6 hadoop 2.0
Hello, I'm upgrading to CDH5. I download latest parcel from http://archive.cloudera.com/cdh5/parcels/latest/ to /oprt/cloudera/parcel-repo next to cluster on cludera under parcels -- I hit the distribution button, started to distribute got to 50% but it does not go any further. any ideas how to proceed? Thanks,
hadoop 2.0 upgrade to 2.4
Hi All, I currently have a hadoop 2.0 cluster in production, I want to upgrade to latest release. current version: [root@doop1 ~]# hadoop version Hadoop 2.0.0-cdh4.6.0 Cluster has the following services: hbase hive hue impala mapreduce oozie sqoop zookeeper can someone point me to a howto upgrade hadoop from 2.0 to hadoop 2.4.0? Thanks in advance,