I tried to abort hung operations following this article:
https://community.hortonworks.com/articles/5461/how-to-abort-a-hung-ambari-operation.html
I executed command to stop all hung operations, restarted ambari-server but
problem still persists. Do you know how to stop hung operation? I am out of
ideas.
screenshot attached
Thanks
______________________________________________________________
Od: <jankyt...@centrum.cz>
Komu: <user@ambari.apache.org>
Datum: 14.12.2016 21:02
Předmět: Re: Ambari 2.2 to 2.4.0. update
attaching errors:
14 Dec 2016 16:20:25,736 WARN [main] Errors:173 - The following warnings have
been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.SettingService.getSettings(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.SettingService.getSetting(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.InstanceService.getInstances(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.InstanceService.getInstance(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.TargetClusterService.getTargetClusters(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.TargetClusterService.getTargetCluster(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.KerberosDescriptorService.getKerberosDescriptors(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.KerberosDescriptorService.getKerberosDescriptor(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionLinksService.getExtensionLinks(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionLinksService.getExtensionLink(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.HostService.getHosts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.HostService.getHost(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.BlueprintService.getBlueprints(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.BlueprintService.getBlueprint(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RequestService.getRequests(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RequestService.getRequest(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ClusterService.getClusters(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ClusterService.getCluster(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ClusterService.getArtifact(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ClusterService.getArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.FeedService.getFeeds(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.FeedService.getFeed(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ViewService.getViews(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ViewService.getView(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.UserService.getUsers(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.UserService.getUser(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionsService.getExtensions(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionsService.getExtension(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionsService.getExtensionVersions(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionsService.getExtensionVersion(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ExtensionsService.getExtensionVersionLinks(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStacks(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStack(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackVersions(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServices(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackConfigurations(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackLevelConfigurations(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackConfigurationDependencies(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackVersion(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getServiceComponents(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getServiceComponent(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackVersionLinks(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackLevelConfiguration(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackService(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackArtifact(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceArtifacts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceThemes(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceTheme(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceQuickLinksConfigurations(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceQuickLinksConfiguration(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackServiceArtifact(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getStackConfiguration(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getServiceComponentDependencies(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.StacksService.getServiceComponentDependency(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getServices(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getService(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getServiceComponents(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getServiceComponent(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getRootHostComponents(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getRootHosts(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getRootHost(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getRootHostComponent(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.RootServiceService.getRootHostComponent(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String,java.lang.String,java.lang.String),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ActionService.getActionDefinitions(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo),
should not consume any entity.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response
org.apache.ambari.server.api.services.ActionService.getActionDefinition(java.lang.String,javax.ws.rs.core.HttpHeaders,javax.ws.rs.core.UriInfo,java.lang.String),
should not consume any entity.
14 Dec 2016 16:20:25,819 INFO [main] AmbariServer:620 - ********* Started
Server **********
14 Dec 2016 16:20:26,866 WARN [ambari-action-scheduler] ActionScheduler:308 -
Exception received
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected
a boolean but was BEGIN_ARRAY at line 1 column 5182
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)
at com.google.gson.Gson.fromJson(Gson.java:795)
at com.google.gson.Gson.fromJson(Gson.java:761)
at com.google.gson.Gson.fromJson(Gson.java:710)
at com.google.gson.Gson.fromJson(Gson.java:682)
at
org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getCommandType(ExecutionCommandWrapper.java:213)
at
org.apache.ambari.server.actionmanager.ActionScheduler.isStageHasBackgroundCommandsOnly(ActionScheduler.java:634)
at
org.apache.ambari.server.actionmanager.ActionScheduler.filterParallelPerHostStages(ActionScheduler.java:589)
at
org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:362)
at
org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:302)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Expected a boolean but was
BEGIN_ARRAY at line 1 column 5182
at com.google.gson.stream.JsonReader.nextBoolean(JsonReader.java:483)
at com.google.gson.internal.bind.TypeAdapters$3.read(TypeAdapters.java:144)
at com.google.gson.internal.bind.TypeAdapters$3.read(TypeAdapters.java:134)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at
com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
... 10 more
______________________________________________________________
> Od: <jankyt...@centrum.cz>
> Komu: Alejandro Fernandez <afernan...@hortonworks.com>, user@ambari.apache.org
<user@ambari.apache.org>
> Datum: 14.12.2016 20:48
> Předmět: Re: Ambari 2.2 to 2.4.0. update
>
Hi Alejandro,
thanks for prompt response. Receiving below distinct errors. Hearbeat is OK. I
tried to truncate ambari.key_value_store table, restarted ambari-server and
executed call:
rl -i -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d
'{"user-pref-admin-dashboard":"","CLUSTER_CURRENT_STATUS":"{\"clusterState\":\"CLUSTER_STARTED_5\"}","admin-settings-show-bg-admin":"true","showJobsForNonAdmin":"true"}'
http://localhost:8080/api/v1/persist <http://localhost:8080/api/v1/persist>
Thanks,
Jan
______________________________________________________________
> Od: Alejandro Fernandez <afernan...@hortonworks.com>
> Komu: "user@ambari.apache.org" <user@ambari.apache.org>, "jankyt...@centrum.cz"
<jankyt...@centrum.cz>
> Datum: 14.12.2016 19:26
> Předmět: Re: Ambari 2.2 to 2.4.0. update
>
Hi Jan,
Can you attach any screenshots, ambari server log, or DB dump?
Thanks,
Alejandro
________________________________________
From: jankyt...@centrum.cz <jankyt...@centrum.cz>
Sent: Wednesday, December 14, 2016 9:10 AM
To: user@ambari.apache.org
Subject: Ambari 2.2 to 2.4.0. update
Hello,
I updated Ambari 2.2. to 2.4.0. During installation there were no errors, I
sucessfully update schema of database and started server/agents. After login to
webUI, there is operation Check Host with sub-task Check Host Actionexecute
which is stalled. What does this process mean?
Thank you
Jan
