[
https://issues.apache.org/jira/browse/SLIDER-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Maron resolved SLIDER-333.
-----------------------------------
Resolution: Fixed
> Slider fails to create/start an application when 2-way SSL is enabled
> ---------------------------------------------------------------------
>
> Key: SLIDER-333
> URL: https://issues.apache.org/jira/browse/SLIDER-333
> Project: Slider
> Issue Type: Bug
> Affects Versions: Slider 0.40
> Reporter: Gour Saha
>
> Enabled 2-way SSL by setting "ssl.server.client.auth" to "true" in
> appConfig.json, but am seeing the following errors in AM and agent logs. This
> is happening in the latest develop branch.
> AM Log:
> {noformat}
> 14/08/18 16:46:54 INFO appmaster.SliderAppMaster: Connecting to RM at
> 38306,address tracking URL=http://c6409.ambari.apache.org:38315
> 14/08/18 16:46:54 INFO agent.AgentUtils: Reading metainfo at
> apache-slider-hbase-0.98.3-hadoop2-app-package-0.31.0-incubating-SNAPSHOT.zip
> 14/08/18 16:46:54 INFO tools.SliderUtils: Reading metainfo.xml of size 3802
> 14/08/18 16:46:54 INFO agent.HeartbeatMonitor: Starting heartbeat monitor
> with interval 60000
> 14/08/18 16:46:54 INFO state.AppState: Adding new role HBASE_REGIONSERVER
> 14/08/18 16:46:54 INFO state.AppState: Role HBASE_REGIONSERVER assigned
> priority 2
> 14/08/18 16:46:54 INFO state.AppState: Adding new role HBASE_MASTER
> 14/08/18 16:46:54 INFO state.AppState: Role HBASE_MASTER assigned priority 1
> 14/08/18 16:46:54 INFO state.AppState: Role slider-appmaster flexed from 0 to
> 1
> 14/08/18 16:46:54 INFO state.AppState: Role HBASE_REGIONSERVER flexed from 0
> to 1
> 14/08/18 16:46:54 INFO state.AppState: Role HBASE_MASTER flexed from 0 to 1
> 14/08/18 16:46:54 INFO state.RoleHistory: Role history bootstrapped
> 14/08/18 16:46:54 INFO appmaster.SliderAppMaster: service instances already
> running: []
> 14/08/18 16:46:55 INFO curator.RegistryBinderService: registering
> ServiceInstance{name='org-apache-slider', id='cl1', address='192.168.64.109',
> port=38315, sslPort=null, payload=ServiceInstanceData{id='cl1',
> serviceType='org-apache-slider'}, registrationTimeUTC=1408380415215,
> serviceType=DYNAMIC, uriSpec=org.apache.curator.x.discovery.UriSpec@bd91710d}
> 14/08/18 16:46:55 INFO curator.RegistryBinderService: registration completed
> ServiceInstance{name='org-apache-slider', id='cl1', address='192.168.64.109',
> port=38315, sslPort=null, payload=ServiceInstanceData{id='cl1',
> serviceType='org-apache-slider'}, registrationTimeUTC=1408380415215,
> serviceType=DYNAMIC, uriSpec=org.apache.curator.x.discovery.UriSpec@bd91710d}
> 14/08/18 16:46:55 INFO appmaster.SliderAppMaster: Chaos monkey disabled
> 14/08/18 16:46:55 INFO appmaster.SliderAppMaster: Adding Chaos Monkey
> scheduled every 0 seconds (0 hours)
> 14/08/18 16:46:55 INFO workflow.WorkflowCompositeService: Child service
> completed Service SliderAMProviderService in state SliderAMProviderService:
> STOPPED; current service null; queued service count=0
> 14/08/18 16:46:55 INFO appmaster.SliderAppMaster: Process has exited with
> exit code 0 mapped to 0 -ignoring
> 14/08/18 16:46:55 INFO state.AppState: RoleStatus{name='HBASE_REGIONSERVER',
> key=2, desired=1, actual=0, requested=0, releasing=0, failed=0, started=0,
> startFailed=0, completed=0, failureMessage=''}
> 14/08/18 16:46:55 INFO state.AppState: HBASE_REGIONSERVER: Asking for 1 more
> nodes(s) for a total of 1
> 14/08/18 16:46:55 INFO state.RoleHistory: There're 0 nodes to consider for
> HBASE_REGIONSERVER
> 14/08/18 16:46:55 INFO state.AppState: Container ask is
> Capability[<memory:256, vCores:1>]Priority[1073741826]
> 14/08/18 16:46:55 INFO state.AppState: RoleStatus{name='HBASE_MASTER', key=1,
> desired=1, actual=0, requested=0, releasing=0, failed=0, started=0,
> startFailed=0, completed=0, failureMessage=''}
> 14/08/18 16:46:55 INFO state.AppState: HBASE_MASTER: Asking for 1 more
> nodes(s) for a total of 1
> 14/08/18 16:46:55 INFO state.RoleHistory: There're 0 nodes to consider for
> HBASE_MASTER
> 14/08/18 16:46:55 INFO state.AppState: Container ask is
> Capability[<memory:256, vCores:1>]Priority[1073741825]
> 14/08/18 16:46:57 INFO impl.AMRMClientImpl: Received new token for :
> c6409.ambari.apache.org:45454
> 14/08/18 16:46:57 INFO appmaster.SliderAppMaster: onContainersAllocated(1)
> 14/08/18 16:46:57 INFO state.AppState: Assigning role HBASE_MASTER to
> container container_1407891977820_0028_01_000002, on
> c6409.ambari.apache.org:45454,
> 14/08/18 16:46:57 INFO appmaster.SliderAppMaster: Diagnostics:
> RoleStatus{name='slider-appmaster', key=0, desired=1, actual=0, requested=0,
> releasing=0, failed=0, started=0, startFailed=0, completed=0,
> failureMessage=''}
> RoleStatus{name='HBASE_REGIONSERVER', key=2, desired=1, actual=0,
> requested=1, releasing=0, failed=0, started=0, startFailed=0, completed=0,
> failureMessage=''}
> RoleStatus{name='HBASE_MASTER', key=1, desired=1, actual=1, requested=0,
> releasing=0, failed=0, started=0, startFailed=0, completed=0,
> failureMessage=''}
> 14/08/18 16:46:57 INFO agent.AgentProviderService: Build launch context for
> Agent
> 14/08/18 16:46:57 INFO agent.AgentProviderService: AGENT_WORK_ROOT set to $PWD
> 14/08/18 16:46:57 INFO agent.AgentProviderService: AGENT_LOG_ROOT set to
> $LOG_DIRS
> 14/08/18 16:46:57 INFO agent.AgentProviderService: PYTHONPATH set to
> ./infra/agent/slider-agent/
> 14/08/18 16:46:57 INFO agent.AgentProviderService: Using
> ./infra/agent/slider-agent/agent/main.py for agent.
> 14/08/18 16:46:57 INFO appmaster.RoleLaunchService: Starting container with
> command: python ./infra/agent/slider-agent/agent/main.py --label
> container_1407891977820_0028_01_000002___HBASE_MASTER --zk-quorum
> c6409.ambari.apache.org:2181 --zk-reg-path /registry/org-apache-slider/cl1 ;
> 14/08/18 16:46:57 INFO impl.NMClientAsyncImpl: Processing Event EventType:
> START_CONTAINER for Container container_1407891977820_0028_01_000002
> 14/08/18 16:46:57 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> c6409.ambari.apache.org:45454
> 14/08/18 16:46:57 INFO appmaster.SliderAppMaster: Started Container
> container_1407891977820_0028_01_000002
> 14/08/18 16:46:57 INFO appmaster.SliderAppMaster: Deployed instance of role
> HBASE_MASTER onto container_1407891977820_0028_01_000002
> 14/08/18 16:46:57 INFO appmaster.SliderAppMaster: Registering component
> container_1407891977820_0028_01_000002
> 14/08/18 16:46:57 INFO impl.NMClientAsyncImpl: Processing Event EventType:
> QUERY_CONTAINER for Container container_1407891977820_0028_01_000002
> 14/08/18 16:46:58 INFO appmaster.SliderAppMaster: onContainersAllocated(1)
> 14/08/18 16:46:58 INFO state.AppState: Assigning role HBASE_REGIONSERVER to
> container container_1407891977820_0028_01_000003, on
> c6409.ambari.apache.org:45454,
> 14/08/18 16:46:58 INFO appmaster.SliderAppMaster: Diagnostics:
> RoleStatus{name='slider-appmaster', key=0, desired=1, actual=0, requested=0,
> releasing=0, failed=0, started=0, startFailed=0, completed=0,
> failureMessage=''}
> RoleStatus{name='HBASE_REGIONSERVER', key=2, desired=1, actual=1,
> requested=0, releasing=0, failed=0, started=0, startFailed=0, completed=0,
> failureMessage=''}
> RoleStatus{name='HBASE_MASTER', key=1, desired=1, actual=1, requested=0,
> releasing=0, failed=0, started=1, startFailed=0, completed=0,
> failureMessage=''}
> 14/08/18 16:46:58 INFO agent.AgentProviderService: Build launch context for
> Agent
> 14/08/18 16:46:58 INFO agent.AgentProviderService: AGENT_WORK_ROOT set to $PWD
> 14/08/18 16:46:58 INFO agent.AgentProviderService: AGENT_LOG_ROOT set to
> $LOG_DIRS
> 14/08/18 16:46:58 INFO agent.AgentProviderService: PYTHONPATH set to
> ./infra/agent/slider-agent/
> 14/08/18 16:46:58 INFO agent.AgentProviderService: Using
> ./infra/agent/slider-agent/agent/main.py for agent.
> 14/08/18 16:46:58 INFO appmaster.RoleLaunchService: Starting container with
> command: python ./infra/agent/slider-agent/agent/main.py --label
> container_1407891977820_0028_01_000003___HBASE_REGIONSERVER --zk-quorum
> c6409.ambari.apache.org:2181 --zk-reg-path /registry/org-apache-slider/cl1 ;
> 14/08/18 16:46:58 INFO impl.NMClientAsyncImpl: Processing Event EventType:
> START_CONTAINER for Container container_1407891977820_0028_01_000003
> 14/08/18 16:46:58 INFO appmaster.SliderAppMaster: Started Container
> container_1407891977820_0028_01_000003
> 14/08/18 16:46:58 INFO appmaster.SliderAppMaster: Deployed instance of role
> HBASE_REGIONSERVER onto container_1407891977820_0028_01_000003
> 14/08/18 16:46:58 INFO appmaster.SliderAppMaster: Registering component
> container_1407891977820_0028_01_000003
> 14/08/18 16:46:58 INFO impl.NMClientAsyncImpl: Processing Event EventType:
> QUERY_CONTAINER for Container container_1407891977820_0028_01_000003
> 14/08/18 16:47:00 WARN mortbay.log: javax.net.ssl.SSLHandshakeException: null
> cert chain
> 14/08/18 16:47:00 WARN mortbay.log: javax.net.ssl.SSLHandshakeException: null
> cert chain
> 14/08/18 16:47:10 WARN mortbay.log: javax.net.ssl.SSLHandshakeException: null
> cert chain
> {noformat}
> Agent Log:
> {noformat}
> INFO 2014-08-18 16:46:59,525 main.py:85 - loglevel=logging.INFO
> INFO 2014-08-18 16:46:59,525 main.py:245 - Using AGENT_WORK_ROOT =
> /hadoop/yarn/local/usercache/yarn/appcache/application_1407891977820_0028/container_1407891977820_0028_01_000002
> INFO 2014-08-18 16:46:59,526 main.py:246 - Using AGENT_LOG_ROOT =
> /hadoop/yarn/log/application_1407891977820_0028/container_1407891977820_0028_01_000002
> INFO 2014-08-18 16:46:59,526 main.py:256 - Connecting to the server at:
> https://c6409.ambari.apache.org:55659/ws/v1/slider/agents/
> INFO 2014-08-18 16:46:59,526 NetUtil.py:67 - DEBUG: Trying to connect to the
> server at https://c6409.ambari.apache.org:55659/ws/v1/slider/agents/
> INFO 2014-08-18 16:46:59,526 NetUtil.py:38 - Connecting to the following url
> https://c6409.ambari.apache.org:55659/ws/v1/slider/agents/
> ERROR 2014-08-18 16:47:00,013 NetUtil.py:52 - [Errno 8] _ssl.c:490: EOF
> occurred in violation of protocol
> ERROR 2014-08-18 16:47:00,014 NetUtil.py:54 - SSLError: Failed to connect.
> Please check openssl library versions.
> Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more
> details.
> INFO 2014-08-18 16:47:00,014 NetUtil.py:76 - Server at
> https://c6409.ambari.apache.org:55659/ws/v1/slider/agents/ is not reachable,
> sleeping for 10 seconds...
> INFO 2014-08-18 16:47:10,024 NetUtil.py:38 - Connecting to the following url
> https://c6409.ambari.apache.org:55659/ws/v1/slider/agents/
> ERROR 2014-08-18 16:47:10,310 NetUtil.py:52 - [Errno 8] _ssl.c:490: EOF
> occurred in violation of protocol
> ERROR 2014-08-18 16:47:10,310 NetUtil.py:54 - SSLError: Failed to connect.
> Please check openssl library versions.
> Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more
> details.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
