Good. Glad to hear that. On Tue, May 23, 2017 at 6:50 PM, Laurens Vets <[email protected]> wrote:
> Hmmm, deploying Metron in AWS from another EC2 instance seems to work > fine... > > I have a fully deployed 10 node Metron install now. > > On 2017-05-19 10:52, Nick Allen wrote: > >> I am not sure what the issue is. Nothing is jumping out at me. >> >> Weird issues can arise when the machine that you are running the Ansible >> deployment from has intermittent connectivity issues to the AWS data >> center. >> >> Is Amazon's us-west-2 data center the one closest to you? You might try a >> different/closer data center. You can alter the data center here [1]. >> >> [1] >> https://github.com/apache/metron/blob/master/metron-deployme >> nt/amazon-ec2/conf/defaults.yml#L27-L39 >> >> Other than that, you could try standing up a small node in AWS and then >> use >> that to run the Metron deployment from. That always works much more >> smoothly. >> >> >> >> >> On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <[email protected]> wrote: >> >> Also, it's only 1 out of 10 hosts that failing it seems. >>> >>> >>> On 2017-05-17 13:25, Laurens Vets wrote: >>> >>> When you say Metron host, do you mean the host I'm pushing from to ec2 >>>> or the master ambari server in ec2? >>>> >>>> On 2017-05-17 11:51, Ryan Merriman wrote: >>>> >>>> That happens when you don't have the zookeeper url configured correctly. >>>>> Can you check the contents of the /etc/sysconfig/metron file on the >>>>> Metron >>>>> host? >>>>> >>>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <[email protected]> >>>>> wrote: >>>>> >>>>> For testing purposes, I decided to spin up the default Metron AWS >>>>> config. >>>>> >>>>>> This resulted in a hang from ansible here: >>>>>> >>>>>> TASK [librdkafka : include] ****************************** >>>>>> ********************** >>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>>> librdkafka/tasks/main.yml:18 >>>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c >>>>>> ompute.amazonaws.com >>>>>> >>>>>> TASK [librdkafka : Install prerequisites] >>>>>> ****************************** >>>>>> ******** >>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>>> librdkafka/tasks/dependencies.yml:18 >>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH >>>>>> CONNECTION >>>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c >>>>>> ompute.amazonaws.com >>>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: >>>>>> FutureWarning: CTR mode needs counter parameter, not IV >>>>>> self._cipher = factory.new(key, *args, **kwargs) >>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 >>>>>> && >>>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp >>>>>> -1495041091.74-92163853889508 >>>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp >>>>>> -1495041091.74-92163853889508 >>>>>> )" ) >>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT >>>>>> /tmp/tmpwnH61y >>>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>>>> 889508/yum >>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c >>>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo >>>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh; >>>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 >>>>>> /usr/bin/python >>>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>>>> 889508/yum; >>>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385 >>>>>> 3889508/" >>>>>> > /dev/null 2>&1'"'"'' >>>>>> >>>>>> Looking in the machine logs, I see the following for Kafka and Metron >>>>>> REST: >>>>>> >>>>>> Kafka: >>>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values: >>>>>> advertised.host.name = null >>>>>> metric.reporters = [] >>>>>> quota.producer.default = 9223372036854775807 >>>>>> offsets.topic.num.partitions = 50 >>>>>> log.flush.interval.messages = 9223372036854775807 >>>>>> auto.create.topics.enable = true >>>>>> controller.socket.timeout.ms = 30000 >>>>>> log.flush.interval.ms = null >>>>>> principal.builder.class = class org.apache.kafka.common.securi >>>>>> ty.auth.DefaultPrincipalBuilder >>>>>> replica.socket.receive.buffer.bytes = 65536 >>>>>> min.insync.replicas = 1 >>>>>> replica.fetch.wait.max.ms = 500 >>>>>> num.recovery.threads.per.data.dir = 1 >>>>>> ssl.keystore.type = JKS >>>>>> sasl.mechanism.inter.broker.protocol = GSSAPI >>>>>> default.replication.factor = 1 >>>>>> ssl.truststore.password = null >>>>>> log.preallocate = false >>>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT] >>>>>> fetch.purgatory.purge.interval.requests = 10000 >>>>>> ssl.endpoint.identification.algorithm = null >>>>>> replica.socket.timeout.ms = 30000 >>>>>> message.max.bytes = 1000000 >>>>>> num.io.threads = 8 >>>>>> offsets.commit.required.acks = -1 >>>>>> log.flush.offset.checkpoint.interval.ms = 60000 >>>>>> delete.topic.enable = false >>>>>> quota.window.size.seconds = 1 >>>>>> ssl.truststore.type = JKS >>>>>> offsets.commit.timeout.ms = 5000 >>>>>> quota.window.num = 11 >>>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c >>>>>> ompute.amazonaws.com:2181 >>>>>> authorizer.class.name = >>>>>> num.replica.fetchers = 1 >>>>>> log.retention.ms = null >>>>>> log.roll.jitter.hours = 0 >>>>>> log.cleaner.enable = true >>>>>> offsets.load.buffer.size = 5242880 >>>>>> log.cleaner.delete.retention.ms = 86400000 >>>>>> ssl.client.auth = none >>>>>> controlled.shutdown.max.retries = 3 >>>>>> queued.max.requests = 500 >>>>>> offsets.topic.replication.factor = 3 >>>>>> log.cleaner.threads = 1 >>>>>> sasl.kerberos.service.name = null >>>>>> sasl.kerberos.ticket.renew.jitter = 0.05 >>>>>> socket.request.max.bytes = 104857600 >>>>>> ssl.trustmanager.algorithm = PKIX >>>>>> zookeeper.session.timeout.ms = 30000 >>>>>> log.retention.bytes = -1 >>>>>> log.message.timestamp.type = CreateTime >>>>>> sasl.kerberos.min.time.before.relogin = 60000 >>>>>> zookeeper.set.acl = false >>>>>> connections.max.idle.ms = 600000 >>>>>> offsets.retention.minutes = 86400000 >>>>>> replica.fetch.backoff.ms = 1000 >>>>>> inter.broker.protocol.version = 0.10.0-IV1 >>>>>> log.retention.hours = 168 >>>>>> num.partitions = 1 >>>>>> broker.id.generation.enable = true >>>>>> listeners = PLAINTEXT://ec2-34-209-53-166. >>>>>> us-west-2.compute.amazonaws.com:6667 >>>>>> ssl.provider = null >>>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] >>>>>> log.roll.ms = null >>>>>> log.flush.scheduler.interval.ms = 9223372036854775807 >>>>>> ssl.cipher.suites = null >>>>>> log.index.size.max.bytes = 10485760 >>>>>> ssl.keymanager.algorithm = SunX509 >>>>>> security.inter.broker.protocol = PLAINTEXT >>>>>> replica.fetch.max.bytes = 1048576 >>>>>> advertised.port = null >>>>>> log.cleaner.dedupe.buffer.size = 134217728 >>>>>> replica.high.watermark.checkpoint.interval.ms = 5000 >>>>>> log.cleaner.io.buffer.size = 524288 >>>>>> sasl.kerberos.ticket.renew.window.factor = 0.8 >>>>>> zookeeper.connection.timeout.ms = 25000 >>>>>> controlled.shutdown.retry.backoff.ms = 5000 >>>>>> log.roll.hours = 168 >>>>>> log.cleanup.policy = delete >>>>>> host.name = >>>>>> log.roll.jitter.ms = null >>>>>> max.connections.per.ip = 2147483647 >>>>>> offsets.topic.segment.bytes = 104857600 >>>>>> background.threads = 10 >>>>>> quota.consumer.default = 9223372036854775807 >>>>>> request.timeout.ms = 30000 >>>>>> log.message.format.version = 0.10.0-IV1 >>>>>> log.index.interval.bytes = 4096 >>>>>> log.dir = /tmp/kafka-logs >>>>>> log.segment.bytes = 1073741824 >>>>>> log.cleaner.backoff.ms = 15000 >>>>>> offset.metadata.max.bytes = 4096 >>>>>> ssl.truststore.location = null >>>>>> group.max.session.timeout.ms = 300000 >>>>>> ssl.keystore.password = null >>>>>> zookeeper.sync.time.ms = 2000 >>>>>> port = 6667 >>>>>> log.retention.minutes = null >>>>>> log.segment.delete.delay.ms = 60000 >>>>>> log.dirs = /data2/kafka-log >>>>>> controlled.shutdown.enable = true >>>>>> compression.type = producer >>>>>> max.connections.per.ip.overrides = >>>>>> log.message.timestamp.difference.max.ms = 9223372036854775807 >>>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit >>>>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 >>>>>> auto.leader.rebalance.enable = true >>>>>> leader.imbalance.check.interval.seconds = 300 >>>>>> log.cleaner.min.cleanable.ratio = 0.5 >>>>>> replica.lag.time.max.ms = 10000 >>>>>> num.network.threads = 3 >>>>>> ssl.key.password = null >>>>>> reserved.broker.max.id = 1000 >>>>>> metrics.num.samples = 2 >>>>>> socket.send.buffer.bytes = 102400 >>>>>> ssl.protocol = TLS >>>>>> socket.receive.buffer.bytes = 102400 >>>>>> ssl.keystore.location = null >>>>>> replica.fetch.min.bytes = 1 >>>>>> broker.rack = null >>>>>> unclean.leader.election.enable = true >>>>>> sasl.enabled.mechanisms = [GSSAPI] >>>>>> group.min.session.timeout.ms = 6000 >>>>>> log.cleaner.io.buffer.load.factor = 0.9 >>>>>> offsets.retention.check.interval.ms = 600000 >>>>>> producer.purgatory.purge.interval.requests = 10000 >>>>>> metrics.sample.window.ms = 30000 >>>>>> broker.id = -1 >>>>>> offsets.topic.compression.codec = 0 >>>>>> log.retention.check.interval.ms = 300000 >>>>>> advertised.listeners = null >>>>>> leader.imbalance.per.broker.percentage = 10 >>>>>> (kafka.server.KafkaConfig) >>>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer) >>>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on >>>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181 >>>>>> (kafka.server.KafkaServer) >>>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer >>>>>> startup. >>>>>> Prepare to shutdown (kafka.server.KafkaServer) >>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect >>>>>> to >>>>>> zookeeper server within timeout: 25000 >>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>>>> scala:75) >>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>>>> at kafka.server.KafkaServerStarta >>>>>> ble.startup(KafkaServerStartab >>>>>> le.scala:37) >>>>>> at kafka.Kafka$.main(Kafka.scala:67) >>>>>> at kafka.Kafka.main(Kafka.scala) >>>>>> [2017-05-17 17:03:40,740] INFO shutting down >>>>>> (kafka.server.KafkaServer) >>>>>> [2017-05-17 17:03:40,747] INFO shut down completed >>>>>> (kafka.server.KafkaServer) >>>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during >>>>>> KafkaServerStartable >>>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable) >>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect >>>>>> to >>>>>> zookeeper server within timeout: 25000 >>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>>>> scala:75) >>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>>>> at kafka.server.KafkaServerStarta >>>>>> ble.startup(KafkaServerStartab >>>>>> le.scala:37) >>>>>> at kafka.Kafka$.main(Kafka.scala:67) >>>>>> at kafka.Kafka.main(Kafka.scala) >>>>>> [2017-05-17 17:03:40,764] INFO shutting down >>>>>> (kafka.server.KafkaServer) >>>>>> >>>>>> Manually starting Kafka via Ambari fixes this... However, Metron REST >>>>>> still doesn't work: >>>>>> >>>>>> . ____ _ __ _ _ >>>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ >>>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ >>>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) ) >>>>>> ' |____| .__|_| |_|_| |_\__, | / / / / >>>>>> =========|_|==============|___/=/_/_/_/ >>>>>> :: Spring Boot :: (v1.4.1.RELEASE) >>>>>> >>>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri >>>>>> ngApplication] >>>>>> - Application startup failed >>>>>> org.springframework.beans.factory.UnsatisfiedDependencyException: >>>>>> Error >>>>>> creating bean with name 'kafkaServiceImpl' defined in URL >>>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT- >>>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic >>>>>> eImpl.class]: >>>>>> Unsatisfied dependency expressed through constructor parameter 0; >>>>>> nested >>>>>> exception is org.springframework.beans.fact >>>>>> ory.UnsatisfiedDependencyException: >>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>>>> expressed through field 'zkClient'; nested exception is >>>>>> org.springframework.beans.factory.BeanCreationException: Error >>>>>> creating >>>>>> bean with name 'zkClient' defined in class path resource >>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>>> instantiation >>>>>> via factory method failed; nested exception is >>>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>>> ZkTimeoutException: >>>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.createArgumentArray(ConstructorResolver.java:749) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.autowireConstructor(ConstructorResolver.java:189) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.autowireConstructor(AbstractAu >>>>>> towireCapableBeanFactory.java:1148) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>>> owireCapableBeanFactory.java:1051) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:510) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:482) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:197) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultListableBea >>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory >>>>>> .java:751) >>>>>> at org.springframework.context.su >>>>>> pport.AbstractApplicationConte >>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.jav >>>>>> a:861) >>>>>> at org.springframework.context.su >>>>>> pport.AbstractApplicationConte >>>>>> xt.refresh(AbstractApplicationContext.java:541) >>>>>> at org.springframework.boot.conte >>>>>> xt.embedded.EmbeddedWebApplica >>>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122) >>>>>> at org.springframework.boot.Sprin >>>>>> gApplication.refresh(SpringApp >>>>>> lication.java:761) >>>>>> at org.springframework.boot.Sprin >>>>>> gApplication.refreshContext(Sp >>>>>> ringApplication.java:371) >>>>>> at org.springframework.boot.Sprin >>>>>> gApplication.run(SpringApplica >>>>>> tion.java:315) >>>>>> at org.springframework.boot.Sprin >>>>>> gApplication.run(SpringApplica >>>>>> tion.java:1186) >>>>>> at org.springframework.boot.Sprin >>>>>> gApplication.run(SpringApplica >>>>>> tion.java:1175) >>>>>> at org.apache.metron.rest.MetronR >>>>>> estApplication.main(MetronRest >>>>>> Application.java:27) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>> Method) >>>>>> at sun.reflect.NativeMethodAccess >>>>>> orImpl.invoke(NativeMethodAcce >>>>>> ssorImpl.java:62) >>>>>> at sun.reflect.DelegatingMethodAc >>>>>> cessorImpl.invoke(DelegatingMe >>>>>> thodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at org.springframework.boot.loade >>>>>> r.MainMethodRunner.run(MainMet >>>>>> hodRunner.java:48) >>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>>>> java:87) >>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>>>> java:50) >>>>>> at org.springframework.boot.loader.PropertiesLauncher.main( >>>>>> PropertiesLauncher.java:525) >>>>>> Caused by: org.springframework.beans.fact >>>>>> ory.UnsatisfiedDependencyExcep >>>>>> tion: >>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>>>> expressed through field 'zkClient'; nested exception is >>>>>> org.springframework.beans.factory.BeanCreationException: Error >>>>>> creating >>>>>> bean with name 'zkClient' defined in class path resource >>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>>> instantiation >>>>>> via factory method failed; nested exception is >>>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>>> ZkTimeoutException: >>>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>>> at org.springframework.beans.fact >>>>>> ory.annotation.AutowiredAnnota >>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>>>> utowiredAnnotationBeanPostProcessor.java:569) >>>>>> at org.springframework.beans.fact >>>>>> ory.annotation.InjectionMetada >>>>>> ta.inject(InjectionMetadata.java:88) >>>>>> at org.springframework.beans.fact >>>>>> ory.annotation.AutowiredAnnota >>>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn >>>>>> otationBeanPostProcessor.java:349) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:1219) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:543) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:482) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:197) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>>>> ireCapableBeanFactory.java:1128) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>>> owireCapableBeanFactory.java:1023) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:510) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:482) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:202) >>>>>> at org.springframework.beans.fact >>>>>> ory.config.DependencyDescripto >>>>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultListableBea >>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultListableBea >>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.createArgumentArray(ConstructorResolver.java:741) >>>>>> ... 27 more >>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: >>>>>> Error >>>>>> creating bean with name 'zkClient' defined in class path resource >>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>>> instantiation >>>>>> via factory method failed; nested exception is >>>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>>> ZkTimeoutException: >>>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>>>> ireCapableBeanFactory.java:1128) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>>> owireCapableBeanFactory.java:1023) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:510) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:482) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:202) >>>>>> at org.springframework.beans.fact >>>>>> ory.config.DependencyDescripto >>>>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultListableBea >>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.DefaultListableBea >>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>>>> at org.springframework.beans.fact >>>>>> ory.annotation.AutowiredAnnota >>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>>>> utowiredAnnotationBeanPostProcessor.java:566) >>>>>> ... 50 more >>>>>> Caused by: org.springframework.beans.BeanInstantiationException: >>>>>> Failed >>>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method >>>>>> 'zkClient' >>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>>> ZkTimeoutException: >>>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>>> at org.springframework.beans.fact >>>>>> ory.support.SimpleInstantiatio >>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.ConstructorResolve >>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588) >>>>>> ... 62 more >>>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable >>>>>> to >>>>>> connect to zookeeper server within timeout: 10000 >>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97) >>>>>> at org.apache.metron.rest.config. >>>>>> ZookeeperConfig.zkClient(Zooke >>>>>> eperConfig.java:52) >>>>>> at org.apache.metron.rest.config. >>>>>> ZookeeperConfig$$EnhancerBySpr >>>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>) >>>>>> at org.apache.metron.rest.config. >>>>>> ZookeeperConfig$$EnhancerBySpr >>>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke( >>>>>> <generated>) >>>>>> at org.springframework.cglib.prox >>>>>> y.MethodProxy.invokeSuper(Meth >>>>>> odProxy.java:228) >>>>>> at org.springframework.context.an >>>>>> notation.ConfigurationClassEnh >>>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha >>>>>> ncer.java:356) >>>>>> at org.apache.metron.rest.config. >>>>>> ZookeeperConfig$$EnhancerBySpr >>>>>> ingCGLIB$$6e0c6742.zkClient(<generated>) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>> Method) >>>>>> at sun.reflect.NativeMethodAccess >>>>>> orImpl.invoke(NativeMethodAcce >>>>>> ssorImpl.java:62) >>>>>> at sun.reflect.DelegatingMethodAc >>>>>> cessorImpl.invoke(DelegatingMe >>>>>> thodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at org.springframework.beans.fact >>>>>> ory.support.SimpleInstantiatio >>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162) >>>>>> ... 63 more >>>>>> >>>>>> Any ideas on what might be going or why the timeout triggers? >>>>>> >>>>>> >>>>>> On 2017-05-17 08:31, Laurens Vets wrote: >>>>>> >>>>>> Hello list, >>>>>> >>>>>>> >>>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the >>>>>>> following changes in playbook.yml so that I'm left with 5 EC2 >>>>>>> instances instead of 10: >>>>>>> >>>>>>> <-- SNIP --> >>>>>>> tasks: >>>>>>> - include: tasks/create-keypair.yml >>>>>>> - include: tasks/create-vpc.yml >>>>>>> - include: tasks/create-open-inbound-security-group.yml >>>>>>> - include: tasks/create-open-outbound-security-group.yml >>>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>>> host_type=sensors,ambari_master,ec2,monit >>>>>>> # - include: tasks/create-hosts.yml host_count=4 >>>>>>> host_type=ambari_slave,ec2 >>>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>>> host_type=pcap_server,monit,ec2 >>>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin >>>>>>> # - include: tasks/create-hosts.yml host_count=2 >>>>>>> host_type=ambari_slave,search,ec2 >>>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>>> host_type=ambari_slave,search,ec2 >>>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>>> host_type=ambari_slave,web,ec2 >>>>>>> <-- END SNIP --> >>>>>>> >>>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py >>>>>>> playbook.yml \ >>>>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>>>> --extra-vars="env=$ENV" \ >>>>>>> $EXTRA_ARGS" >>>>>>> >>>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \ >>>>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>>>> --extra-vars="env=$ENV" \ >>>>>>> $EXTRA_ARGS -c paramiko" >>>>>>> in run.sh otherwise ssh would time out. >>>>>>> >>>>>>> However, Ansible errors out with: >>>>>>> >>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config : >>>>>>> Deploy cluster with Ambari; >>>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] *** >>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path: >>>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>>>> ambari_config/tasks/main.yml:36 >>>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal: >>>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! => >>>>>>> {"changed": false, "failed": true, "invocation": {"module_args": >>>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups": >>>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name": >>>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name": >>>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, >>>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}], >>>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1, >>>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"}, >>>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name": >>>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name": >>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": >>>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [], >>>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name": >>>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name": >>>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"}, >>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name": >>>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], >>>>>>> "name": >>>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"}, >>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>>>> "name": "slaves"}], "required_configurations": [{"metron-env": >>>>>>> {"es_hosts": >>>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20 >>>>>>> 8-94-45.us-west-2.compute.amazonaws.com", >>>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": >>>>>>> "root", >>>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url": >>>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root", >>>>>>> "storm_rest_addr": >>>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744", >>>>>>> "zeppelin_server_url": >>>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, >>>>>>> {"kibana-env": >>>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard", >>>>>>> "kibana_es_url": >>>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200";, >>>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir": >>>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": >>>>>>> "HDP", >>>>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state": >>>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir": >>>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024, >>>>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize": >>>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site": >>>>>>> {"dfs.datanode.data.dir": >>>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data", >>>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode", >>>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary", >>>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}}, >>>>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path": >>>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229, >>>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site": >>>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local", >>>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log", >>>>>>> "yarn.nodemanager.resource.memory-mb": 4096, >>>>>>> "yarn.timeline-service.leveldb-state-store.path": >>>>>>> "/data1/hadoop/yarn/timeline", >>>>>>> "yarn.timeline-service.leveldb-timeline-store.path": >>>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir": >>>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702, >>>>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}}, >>>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env": >>>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site": >>>>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": >>>>>>> 1, >>>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]", >>>>>>> "zen_discovery_ping_unicast_hosts": "[ >>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com, >>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host": >>>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password": >>>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete": >>>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client >>>>>>> exception occurred: Could not create cluster: request code 400, >>>>>>> request message {\n \"status\" : 400,\n \"message\" : >>>>>>> \"Invalid Cluster Creation Template: >>>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException: >>>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a >>>>>>> 'host_count' value\"\n}"} >>>>>>> >>>>>>> PLAY RECAP ****************************** >>>>>>> ****************************** >>>>>>> ********* >>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 >>>>>>> changed=27 >>>>>>> unreachable=0 failed=0 >>>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 >>>>>>> changed=8 >>>>>>> unreachable=0 failed=0 >>>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 >>>>>>> changed=31 >>>>>>> unreachable=0 failed=0 >>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 >>>>>>> changed=27 >>>>>>> unreachable=0 failed=0 >>>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 >>>>>>> changed=28 >>>>>>> unreachable=0 failed=1 >>>>>>> localhost : ok=29 changed=12 unreachable=0 >>>>>>> failed=0 >>>>>>> >>>>>>> Any idea what might be going on? >>>>>>> >>>>>>> (Don't worry about the URLs, the instances have been terminated) >>>>>>> >>>>>>> >>>>>>> >>>>>>
