This one should do the trick for you.
Due to how smal_cluster.yml is structured (designed for 10+ nodes)- you'll
only get 2 data nodes, so you'll have to reduce replication to 2 if you
don't want to see under-replicated block warnings.
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ambari_slave,web,ec2
The reason the first one didn't work is because order matters. In the that
example, the host created by
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
was ending up in 2 separate hostgroups, which is not allowed. Relocating
that line solves the problem. I also relocated the web hosts (ES Master and
Kibana) since there were fewer hosts.
-D...
On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <laur...@daemon.be> wrote:
> I put in what you said previously :)
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,web,ec2
>
> On 2017-05-30 08:53, David Lyle wrote:
>
>> That's actually a different error than you posted before. That error is
>> Ambari telling you that the blueprint that Ansible submitted is invalid.
>> Likely a result of a required hostgroup not being defined. Can you (re?)
>> post what you've put in amazon-ec2/playbook.yml?
>>
>> -D...
>>
>>
>> On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <laur...@daemon.be> wrote:
>>
>> Same problem as all the previous ones...
>>>
>>> TASK [ambari_config : Install python-requests]
>>> *********************************
>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"attempts":
>>> 1, "changed": false, "msg": "", "rc": 0, "results":
>>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is
>>> already
>>> installed"]}
>>>
>>> TASK [ambari_config : check if ambari-server is up on
>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed":
>>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
>>> "state": "started"}
>>>
>>> TASK [ambari_config : Deploy cluster with Ambari;
>>> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
>>> {"changed": false, "failed": true, "msg": "Ambari client exception
>>> occurred: Could not create cluster: request code 400,
>>> request message {\n \"status\" : 400,\n \"message\" : \"Topology
>>> validation failed: org.apache.ambari.server.topol
>>> ogy.InvalidTopologyException:
>>> The following hosts are mapped to multiple host groups: [
>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
>>> names are converted to lowercase, case differences do not matter in
>>> Ambari
>>> deployments.\"\n}"}
>>> to retry, use: --limit @/root/metron/metron-deploymen
>>> t/amazon-ec2/playbook.retry
>>>
>>> PLAY RECAP ************************************************************
>>> *********
>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
>>> unreachable=0 failed=0
>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
>>> unreachable=0 failed=1
>>> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
>>> unreachable=0 failed=0
>>> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> localhost : ok=18 changed=13 unreachable=0
>>> failed=0
>>>
>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>
>>>
>>> On 2017-05-27 04:45, David Lyle wrote:
>>>
>>> For some reason, kibana_hosts is undefined, so the curl command that
>>>> loads
>>>> the Kibana dashboard doesn't contain the URL for Kibana. If you look at
>>>> small_cluster.yml, you'll see that the Kibana should have been installed
>>>> on
>>>> the first host in the "web" hostgroup. Verify that you placed a host in
>>>> that hostgroup (it's the 5th line in Mike's example). If you did, check
>>>> that host on Ambari and verify that Kibana was installed, if not, you'll
>>>> want to dig through the install log to find out why.
>>>>
>>>> Also- now that I've looked at the example carefully, I noticed that the
>>>> Zeppelin hostgroup was duplicated. That's not a problem, because the 2nd
>>>> host will be ignored, maybe you caught it when you created your own.
>>>>
>>>> I would try something like (for a 6 node host cluster):
>>>>
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_
>>>> master,ec2,monit
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=pcap_server,monit,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,
>>>> enrichment,metron,ec2,zeppelin
>>>> - include: tasks/create-hosts.yml host_count=2
>>>> host_type=ambari_slave,search,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,web,ec2
>>>>
>>>> -D...
>>>>
>>>>
>>>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <laur...@daemon.be>
>>>> wrote:
>>>>
>>>> Different error than before, but it still fails:
>>>>
>>>>>
>>>>> TASK [ambari_gather_facts : set_fact] ******************************
>>>>> ************
>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>>>
>>>>> TASK [ambari_gather_facts : debug] ******************************
>>>>> ***************
>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>>>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>>>> kafka_broker_port
>>>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>>>> ompute.amazonaws.com'], kafka_broker_url =
>>>>> ec2-34-210-18-244.us-west-2.co
>>>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>>>> 2.compute.amazonaws.com']"
>>>>> }
>>>>>
>>>>> TASK [load_web_templates : Load ES Templates]
>>>>> **********************************
>>>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
>>>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
>>>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>>>> \"command\": \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>>>> filters\":
>>>>> [{\"service_name\": \"METRON\",\"component_name\":
>>>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>>>>> "failed":
>>>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>>>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>>>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>>>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>>>>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202",
>>>>> "stdout_lines":
>>>>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>>>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/
>>>>> metron/requests/14%5C>
>>>>>
>>>>> ",",
>>>>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>>>>> \"Accepted\"",
>>>>> " }", "}202"], "warnings": []}
>>>>>
>>>>> TASK [load_web_templates : Load Kibana Dashboard]
>>>>> ******************************
>>>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED!
>>>>> =>
>>>>> {"failed": true, "msg": "the field 'args' has an invalid value, which
>>>>> appears to include a variable that is undefined. The error was: list
>>>>> object
>>>>> has no element 0\n\nThe error appears to have been in
>>>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>>>> tasks/main.yml':
>>>>> line 26, column 3, but may\nbe elsewhere in the file depending on the
>>>>> exact
>>>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
>>>>> Kibana Dashboard\n ^ here\n"}
>>>>> to retry, use: --limit @/root/metron/metron-deploymen
>>>>> t/amazon-ec2/playbook.retry
>>>>>
>>>>> PLAY RECAP ******************************
>>>>> ******************************
>>>>> *********
>>>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>>>>> changed=31
>>>>> unreachable=0 failed=1
>>>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>>>>> changed=31
>>>>> unreachable=0 failed=0
>>>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> localhost : ok=16 changed=11 unreachable=0
>>>>> failed=0
>>>>>
>>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>>
>>>>>
>>>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>>>
>>>>> Maybe you could re-purpose the playbook.yml for the 10-node install -
>>>>>
>>>>>> comment out some of the host creation lines and move the host types
>>>>>> where
>>>>>> you want them installed, e.g.
>>>>>>
>>>>>> From this
>>>>>>
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>> - include: tasks/create-hosts.yml host_count=4
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>> - include: tasks/create-hosts.yml host_count=2
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,web,ec2
>>>>>>
>>>>>> to this (6-node)
>>>>>>
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>>>> - include: tasks/create-hosts.yml host_count=3
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>>>>> ambari_slave
>>>>>> ,pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>
>>>>>> Of course, this is just to demonstrate re-locating items, not
>>>>>> necessarily
>>>>>> how you would want to lay them out.
>>>>>>
>>>>>> Best,
>>>>>> Mike Miklavcic
>>>>>>
>>>>>>
>>>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <laur...@daemon.be>
>>>>>> wrote:
>>>>>>
>>>>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>>>>> default
>>>>>>
>>>>>> 10 node cluster install... So far no luck. Whatever I do, it always
>>>>>>> ends
>>>>>>> in
>>>>>>> errors...
>>>>>>>
>>>>>>>
>>>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>>>
>>>>>>> I'm gonna to venture a 'no' simply because I don't know what that is.
>>>>>>>
>>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>>>>>>> trying
>>>>>>>> to
>>>>>>>> do?
>>>>>>>>
>>>>>>>> -D...
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <laur...@daemon.be>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Is AWS single_node_vm still supported?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
