** Also affects: sahara Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1400477
Title: In Juno Cannot Create Spark Cluster From Horizon Status in OpenStack Dashboard (Horizon): New Status in OpenStack Data Processing (Sahara, ex. Savanna): New Bug description: Trying to instantiate a Spark 1.0.0 cluster, using “Data Processing” element under Horizon, and am having the following problems: 1- Security Group: Having problem with security groups, when either defining "Node Group Templates" or instantiating a cluster. In the first case, if I use an existing group, say "default", gui shows an error stating "Error Security group '2' not found". Sahara log file indicates the same thing: 2014-12-03 23:41:24.144 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17 2014-12-03 23:41:24.146 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375 2014-12-03 23:41:24.163 30234 DEBUG urllib3.connectionpool [-] "GET /v2/926c31c887f441f6a4e4b8031b8cc528/os-security-groups HTTP/1.1" 200 682 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415 2014-12-03 23:41:24.165 30234 DEBUG sahara.utils.api [-] Validation Error occurred: error_code=400, error_message=Security group '2' not found, error_name=INVALID_REFERENCE bad_request /usr/local/lib/python2.7/dist-packages/sahara/utils/api.py:245 2014-12-03 23:41:24.165 30234 INFO sahara.cli.sahara_all [-] 10.0.0.86 - - [03/Dec/2014 23:41:24] "POST /v1.1/926c31c887f441f6a4e4b8031b8cc528/node-group-templates HTTP/1.1" 400 221 0.063121 2014-12-03 23:41:24.257 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650 2014-12-03 23:41:24.258 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707 I can, temporarily, avoid the problem by selecting "Auto Security Group" option. This would allow for a node group to be created; however, I do not see any new security group, under Compute -> Access & Security. At any rate, this also fails during cluster instantiation: 2014-12-03 23:55:06.285 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17 2014-12-03 23:55:06.286 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375 2014-12-03 23:55:06.409 30234 DEBUG urllib3.connectionpool [-] "POST /v2/926c31c887f441f6a4e4b8031b8cc528/servers HTTP/1.1" 400 116 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415 2014-12-03 23:55:06.451 30234 ERROR sahara.service.ops [-] Error during operating cluster 'Sprk265' (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400)) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops Traceback (most recent call last): 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 113, in wrapper 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops f(cluster_id, *args, **kwds) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 198, in _provision_cluster 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops INFRA.create_cluster(cluster) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 51, in create_cluster 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops self._create_instances(cluster) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 168, in _create_instances 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops self._run_instance(cluster, node_group, idx, aa_group=aa_group) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 305, in _run_instance 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops **nova_kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 883, in create 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops **boot_kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 546, in _boot 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops return_raw=return_raw, **kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 100, in _create 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops _resp, body = self.api.client.post(url, body=body) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 490, in post 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops return self._cs_request(url, 'POST', **kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 465, in _cs_request 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops resp, body = self._time_request(url, method, **kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in _time_request 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops resp, body = self.request(url, method, **kwargs) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 433, in request 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops raise exceptions.from_response(resp, body, url, method) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops BadRequest: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400) 2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops 2014-12-03 23:55:06.611 30234 INFO sahara.service.direct_engine [-] Cluster 'Sprk265' creation rollback (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400)) 2014-12-03 23:55:06.616 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17 2014-12-03 23:55:06.622 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375 2014-12-03 23:55:07.018 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650 2014-12-03 23:55:07.019 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707 Here is my conf file: [DEFAULT] use_floating_ips=True use_neutron=True [keystone_authtoken] auth_uri = http://keystone1.maas17:5000/v2.0/ identity_uri=http://keystone1.maas17:35357/ admin_user=sahara admin_password=sahara admin_tenant_name=sahara periodic_enable=true plugins=vanilla,hdp,idh,spark,cdh [database] connection=mysql://sahara:sahara@mysql1.maas17/sahara Sahara is also registered with Keystone: # keystone service-list +----------------------------------+----------+-----------------+----------------------------+ | id | name | type | description | +----------------------------------+----------+-----------------+----------------------------+ | 75b2c466c35a44d5bbe7167c1ed38e20 | cinder | volume | Cinder Volume Service | | d5b014c2d96e4d619f9d9b8e646f0f5b | ec2 | ec2 | EC2 Compatibility Layer | | 8180061e79a24627be43485910a9e16a | glance | image | Glance Image Service | | 4dad7b5145c842a4a8fbbab1f158629c | keystone | identity | Keystone Identity Service | | d833594550e14d49be4df4394886c849 | nova | compute | Nova Compute Service | | fe42b5328002433989ffd6b18414aacc | quantum | network | Quantum Networking Service | | fac3f901841c4528b3902ae1b7265b4e | s3 | s3 | S3 Compatible object-store | | 12104e23db1441a28cb42cf8c9437139 | sahara | data_processing | Data processing service | +----------------------------------+----------+-----------------+----------------------------+ See https://ask.openstack.org/en/question/55161/juno-sahara-spark-100-security-group-error/ for workarounds. 2- Spark Login: I am not able to login to the recommended Spark Image, i.e., http://sahara-files.mirantis.com/saha... . Launching this image either by itself for through Sahara/Data Processing, results in invalid user ubuntu: Generation complete. * Stopping Handle applying cloud-config[74G[ OK ] * Starting Hadoop namenode: starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-ubuntu.out open-vm-tools: not starting as this is not a VMware VM landscape-client is not configured, please run landscape-config. * Restoring resolver state... [80G [74G[ OK ] chown: invalid user: 'ubuntu:ubuntu' chown: invalid user: 'ubuntu:ubuntu' rm: cannot remove '/tmp/in_target.d/post-install.d/20-spark': No such file or directory * Stopping System V runlevel compatibility[74G[ OK ] * Starting execute cloud user/final scripts[74G[ OK ] Cloud-init v. 0.7.5 running 'modules:final' at Thu, 04 Dec 2014 04:48:56 +0000. Up 28.44 seconds. 2014-12-04 04:48:56,208 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed ec2: ec2: ############################################################# ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- ec2: -----END SSH HOST KEY FINGERPRINTS----- ec2: ############################################################# -----BEGIN SSH HOST KEY KEYS----- -----END SSH HOST KEY KEYS----- Cloud-init v. 0.7.5 finished at Thu, 04 Dec 2014 04:48:56 +0000. Datasource DataSourceNone. Up 28.55 seconds 2014-12-04 04:48:56,237 - cc_final_message.py[WARNING]: Used fallback datasource Ubuntu 14.04.1 LTS ubuntu ttyS0 ubuntu login: Sahara log file shows repeated failed login attempts: 2014-12-04 05:01:40.099 30327 DEBUG sahara.service.engine [-] Can't login to node sprk265-worker-002 (10.0.200.59), reason error: [Errno 110] Connection timed out _wait_until_accessible /usr/local/lib/python2.7/dist-packages/sahara/service/engine.py:110 2014-12-04 05:01:40.116 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-worker-001] _execute_command took 127.5 seconds to complete _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459 2014-12-04 05:01:40.117 30327 DEBUG sahara.service.engine [-] Can't login to node sprk265-worker-001 (10.0.200.56), reason error: [Errno 110] Connection timed out _wait_until_accessible /usr/local/lib/python2.7/dist-packages/sahara/service/engine.py:110 2014-12-04 05:01:44.644 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-worker-003] Executing "ls .ssh/authorized_keys" _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459 2014-12-04 05:01:45.015 30327 DEBUG sahara.utils.ssh_remote [-] [sprk265-controller-001] Executing "ls .ssh/authorized_keys" _log_command /usr/local/lib/python2.7/dist-packages/sahara/utils/ssh_remote.py:459 2014-12-04 05:01:45.141 30327 DEBUG sahara.openstack.common.periodic_task [-] Running periodic task SaharaPeriodicTasks.update_job_statuses run_periodic_tasks /usr/local/lib/python2.7/dist-packages/sahara/openstack/common/periodic_task.py:193 Note that I am able to launch other ubuntu images using the same key pair To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1400477/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp