Re: [jclouds/jclouds-labs] JCLOUDS-1386 1&1 Baremetal servers (#431)
@andreaturli I have addressed the issues you mentioned, could you now please take a look? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/431#issuecomment-414971479
[jira] [Created] (JCLOUDS-1444) Outage on any endpoint in catalog blocks usage of other endpoints
Gilles Coremans created JCLOUDS-1444: Summary: Outage on any endpoint in catalog blocks usage of other endpoints Key: JCLOUDS-1444 URL: https://issues.apache.org/jira/browse/JCLOUDS-1444 Project: jclouds Issue Type: Bug Components: jclouds-compute Affects Versions: 2.1.1 Environment: OVH OpenStack Reporter: Gilles Coremans I was trying to create a template using the ComputeService backed by the nova API (I was connecting to OVH, which does not have a jClouds provider). However, one of their regions was down for maintenance. This caused the following error: {code:java} SEVERE: Cannot retry after server error, command has exceeded retry limit 5: [method=org.jclouds.openstack.nova.v2_0.features.ImageApi.public abstract org.jclouds.collect.PagedIterable org.jclouds.openstack.nova.v2_0.features.ImageApi.listInDetail()[], request=GET https://compute.bhs3.cloud.ovh.net/v2/[tenant]/images/detail HTTP/1.1{code} I feel that jClouds shold handle this case gracefully and simply skip the offending API, rather than failing with NullPointerExceptions and making it impossible to create templates as long as any endpoint is down for whatever reason. This is the reply one gets when querying the given URL directly: {code:java} $ curl https://compute.bhs3.cloud.ovh.net/v2/[tenant]/images/detail curl: (52) Empty reply from server {code} I verified on OVH's site that this region was indeed down for maintenance: [http://travaux.ovh.net/?do=details&id=32795&PHPSESSID=b0747f889c42d0b97fedeb393cc16b0e|http://travaux.ovh.net/?do=details&id=32795&PHPSESSID=b0747f889c42d0b97fedeb393cc16b0e)] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > + + final Image image = imageInRegionToImage.apply(ImageInRegion.create(Regions.EU_CENTRAL_1.getName(), ecsImage)); + assertEquals(ecsImage.id(), image.getProviderId()); + assertEquals(ecsImage.name(), image.getName()); + assertEquals(Image.Status.AVAILABLE, image.getStatus()); + final org.jclouds.compute.domain.OperatingSystem operatingSystem = image.getOperatingSystem(); + + assertEquals(ecsImage.osName(), operatingSystem.getName()); + assertEquals(ecsImage.description(), operatingSystem.getDescription()); + assertTrue(operatingSystem.is64Bit()); + assertEquals(region, image.getLocation()); + } + + Date parseDate(final String dateString) { + return DatatypeConverter.parseDateTime(dateString).getTime(); + } done -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#discussion_r211980875
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.jclouds.aliyun.ecs.domain; + +import com.google.common.base.Enums; +import com.google.common.base.Joiner; +import com.google.common.base.Optional; + +import static com.google.common.base.Preconditions.checkArgument; + +/** + * The type of the ECS resource. All values must be lowercase. done -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#discussion_r211981107
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > + + public void testList() { + final AtomicInteger found = new AtomicInteger(0); + assertTrue(Iterables.all(api().list(Regions.EU_CENTRAL_1.getName()).concat(), new Predicate() { + @Override + public boolean apply(VPC input) { +found.incrementAndGet(); +return !isNullOrEmpty(input.id()); + } + }), "All vpcs must have at least the 'id' field populated"); + assertTrue(found.get() > 0, "Expected some vpc to be returned"); + } + + private VPCApi api() { + return api.vpcApi(); + } added missing -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#discussion_r211981335
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > + + public void testList() { + final AtomicInteger found = new AtomicInteger(0); + assertTrue(Iterables.all(api().list(Regions.EU_CENTRAL_1.getName()).concat(), new Predicate() { + @Override + public boolean apply(VSwitch input) { +found.incrementAndGet(); +return !isNullOrEmpty(input.id()); + } + }), "All vSwitches must have at least the 'id' field populated"); + assertTrue(found.get() > 0, "Expected some vSwitch to be returned"); + } + + private VSwitchApi api() { + return api.vSwitchApi(); + } added missing -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#discussion_r211981309
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > + this.instanceSuspendedPredicate = instanceSuspendedPredicate; + this.regionIds = regionIds; + this.cleanupResources = cleanupResources; + } + + @Override + public NodeAndInitialCredentials createNodeWithGroupEncodedIntoName(String group, String name, Template template) { + String instanceType = template.getHardware().getId(); + String regionId = template.getLocation().getId(); + String imageId = template.getImage().getId(); + + ECSServiceTemplateOptions templateOptions = template.getOptions().as(ECSServiceTemplateOptions.class); + + String keyPairName = templateOptions.getKeyPairName(); + String securityGroupId = Iterables.getOnlyElement(templateOptions.getGroups()); + String vSwitchId = templateOptions.getVSwitchId(); I think I've added the necessary code to support this design: vpcApi vSwitchApi and I've modified the `CreateResourcesThenCreateNodes` accordingly -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#discussion_r211981725
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
andreaturli commented on this pull request. > +import org.jclouds.http.HttpRetryHandler; +import org.jclouds.http.HttpUtils; +import org.jclouds.http.annotation.ClientError; +import org.jclouds.http.handlers.BackoffLimitedRetryHandler; +import org.jclouds.json.Json; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.util.Set; + +import static org.jclouds.http.HttpUtils.closeClientButKeepContentStream; + +/** + * Handles Retryable responses with error codes in the 4xx range + */ +public class ECSErrorRetryHandler implements HttpRetryHandler { @nacx I think this implementation reflects better your suggestion, but please advice if it is not what you meant! thanks -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#pullrequestreview-148517914
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
sorry again as this PR is becoming huge. @nacx I think I've addressed all of your comments now and I've also simplified the RetryHandler for 4xx errors I'm running the live tests to see the result, some of them are expected to fail though - particularly from `ECSComputeServiceLiveTest`. I'll keep you posted Thanks again for your help -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#issuecomment-415059572
Re: [jclouds/jclouds-labs] [JCLOUDS-1430] Aliyun ECS (#443)
here's the live tests result ``` Results : Failed tests: SshKeyPairApiLiveTest.testImport:58 » IllegalArgument {"RequestId":"4084B85E-D... ECSComputeServiceLiveTest.testCorrectAuthException » Test Method BaseComputeS... ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testCreateAndRunAService:747 » NoSuchElement ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testGet:553 » NoSuchElement ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testImageById:211 expected [{id=eu-central-1/centos_7_04_64_20G_alibase_20180419.vhd, providerId=centos_7_04_64_20G_alibase_20180419.vhd, name=centos_7_04_64_20G_alibase_20180419.vhd, location={scope=REGION, id=eu-central-1, description=欧洲中部 1 (法兰克福), parent=alibaba-ecs}, os={family=centos, name=CentOS 7.4 64位, version=7.4, description=, is64Bit=true}, description=, status=AVAILABLE, loginUser=root}] but found [null] ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testListSizes:888->BaseComputeServiceLiveTest.checkVolumes:893 {id=ecs.sn1.medium, providerId=ecs.sn1.medium, name=ecs.sn1.medium, processors=[{cores=2.0, speed=2.0}], ram=4096, hypervisor=none, supportsImage=Predicates.alwaysTrue()} ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testOptionToNotBlock:866 » RunNodes ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testWeCanCancelTasks:282 » RunNodes ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testAScriptExecutionAfterBootWithBasicTemplate:227 » RunNodes ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testConcurrentUseOfComputeServiceToCreateNodes:498 » Execution ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testCreateTwoNodesWithOneSpecifiedName:383 » RunNodes ECSComputeServiceLiveTest>BaseComputeServiceLiveTest.testCreateAnotherNodeWithANewContextToEnsureSharedMemIsntRequired:435 » RunNodes Tests run: 55, Failures: 12, Errors: 0, Skipped: 7 ``` `SshKeyPairApiLiveTest` is a quite weird as by the doc it should work I'll concentrate on `ECSComputeServiceLiveTest` failures of `{testGet, testImageById, testListSizes}` as they should be fine. `testCreateAndRunAService` fails because of jetty on centos7 issue -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/443#issuecomment-415086300
Re: [jclouds/jclouds] JCLOUDS-1339: Support launching an x1 EC2 instance (#1235)
andreaturli commented on this pull request. > @@ -290,6 +290,19 @@ private EC2HardwareBuilder hs1() { virtualizationTypes(VirtualizationType.HVM, VirtualizationType.PARAVIRTUAL); return this; } + + private EC2HardwareBuilder x1() { + virtualizationTypes(VirtualizationType.HVM); + + // TODO X1 is not deprecated, but it requires that you are using a VPC + // until we have a way for hardware instances to be filtered based on network + // we do NOT want X1 selected automatically. + // You get: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='2300b99e-ddc0-42ab-b1ed-9d628a161be4', requestToken='null', code='VPCResourceNotSpecified', message='The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.', context='{Response=, Errors=}'} + // A user can explicitly request a x1 if they are also setting up a VPC. + deprecated(); [minor] do you think it is still a valid comment? Look quite unusual to not deploy on a VPC these days, is it still important to filter those instance types -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1235#pullrequestreview-148559738
Re: [jclouds/jclouds] JCLOUDS-1339: Support launching an x1 EC2 instance (#1235)
thanks @lemmy for your contribution! only a minor comment from my side otherwise looks sensible. This manual process is really tedious, thanks for helping us out! I wish AWS will offer a simple instanceTypes API at some point! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1235#issuecomment-415090459
Re: [jclouds/jclouds] [JCLOUDS-1443] - Fix resolution uri (#1234)
@gaul do you think it is good to be merged now? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1234#issuecomment-415090751
Re: [jclouds/jclouds] Improve Jetty installation for live tests (#1230)
rebuild please -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1230#issuecomment-415091039
Re: [jclouds/jclouds] JCLOUDS-1339: Support launching an x1 EC2 instance (#1235)
lemmy commented on this pull request. > @@ -290,6 +290,19 @@ private EC2HardwareBuilder hs1() { virtualizationTypes(VirtualizationType.HVM, VirtualizationType.PARAVIRTUAL); return this; } + + private EC2HardwareBuilder x1() { + virtualizationTypes(VirtualizationType.HVM); + + // TODO X1 is not deprecated, but it requires that you are using a VPC + // until we have a way for hardware instances to be filtered based on network + // we do NOT want X1 selected automatically. + // You get: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='2300b99e-ddc0-42ab-b1ed-9d628a161be4', requestToken='null', code='VPCResourceNotSpecified', message='The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.', context='{Response=, Errors=}'} + // A user can explicitly request a x1 if they are also setting up a VPC. + deprecated(); @andreaturli I just copied this from https://github.com/jclouds/jclouds/blob/cf67233765855a850e40f4b3d3b3ba320678dbbd/apis/ec2/src/main/java/org/jclouds/ec2/compute/domain/EC2HardwareBuilder.java#L218-L230 assuming the same reasoning applies to X1 instances. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1235#discussion_r212018738
Re: [jclouds/jclouds] [JCLOUDS-1374]This is the first commit of Softlayer LoadBalancer API (#1176)
hi @swaqos any update on that? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1176#issuecomment-415091546
Re: [jclouds/jclouds-labs] JCLOUDS-1386 1&1 Baremetal servers (#431)
andreaturli commented on this pull request. > @Inject OneandoneComputeServiceAdapter(OneAndOneApi api, CleanupResources cleanupResources, + GenerateHardwareRequest generateHardwareRequest, @Named(POLL_PREDICATE_SERVER) Predicate waitServerUntilAvailable, PasswordGenerator.Config passwordGenerator) { `PasswordGenerator.Config passwordGenerator` is not used, remove it -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/431#discussion_r212021489
Re: [jclouds/jclouds-labs] JCLOUDS-1386 1&1 Baremetal servers (#431)
andreaturli requested changes on this pull request. @alibazlamit I've added some more comments but I could not finish up the review as the code is becoming too complicated to be reviewed. As I told you other times it would be much easier to add small chunks of code in small PRs. For example you could focus on a PR that adds `RecoveryImageApi` with Mock and Live Tests only (+ domain objects needed) Then we can focus on the baremetal feature itself as it requires a huge number of changes! Does it sound like a plan? Thanks again > @@ -253,7 +253,7 @@ public static ApplianceType fromValue(String v) { } public enum OSImageType { - Standard, Minimal, Personal, ISO_OS, ISO_TOOL, NULL, UNRECOGNIZED; + STANDARD, MINIMAL, Personal, ISO_OS, ISO_TOOL, NULL, UNRECOGNIZED; please make it all capital > @@ -283,4 +283,22 @@ public String toString() { return value; } } + + public enum ServerType { + cloud, baremetal, UNRECOGNIZED; + + public static ServerType fromValue(String v) { + return Enums.getIfPresent(ServerType.class, v).or(UNRECOGNIZED); + } + + } + + public enum ServerTypeCompatibility { + vps, cloud, baremetal, UNRECOGNIZED; upper case? > @@ -283,4 +283,22 @@ public String toString() { return value; } } + + public enum ServerType { + cloud, baremetal, UNRECOGNIZED; why not upper case? > @@ -179,9 +179,46 @@ return new NodeAndInitialCredentials(updateServer, updateServer.id(), serverCredentials); } + private ServerAppliance findWorkingImage(String imageId, Types.ServerTypeCompatibility serverType) { why this name? it seems that it tries to find a `ServerAppliance`, doesn't it? > -//check if the bootable device has enough size to run the > appliance(image). -float minHddSize = volume.getSize(); -if (volume.isBootDevice()) { - SingleServerAppliance appliance = api.serverApplianceApi().get(image.getId()); - if (appliance.minHddSize() > volume.getSize()) { - minHddSize = appliance.minHddSize(); - } -} -Hdd.CreateHdd hdd = Hdd.CreateHdd.create(minHddSize, volume.isBootDevice()); -hdds.add(hdd); - } catch (Exception ex) { -throw Throwables.propagate(ex); - + String imageId = image.getId(); + Hardware hardwareModel = generateHardwareRequest.isFlavor(hardware.getId()); + boolean isBaremetal = hardware.getName() == null ? false : hardware.getName().contains(baremetalModelsKey); wouldn't be better to introduce a custom `TemplateOptions` where you can pass in `bareMetal` boolean parameter? see https://github.com/jclouds/jclouds/blob/master/providers/digitalocean2/src/main/java/org/jclouds/digitalocean2/compute/options/DigitalOcean2TemplateOptions.java#L38 for an example > - SingleServerAppliance appliance = > api.serverApplianceApi().get(image.getId()); - if (appliance.minHddSize() > volume.getSize()) { - minHddSize = appliance.minHddSize(); - } -} -Hdd.CreateHdd hdd = Hdd.CreateHdd.create(minHddSize, volume.isBootDevice()); -hdds.add(hdd); - } catch (Exception ex) { -throw Throwables.propagate(ex); - + String imageId = image.getId(); + Hardware hardwareModel = generateHardwareRequest.isFlavor(hardware.getId()); + boolean isBaremetal = hardware.getName() == null ? false : hardware.getName().contains(baremetalModelsKey); + ServerAppliance workingImage; + + //choose the correct image based on the server type baremetal or cloud move this business logic into its own method > } } - // provision server - Server server = null; - Double cores = ComputeServiceUtils.getCores(hardware); - Double ram = (double) hardware.getRam(); - if (ram < 1024) { - ram = 0.5; - } else { - ram = ram / 1024; + //configuring Firewall rules + Map portsRange = getPortRangesFromList(inboundPorts); all the reasonings for the firewalls / inbound ports shouldn't probably happen in the adapter: firewalls are created once and attached to all the nodes of a group. See `CreateResourcesThenCreateNodes` as a reference at https://github.com/jclouds/jclouds/blob/master/providers/azurecompute-arm/src/main/java/org/jclouds/azurecompute/arm/compute/strategy/CreateResourcesThenCreateNodes.java#L138 > - ram = ram / 1024; + //configuring Firewall rules + Map portsRange = getPortRangesFromList(inboundPorts); + List rules = new ArrayList(); + + for (Map.Entry range : portsRange.entrySet()) { + FirewallPolicy.Rule.CreatePayload rule = FirewallPolicy.Rule.CreatePayload.builder() + .portFrom(range.getKey()) + .portTo(range.getValue()) + .protocol(Types.RuleProtocol
Re: [jclouds/jclouds] [JCLOUDS-1443] - Fix resolution uri (#1234)
I think so, let's get this merged and backported -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1234#issuecomment-415130329
Re: [jclouds/jclouds] JCLOUDS-1339: Support launching an x1 EC2 instance (#1235)
andreaturli commented on this pull request. > @@ -290,6 +290,19 @@ private EC2HardwareBuilder hs1() { virtualizationTypes(VirtualizationType.HVM, VirtualizationType.PARAVIRTUAL); return this; } + + private EC2HardwareBuilder x1() { + virtualizationTypes(VirtualizationType.HVM); + + // TODO X1 is not deprecated, but it requires that you are using a VPC + // until we have a way for hardware instances to be filtered based on network + // we do NOT want X1 selected automatically. + // You get: org.jclouds.aws.AWSResponseException: request POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='2300b99e-ddc0-42ab-b1ed-9d628a161be4', requestToken='null', code='VPCResourceNotSpecified', message='The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.', context='{Response=, Errors=}'} + // A user can explicitly request a x1 if they are also setting up a VPC. + deprecated(); Thanks @lemmy @nacx any thoughts on this? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1235#discussion_r212064403
Re: [jclouds/jclouds] [JCLOUDS-1443] - Fix resolution uri (#1234)
gaul approved this pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1234#pullrequestreview-148643508