Re: move jclouds to the attic?

[cen]

That is quite sad to hear but not entirely unexpected.

I've been using jclouds on and off for the past 5 years, mainly for 
cloud storage and some cloud VM management.


Nowadays for the IaC use case, to me at least, the declarative tools 
such as Terraform make more sense.



I would like to thank the jclouds community for making my life as 
developer easier all these years.



Best regards

On 10/10/2022 15:03, Andrew Gaul wrote:

jclouds development has slowed from 123 commits from 26 contributors in
2018 to just 24 from 6 contributors in 2022.  This is despite growing
downloads over the last 12 months from 50,000 to 80,000 for jclouds-core
alone.  Unfortunately the number of active committers has shrunk and we
will soon lack quorum for future releases.  This means that the project
must move to the Apache attic.

Ideally the community could step up to sustain the project, e.g.,
reviewing pull requests, fixing issues, responding to mailing list
queries, and eventually becoming committers themselves.  Does anyone
have a multi-year interest in jclouds that wants to help out?

If not, I will cut a final 2.6.0 release before retiring the project.

S3 bucket policies

[cen]

Hi,

I am trying to use jClouds 2.0 with minio (s3 clone). Minio does not 
support ACLs but they do support bucket policies.


Does jClouds blob store support bucket policies? I couldn't find any 
mentions in the docs and my source code exploration didn't find anything 
that fits the description. If this exists an example would be most welcome.


Best regards, cen

Re: JClouds TLS SNI support?

[cen]

Sure. Should I open the issue specifically about Apache client and 
another one for default Java client?


The unexplained thing is why the default Java client isn't working with 
SNI by default, even though it should in theory (Java 8 sends SNI 
automatically and by default according to Oracle docs). The closest 
thing I found is this: 
http://stackoverflow.com/questions/30817934/extended-server-name-sni-extension-not-sent-with-jdk1-8-0-but-send-with-jdk1-7 
a possible OpenJDK bug.


A quick dig into JClouds code seems to confirm that 
setHostnameVerifier() is used so this could be the case.



Ignasi Barrera je 15. 09. 2016 ob 12:06 napisal:

Thanks for the feedback and all the details cen!

Would you mind opening an issue in our JIRA so we can track and fix
the Apache driver?



On 15 September 2016 at 11:17, cen <imba...@gmail.com> wrote:

Hi

Default driver and Apache driver failed me but OkHTTP worked.

For Apache, I found a similar bug in Keycloak JIRA:
https://issues.jboss.org/browse/KEYCLOAK-2439

The interesting part is:

"Client adapter uses a deprecated API when setting up HttpClient object in
org.keycloak.adapters.HttpClientBuilder. As a result, a SNI patch which is
part of HttpClient library since version 4.3.2, and which seems to delegate
this part to Java SDK classes, where SNI is automatically set, isn't
activated."

It's a guess on my part but I assume JClouds instantiates the HttpClient in
a way that SNI does not get activated.

I digged more into Apache driver and the way SSLSocketFactory is used by
JClouds is very similar to pre-patched Keycloak from that Jira issue
(according to pull requests). Might be worth looking into.

Best regards, cen


Ignasi Barrera je 12. 09. 2016 ob 21:04 napisal:

Hi!

jclouds supports several HTTP drivers. By default it relies on the java
HttpUrlConection, but you can also configure it to use the Apache Http
client or OkHttp [1]. Using those drivers is as simple as adding the
corresponding Guice module when creating the context (have a look at the
OkHttp driver readme for an example [2]) so feel free to use the one that
is better for your use case.

If you need more control on how the http client is configured, you can take
the jclouds Docker api as an example. It configures the OkHttp to support
TLS connections. You can have a look at its docker http module [3] and
create a similar module that initializes the OkHtttpClient as needed, and
then pass it to the ContextBuilder when creating the jclouds context.

HTH!

I.

[1] https://github.com/jclouds/jclouds/tree/master/drivers
[2] https://github.com/jclouds/jclouds/blob/master/drivers/okhttp/README.md
[3]
https://github.com/jclouds/jclouds/blob/master/apis/docker/src/main/java/org/jclouds/docker/config/DockerHttpApiModule.java

El 12 sept. 2016 7:02 p. m., "cen" <imba...@gmail.com> escribió:

Hi

We have a FakeS3 instance behind a reverse proxy which handles several
subdomains over a single IP. We use let's encrypt certificate to sign the
subdomains. We have the latest Java 8 installed which has the let's encrypt
root in it's truststore. However, JClouds fails to connect to our FakeS3
instance over https (http works). We believe it is because TLS SNI is not
supported in JClouds since this is the most common problem we found other
people having when googling around. I browsed around org.jclouds.http
package but I was unable to determine what HTTP client does JClouds use
behind the scenes or if it's a custom implementation. Could I get some
feedback whether my assumptions are correct and how hard would it be to fix
this? This is the stacktrace:


PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target connecting to
HEAD https://s3.demo.mydomain.com/productname HTTP/1.1
 at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(
BaseHttpCommandExecutorService.java:121)
 at org.jclouds.rest.internal.InvokeHttpMethod.invoke(
InvokeHttpMethod.java:90)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(
InvokeHttpMethod.java:73)
 at org.jclouds.rest.internal.InvokeHttpMethod.apply(
InvokeHttpMethod.java:44)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(
DelegatesToInvocationFunction.java:156)
 at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(
DelegatesToInvocationFunction.java:123)
 at com.sun.proxy.$Proxy146.bucketExists(Unknown Source)
 at org.jclouds.s3.blobstore.S3BlobStore.containerExists(
S3BlobStore.java:131)
 at com.redacted.util.storage.S3Storage.saveBlob(S3Storage.java:42)
 at com.redacted.util.storage.BlobStorageImpl.saveBlob(
BlobStorageImpl.java:19)
 at com.redacted.api.rest.v1.resources.ImagesResourceImpl.createTenant(
ImagesResourceImpl.java:90)
 at com.redacted.api.rest.v1.resources.ImagesResourceImpl$
Proxy$_$$_WeldSubclass.createTenant$$super(Unknown Source)
 at sun.reflect.

Re: Group name not set for new nodes on EC2

[cen]

Since createNodesInGroup() accepts group name as a parameter am I
correct to assume that the current logic with key pairs and security
groups should be ignored in this case and the name provided should be
used as the group name?
Also, if I created security group and/or key together with the node,
would the group name be respected or would Jclouds still try to come up
with it's own name?
As a workaround I will try to manually create metadata tag for group name.

To explain my use case a bit further, I want to group the nodes by
operating system, so I can select nodes like: "Give me all Ubuntu nodes"
or "Give me all FreeBSD nodes" or "Give me all Windows nodes" etc. That
is why I want to provide the group name by myself which is basically
"MyProjectName"+amiId.

On 21. 03. 2016 11:29, Ignasi Barrera wrote:
> Regarding the group names thing, jclouds tries to infer the group name
> from the security groups and key pairs [1]. When key pairs or security
> groups are (indirectly) created as part of the "createNodesInGroup"
> call, jclouds applies the naming convention taking into account the
> group name. The code that tries to infer the groups for existing
> instances seems to rely on that fact, which conflicts with the use
> case when security groups or key pairs already exist or are created
> using the security group extension (which has no "group" parameter).
> 
> This could definitely be done better by applying tags to the created
> instance, or whatever, so mind filing a JIRA issue with the code
> snippet you pasted, so we can properly track and improve this?
> 
> 
> 
> I.
> 
> [1] 
> https://github.com/jclouds/jclouds/blob/master/apis/ec2/src/main/java/org/jclouds/ec2/compute/functions/RunningInstanceToNodeMetadata.java#L186-L194
> [2] https://issues.apache.org/jira/browse/JCLOUDS
> 
> On 20 March 2016 at 13:31, Andrew Phillips <andr...@apache.org> wrote:
>>
>>
>> On 2016-03-20 10:00, cen wrote:
>>>
>>> Agree with both statements. If there was documentation about this on
>>> createSecurityGroup I'd knew instantly what is going on. It could even
>>> be a generic statement that a certain method adheres to
>>> GroupNamingConvention.
>>>
>>> The pattern does not really match my use case because I create a single
>>> security group on first run and then re-use it for each new VM (that's
>>> why I hardcoded the name).
>>
>>
>> Ah, like that - in that case, I would imagine Ignasi's suggestion of either
>> setting the RESOURCENAME_PREFIX property, or going a step further and
>> actually overriding the naming convention "generator", might be the right
>> thing to do?
>>
>> Then you *know* exactly what the resources will look like, and don't have to
>> save that information for reuse later.
>>
>> Regards
>>
>> ap

Re: Group name not set for new nodes on EC2

[cen]

Agree with both statements. If there was documentation about this on
createSecurityGroup I'd knew instantly what is going on. It could even
be a generic statement that a certain method adheres to
GroupNamingConvention.

The pattern does not really match my use case because I create a single
security group on first run and then re-use it for each new VM (that's
why I hardcoded the name). I'd probably have to save the name to
database in this case and I think that's what I'll actually do.

On 19. 03. 2016 17:58, Andrew Phillips wrote:
>>> Regarding the group name in the nodes, could you share the code you
>>> use to build the template, including the template options?
> 
> Thanks for the additional details! More information on the resource
> prefixing Ignasi was mentioning is here, by the way:
> 
> http://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/compute/functions/GroupNamingConvention.html
> 
> 
> Still, it seems we are short on documentation on where exactly this
> prefixing is applied. For example, the documentation for
> createSecurityGroup [1] does not mention this, even though from the code
> [2] and what we've seen in this thread a prefix *is* being added.
> 
> In order to avoid this problem, would it make sense to recommend the
> following pattern?
> 
> SecurityGroup sg = securityGroupExtension.createSecurityGroup("name",
> region);
> 
> // get the name of the actually created item!!
> String sgName = sg.getName();
> ...
> EC2TemplateOptions o = EC2TemplateOptions.Builder.securityGroups(sgName);
> ...
> 
> In other words, instead of assuming that the input parameter to
> createSecurityGroup (or similar calls) can be used as-is, always use the
> values of the resulting object?
> 
> Regards
> 
> ap
> 
> [1]
> http://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/aws/ec2/compute/extensions/AWSEC2SecurityGroupExtension.html#createSecurityGroup(java.lang.String,
> java.lang.String)
> [2]
> https://github.com/jclouds/jclouds/blob/master/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/extensions/AWSEC2SecurityGroupExtension.java#L73
> 

Group name not set for new nodes on EC2

[cen]

Hi

For some reason the new nodes I create on EC2 are missing the group names.

NodeMetadata node =
getOnlyElement(compute.createNodesInGroup(vmSettings.getGroupName(), 1,
template));
logger.info("New node " + node.getId() + " " +
concat(node.getPrivateAddresses(), node.getPublicAddresses())+" in group
"+vmSettings.getGroupName());
...
upload some files and run some script on the node
... 
logger.info("Node in group "+node.getGroup()+" initiallized.");


Output:
New node eu-central-1/i-1ff660a3 [ip redacted, ip redacted] in group
crossbuild-ami-87564feb
Node in group null initiallized.


And indeed the group tag is missing in EC2 console. I hardly believe
such basic functionality would be a bug in Jclouds but I also can't see
what I am doing wrong. Any ideas?

Re: Group name not set for new nodes on EC2

[cen]

Yeah sure. This is my final code to create the security group:

AWSEC2SecurityGroupExtension client =
(AWSEC2SecurityGroupExtension)compute.getSecurityGroupExtension().get();

SecurityGroup sg = client.createSecurityGroup("crossbuild", CLOUD_REGION);
Builder b = IpPermission.builder();
b.fromPort(0);
b.toPort(65535);
b.groupId(sg.getProviderId());
b.ipProtocol(IpProtocol.TCP);
b.cidrBlock(CROSSBUILD_SECURITY_GROUP); //sth like 0.0.0.0/0
client.addIpPermission(b.build(), sg);


And now the catch, when setting the group name, the name is not actually
what you just wanted uit to be because Jclouds adds prefix "jclouds#" to
it. So when using security group in template builder, make sure you add
that prefix:

EC2TemplateOptions o = EC2TemplateOptions.Builder.  
.overrideLoginCredentials(getLoginForCommandExecution(vmSettings)).securityGroups("jclouds#crossbuild");

On 16. 03. 2016 18:18, Andrew Phillips wrote:
>> After changing the group name in template to "jclouds#crossbuild" it
>> started working.
> 
> Ah, glad to hear! Could you share your final code and perhaps a short
> description of "this is what I was trying to do, this approach didn't
> work, this different approach did"? I suspect that may turn into pretty
> useful reference material for others trying to do the same thing in
> future ;-)
> 
> Regards
> 
> ap

Re: Group name not set for new nodes on EC2

[cen]

Thank you.

I have already solved the security group issue. The problem was the
"cosmetic" one I mentioned in the email on the end. I was creating a
group with name "crossbuild" and tried to use the same name when
creating the node. The exception was thrown when creating the node, not
when creating the group. After changing the group name in template to
"jclouds#crossbuild" it started working.


On 16. 03. 2016 18:12, Ignasi Barrera wrote:
> Hi Cen,
> 
> I'll have a look at this and the security group thing later today.
> 
> Apologies for the late reply!
> 
> I.
> 
> Hi
> 
> For some reason the new nodes I create on EC2 are missing the group names.
> 
> NodeMetadata node =
> getOnlyElement(compute.createNodesInGroup(vmSettings.getGroupName(), 1,
> template));
> logger.info <http://logger.info>("New node " + node.getId() + " " +
> concat(node.getPrivateAddresses(), node.getPublicAddresses())+" in group
> "+vmSettings.getGroupName());
> ...
> upload some files and run some script on the node
> ...
> logger.info <http://logger.info>("Node in group "+node.getGroup()+"
> initiallized.");
> 
> 
> Output:
> New node eu-central-1/i-1ff660a3 [ip redacted, ip redacted] in group
> crossbuild-ami-87564feb
> Node in group null initiallized.
> 
> 
> And indeed the group tag is missing in EC2 console. I hardly believe
> such basic functionality would be a bug in Jclouds but I also can't see
> what I am doing wrong. Any ideas?

Re: Problem bootstraping FreeBSD on EC2

[cen]

Even using full path for the command it still doesn't work. It's like 
I'm in SSH limbo with access completely restricted.
I'll play around with it a bit more but I think I will ultimately create 
my own AMI with sudo and bash preinstalled.


Ignasi Barrera je 28. 01. 2016 ob 11:12 napisal:

I have no experience with BSD instances, but could it be a PATH issue?
Can you try setting the PATH manually?

On 28 January 2016 at 09:13, Klemen Ferjančič  wrote:

It seems I've hit a wall.

1. For some reason, even when disabling init script on template builder,
it still tries to run it. So I decided to skip it for now.

.wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);

2. Running as root and disabling sudo does not actually remove sudo.
Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
seems that sudo is always run in front.

opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
.overrideLoginCredentials(getLoginForCommandExecution(os));

Result:
`sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...

which means I can't run as root.

3. So I decided to run as regular user (not root and not sudo) and just
switch to root manually. Works if I do it by hand:
su root
pkg install -y sudo
pkg install -y bash

Opts:
.wrapInInitScript(false).runAsRoot(false)
.overrideLoginCredentials(getLoginForCommandExecution(os))

But for some reason, it seems that jclouds puts me in some weird
restricted shell where I can't execute anything.
su returns "Sorry" and trying to run a simple command like "whoami"
results in: not found. It's clear to me that this is not the same
environment as the one
when I manually ssh into the machine so what is the difference?


Best regards, Klemen

On 27. 01. 2016 11:06, Ignasi Barrera wrote:

No. You can use the one in the template builder to run the first one and
the run the rest by calling computeService.runScriptOnNode afterhaving
created the node.

El 27/1/2016 11:03 a. m., "Klemen Ferjančič" > escribió:

 If I add additional scripts on top of template builder are they going to
 be executed in order or is the solution more complex than this?

 Something like:

 templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
 .wrapInitScript(true).runScript(everythingElse())

 On 27. 01. 2016 10:56, Ignasi Barrera wrote:
 > Yes, the RunScriptOptions provide options to run the scripts as root,
 > and also to explicitly disable sudo.
 >
 > Currently the wrapper script requires bash, so you'll have to install
 > it first if you need to use it. You could split the runscript in two
 > operations: a first one with the wrapInitScript(false) to just install
 > bash, and a second one to run the desired script.
 >
 > That wrapper script, as said, contains several helpers to let jclouds
 > query the status of the script execution: see if it is still running,
 > allow to abort it, etc. If you are running scripts that take time, I'd
 > recommend you run them with the wrapper script.
 >
 > I.
 >
 > On 27 January 2016 at 10:48, Klemen Ferjančič > wrote:
 >> Good advice, I suspect I know what is going on.
 >>
 >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
 >> However, if I run ./init-bootstrap it says "not found". I checked the
 >> init script and it seems that bash is expected to exist on the system
 >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
 >> could install bash with runScript but I don't think personal
 script is
 >> executed before the init one? Another interesting note: sudo is not
 >> preinstalled either so I probably need to run my script as root. I'll
 >> play around with this and see what I can do.
 >>
 >> Code snippet:
 >>
 >> templateBuilder.osFamily(OsFamily.FREEBSD);
 >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
 >> EC2TemplateOptions o =
 EC2TemplateOptions.Builder.keyPair("mykeypair")
 >>
 
.overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
 >> templateBuilder.locationId(REGION);
 >> templateBuilder.hardwareId(INSTANCE);
 >> templateBuilder.options(o);
 >>
 >>
 >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
 >>> Hi Klemen,
 >>>
 >>> jclouds generates and uploads that script to the node. It is a
 wrapper
 >>> script that provides some helpers to get the status of the
 configured
 >>> script, and to properly capture the output, stderr, and make it
 >>> possible to abort its execution.
 >>>
 >>> After the failure, can you log in to the instance and see which
 files
 >>> do you have in the user's home directory and in /tmp?
 >>>
 >>> You can also disable the wrapper script by 

Re: Problem bootstraping FreeBSD on EC2

[cen]

Found the solution. If I execute commands in a single line using &&, 
written as a java string they work as expected.
But reading them from a file where I write them line by line it cause 
the problems mentioned previously:


Files.toString(file, Charsets.UTF_8)

If I write them in a single line in the file it works as expected too. I 
tried to print the commands when they are line by line
and everything seems normal in console. Maybe it's the line breaks in 
combination with tcsh or something in that regard causing the problem.


Plus another find: you can't actually su root && command, the proper way 
is su - root -c "commands && here". su root seems to break the flow.



Ignasi Barrera je 28. 01. 2016 ob 11:12 napisal:

I have no experience with BSD instances, but could it be a PATH issue?
Can you try setting the PATH manually?

On 28 January 2016 at 09:13, Klemen Ferjančič  wrote:

It seems I've hit a wall.

1. For some reason, even when disabling init script on template builder,
it still tries to run it. So I decided to skip it for now.

.wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);

2. Running as root and disabling sudo does not actually remove sudo.
Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
seems that sudo is always run in front.

opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
.overrideLoginCredentials(getLoginForCommandExecution(os));

Result:
`sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...

which means I can't run as root.

3. So I decided to run as regular user (not root and not sudo) and just
switch to root manually. Works if I do it by hand:
su root
pkg install -y sudo
pkg install -y bash

Opts:
.wrapInInitScript(false).runAsRoot(false)
.overrideLoginCredentials(getLoginForCommandExecution(os))

But for some reason, it seems that jclouds puts me in some weird
restricted shell where I can't execute anything.
su returns "Sorry" and trying to run a simple command like "whoami"
results in: not found. It's clear to me that this is not the same
environment as the one
when I manually ssh into the machine so what is the difference?


Best regards, Klemen

On 27. 01. 2016 11:06, Ignasi Barrera wrote:

No. You can use the one in the template builder to run the first one and
the run the rest by calling computeService.runScriptOnNode afterhaving
created the node.

El 27/1/2016 11:03 a. m., "Klemen Ferjančič" > escribió:

 If I add additional scripts on top of template builder are they going to
 be executed in order or is the solution more complex than this?

 Something like:

 templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
 .wrapInitScript(true).runScript(everythingElse())

 On 27. 01. 2016 10:56, Ignasi Barrera wrote:
 > Yes, the RunScriptOptions provide options to run the scripts as root,
 > and also to explicitly disable sudo.
 >
 > Currently the wrapper script requires bash, so you'll have to install
 > it first if you need to use it. You could split the runscript in two
 > operations: a first one with the wrapInitScript(false) to just install
 > bash, and a second one to run the desired script.
 >
 > That wrapper script, as said, contains several helpers to let jclouds
 > query the status of the script execution: see if it is still running,
 > allow to abort it, etc. If you are running scripts that take time, I'd
 > recommend you run them with the wrapper script.
 >
 > I.
 >
 > On 27 January 2016 at 10:48, Klemen Ferjančič > wrote:
 >> Good advice, I suspect I know what is going on.
 >>
 >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
 >> However, if I run ./init-bootstrap it says "not found". I checked the
 >> init script and it seems that bash is expected to exist on the system
 >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
 >> could install bash with runScript but I don't think personal
 script is
 >> executed before the init one? Another interesting note: sudo is not
 >> preinstalled either so I probably need to run my script as root. I'll
 >> play around with this and see what I can do.
 >>
 >> Code snippet:
 >>
 >> templateBuilder.osFamily(OsFamily.FREEBSD);
 >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
 >> EC2TemplateOptions o =
 EC2TemplateOptions.Builder.keyPair("mykeypair")
 >>
 
.overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
 >> templateBuilder.locationId(REGION);
 >> templateBuilder.hardwareId(INSTANCE);
 >> templateBuilder.options(o);
 >>
 >>
 >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
 >>> Hi Klemen,
 >>>
 >>> jclouds generates and uploads that script to the node.