Re: Use JClouds to talk to non AWS cloud with S3 API

[Ranjith R]

We did face a similar issue in the past, and found that the accept header
set by the HttpClient was being rejected by the Swift3 middleware.  We
replaced it with apache httpclient, and it worked. Perhaps you could give
it a try.

Iterable modules = ImmutableSet. of(new
ApacheHCHttpCommandExecutorServiceModule());

BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())

.endpoint("http://myserver:8080";)

.credentials(username, password)

.overrides(overrides)

.modules(modules)

.buildView(BlobStoreContext.class);


Regards,
Ranjith
On Fri, Jan 29, 2016 at 10:34 AM, Andrew Phillips 
wrote:

> Hi Julie
>
> Could you enable wire and debug logging so we can see exactly which calls
> jclouds is making [1]? Also, could you share a code snippet of how you are
> successfully using boto to do the same thing? Hopefully, we can find out
> where the calls from jclouds differ from what boto is doing...
>
> Regards
>
> ap
>
> [1] https://jclouds.apache.org/reference/logging/
>

Re: Use JClouds to talk to non AWS cloud with S3 API

[Andrew Phillips]

Hi Julie

Could you enable wire and debug logging so we can see exactly which 
calls jclouds is making [1]? Also, could you share a code snippet of how 
you are successfully using boto to do the same thing? Hopefully, we can 
find out where the calls from jclouds differ from what boto is doing...


Regards

ap

[1] https://jclouds.apache.org/reference/logging/

Use JClouds to talk to non AWS cloud with S3 API

[Yao, Julie]

I am using jcloud 1.9.2. We have set up our own swift repository with s3 api. 
We can access the repository using boto python script

I am using the code in the following post

http://stackoverflow.com/questions/9389840/use-jclouds-to-talk-to-non-aws-cloud-with-s3-api

but I got error.:

public static void main(String args[]) {



String username = "test:tester";



String password = "testing";

Properties overrides = new Properties();

overrides.setProperty(PROPERTY_S3_VIRTUAL_HOST_BUCKETS, "false");



BlobStoreContext context = ContextBuilder.newBuilder(new S3ApiMetadata())

.endpoint("http://myserver:8080";)

.credentials(username, password)

.overrides(overrides)

.buildView(BlobStoreContext.class);

BlobStore blobStore = context.getBlobStore();

String bucket="public";

blobStore.containerExists(bucket);



System.out.println("bucket exist "+ bucket);

}

Exception in thread "main" org.jclouds.http.HttpResponseException: request: 
HEADhttp://myserver:8080/public HTTP/1.1 failed with response: HTTP/1.1 500 
Internal Server Error at 
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)



It throw exception on the line:

blobStore.containerExists(bucket);

Thanks

Re: Problem bootstraping FreeBSD on EC2

[cen]

Found the solution. If I execute commands in a single line using &&, 
written as a java string they work as expected.
But reading them from a file where I write them line by line it cause 
the problems mentioned previously:


Files.toString(file, Charsets.UTF_8)

If I write them in a single line in the file it works as expected too. I 
tried to print the commands when they are line by line
and everything seems normal in console. Maybe it's the line breaks in 
combination with tcsh or something in that regard causing the problem.


Plus another find: you can't actually su root && command, the proper way 
is su - root -c "commands && here". su root seems to break the flow.



Ignasi Barrera je 28. 01. 2016 ob 11:12 napisal:

I have no experience with BSD instances, but could it be a PATH issue?
Can you try setting the PATH manually?

On 28 January 2016 at 09:13, Klemen Ferjančič  wrote:

It seems I've hit a wall.

1. For some reason, even when disabling init script on template builder,
it still tries to run it. So I decided to skip it for now.

.wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);

2. Running as root and disabling sudo does not actually remove sudo.
Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
seems that sudo is always run in front.

opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
.overrideLoginCredentials(getLoginForCommandExecution(os));

Result:
`sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...

which means I can't run as root.

3. So I decided to run as regular user (not root and not sudo) and just
switch to root manually. Works if I do it by hand:
su root
pkg install -y sudo
pkg install -y bash

Opts:
.wrapInInitScript(false).runAsRoot(false)
.overrideLoginCredentials(getLoginForCommandExecution(os))

But for some reason, it seems that jclouds puts me in some weird
restricted shell where I can't execute anything.
su returns "Sorry" and trying to run a simple command like "whoami"
results in: not found. It's clear to me that this is not the same
environment as the one
when I manually ssh into the machine so what is the difference?


Best regards, Klemen

On 27. 01. 2016 11:06, Ignasi Barrera wrote:

No. You can use the one in the template builder to run the first one and
the run the rest by calling computeService.runScriptOnNode afterhaving
created the node.

El 27/1/2016 11:03 a. m., "Klemen Ferjančič" mailto:imba...@gmail.com>> escribió:

 If I add additional scripts on top of template builder are they going to
 be executed in order or is the solution more complex than this?

 Something like:

 templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
 .wrapInitScript(true).runScript(everythingElse())

 On 27. 01. 2016 10:56, Ignasi Barrera wrote:
 > Yes, the RunScriptOptions provide options to run the scripts as root,
 > and also to explicitly disable sudo.
 >
 > Currently the wrapper script requires bash, so you'll have to install
 > it first if you need to use it. You could split the runscript in two
 > operations: a first one with the wrapInitScript(false) to just install
 > bash, and a second one to run the desired script.
 >
 > That wrapper script, as said, contains several helpers to let jclouds
 > query the status of the script execution: see if it is still running,
 > allow to abort it, etc. If you are running scripts that take time, I'd
 > recommend you run them with the wrapper script.
 >
 > I.
 >
 > On 27 January 2016 at 10:48, Klemen Ferjančič mailto:imba...@gmail.com>> wrote:
 >> Good advice, I suspect I know what is going on.
 >>
 >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
 >> However, if I run ./init-bootstrap it says "not found". I checked the
 >> init script and it seems that bash is expected to exist on the system
 >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
 >> could install bash with runScript but I don't think personal
 script is
 >> executed before the init one? Another interesting note: sudo is not
 >> preinstalled either so I probably need to run my script as root. I'll
 >> play around with this and see what I can do.
 >>
 >> Code snippet:
 >>
 >> templateBuilder.osFamily(OsFamily.FREEBSD);
 >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
 >> EC2TemplateOptions o =
 EC2TemplateOptions.Builder.keyPair("mykeypair")
 >>
 
.overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
 >> templateBuilder.locationId(REGION);
 >> templateBuilder.hardwareId(INSTANCE);
 >> templateBuilder.options(o);
 >>
 >>
 >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
 >>> Hi Klemen,
 >>>
 >>> jclouds generates and uploads that script to the node. It is a
 wrapper
 >>> script that provides some helpers 

Re: Problem bootstraping FreeBSD on EC2

[cen]

Even using full path for the command it still doesn't work. It's like 
I'm in SSH limbo with access completely restricted.
I'll play around with it a bit more but I think I will ultimately create 
my own AMI with sudo and bash preinstalled.


Ignasi Barrera je 28. 01. 2016 ob 11:12 napisal:

I have no experience with BSD instances, but could it be a PATH issue?
Can you try setting the PATH manually?

On 28 January 2016 at 09:13, Klemen Ferjančič  wrote:

It seems I've hit a wall.

1. For some reason, even when disabling init script on template builder,
it still tries to run it. So I decided to skip it for now.

.wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);

2. Running as root and disabling sudo does not actually remove sudo.
Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
seems that sudo is always run in front.

opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
.overrideLoginCredentials(getLoginForCommandExecution(os));

Result:
`sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...

which means I can't run as root.

3. So I decided to run as regular user (not root and not sudo) and just
switch to root manually. Works if I do it by hand:
su root
pkg install -y sudo
pkg install -y bash

Opts:
.wrapInInitScript(false).runAsRoot(false)
.overrideLoginCredentials(getLoginForCommandExecution(os))

But for some reason, it seems that jclouds puts me in some weird
restricted shell where I can't execute anything.
su returns "Sorry" and trying to run a simple command like "whoami"
results in: not found. It's clear to me that this is not the same
environment as the one
when I manually ssh into the machine so what is the difference?


Best regards, Klemen

On 27. 01. 2016 11:06, Ignasi Barrera wrote:

No. You can use the one in the template builder to run the first one and
the run the rest by calling computeService.runScriptOnNode afterhaving
created the node.

El 27/1/2016 11:03 a. m., "Klemen Ferjančič" mailto:imba...@gmail.com>> escribió:

 If I add additional scripts on top of template builder are they going to
 be executed in order or is the solution more complex than this?

 Something like:

 templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
 .wrapInitScript(true).runScript(everythingElse())

 On 27. 01. 2016 10:56, Ignasi Barrera wrote:
 > Yes, the RunScriptOptions provide options to run the scripts as root,
 > and also to explicitly disable sudo.
 >
 > Currently the wrapper script requires bash, so you'll have to install
 > it first if you need to use it. You could split the runscript in two
 > operations: a first one with the wrapInitScript(false) to just install
 > bash, and a second one to run the desired script.
 >
 > That wrapper script, as said, contains several helpers to let jclouds
 > query the status of the script execution: see if it is still running,
 > allow to abort it, etc. If you are running scripts that take time, I'd
 > recommend you run them with the wrapper script.
 >
 > I.
 >
 > On 27 January 2016 at 10:48, Klemen Ferjančič mailto:imba...@gmail.com>> wrote:
 >> Good advice, I suspect I know what is going on.
 >>
 >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
 >> However, if I run ./init-bootstrap it says "not found". I checked the
 >> init script and it seems that bash is expected to exist on the system
 >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
 >> could install bash with runScript but I don't think personal
 script is
 >> executed before the init one? Another interesting note: sudo is not
 >> preinstalled either so I probably need to run my script as root. I'll
 >> play around with this and see what I can do.
 >>
 >> Code snippet:
 >>
 >> templateBuilder.osFamily(OsFamily.FREEBSD);
 >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
 >> EC2TemplateOptions o =
 EC2TemplateOptions.Builder.keyPair("mykeypair")
 >>
 
.overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
 >> templateBuilder.locationId(REGION);
 >> templateBuilder.hardwareId(INSTANCE);
 >> templateBuilder.options(o);
 >>
 >>
 >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
 >>> Hi Klemen,
 >>>
 >>> jclouds generates and uploads that script to the node. It is a
 wrapper
 >>> script that provides some helpers to get the status of the
 configured
 >>> script, and to properly capture the output, stderr, and make it
 >>> possible to abort its execution.
 >>>
 >>> After the failure, can you log in to the instance and see which
 files
 >>> do you have in the user's home directory and in /tmp?
 >>>
 >>> You can also disable the wrapper script by configuring the
 >>> "wrapInitScript(false)" in the RunScriptOp

Re: Problem bootstraping FreeBSD on EC2

[Ignasi Barrera]

I have no experience with BSD instances, but could it be a PATH issue?
Can you try setting the PATH manually?

On 28 January 2016 at 09:13, Klemen Ferjančič  wrote:
> It seems I've hit a wall.
>
> 1. For some reason, even when disabling init script on template builder,
> it still tries to run it. So I decided to skip it for now.
>
> .wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);
>
> 2. Running as root and disabling sudo does not actually remove sudo.
> Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
> seems that sudo is always run in front.
>
> opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
> .overrideLoginCredentials(getLoginForCommandExecution(os));
>
> Result:
> `sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...
>
> which means I can't run as root.
>
> 3. So I decided to run as regular user (not root and not sudo) and just
> switch to root manually. Works if I do it by hand:
> su root
> pkg install -y sudo
> pkg install -y bash
>
> Opts:
> .wrapInInitScript(false).runAsRoot(false)
> .overrideLoginCredentials(getLoginForCommandExecution(os))
>
> But for some reason, it seems that jclouds puts me in some weird
> restricted shell where I can't execute anything.
> su returns "Sorry" and trying to run a simple command like "whoami"
> results in: not found. It's clear to me that this is not the same
> environment as the one
> when I manually ssh into the machine so what is the difference?
>
>
> Best regards, Klemen
>
> On 27. 01. 2016 11:06, Ignasi Barrera wrote:
>> No. You can use the one in the template builder to run the first one and
>> the run the rest by calling computeService.runScriptOnNode afterhaving
>> created the node.
>>
>> El 27/1/2016 11:03 a. m., "Klemen Ferjančič" > > escribió:
>>
>> If I add additional scripts on top of template builder are they going to
>> be executed in order or is the solution more complex than this?
>>
>> Something like:
>>
>> templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
>> .wrapInitScript(true).runScript(everythingElse())
>>
>> On 27. 01. 2016 10:56, Ignasi Barrera wrote:
>> > Yes, the RunScriptOptions provide options to run the scripts as root,
>> > and also to explicitly disable sudo.
>> >
>> > Currently the wrapper script requires bash, so you'll have to install
>> > it first if you need to use it. You could split the runscript in two
>> > operations: a first one with the wrapInitScript(false) to just install
>> > bash, and a second one to run the desired script.
>> >
>> > That wrapper script, as said, contains several helpers to let jclouds
>> > query the status of the script execution: see if it is still running,
>> > allow to abort it, etc. If you are running scripts that take time, I'd
>> > recommend you run them with the wrapper script.
>> >
>> > I.
>> >
>> > On 27 January 2016 at 10:48, Klemen Ferjančič > > wrote:
>> >> Good advice, I suspect I know what is going on.
>> >>
>> >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
>> >> However, if I run ./init-bootstrap it says "not found". I checked the
>> >> init script and it seems that bash is expected to exist on the system
>> >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
>> >> could install bash with runScript but I don't think personal
>> script is
>> >> executed before the init one? Another interesting note: sudo is not
>> >> preinstalled either so I probably need to run my script as root. I'll
>> >> play around with this and see what I can do.
>> >>
>> >> Code snippet:
>> >>
>> >> templateBuilder.osFamily(OsFamily.FREEBSD);
>> >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
>> >> EC2TemplateOptions o =
>> EC2TemplateOptions.Builder.keyPair("mykeypair")
>> >>
>> 
>> .overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
>> >> templateBuilder.locationId(REGION);
>> >> templateBuilder.hardwareId(INSTANCE);
>> >> templateBuilder.options(o);
>> >>
>> >>
>> >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
>> >>> Hi Klemen,
>> >>>
>> >>> jclouds generates and uploads that script to the node. It is a
>> wrapper
>> >>> script that provides some helpers to get the status of the
>> configured
>> >>> script, and to properly capture the output, stderr, and make it
>> >>> possible to abort its execution.
>> >>>
>> >>> After the failure, can you log in to the instance and see which
>> files
>> >>> do you have in the user's home directory and in /tmp?
>> >>>
>> >>> You can also disable the wrapper script by configuring the
>> >>> "wrapInitScript(false)" in the RunScriptOptions, and see if the
>> >>> problem persists.
>> >>>
>>  

Re: Problem bootstraping FreeBSD on EC2

[Klemen Ferjančič]

It seems I've hit a wall.

1. For some reason, even when disabling init script on template builder,
it still tries to run it. So I decided to skip it for now.

.wrapInInitScript(false).runScript(getBootInstructions(os)).runAsRoot(true);

2. Running as root and disabling sudo does not actually remove sudo.
Taking a closer look at RunScriptOnNodeUsingSsh.java->execAsRoot, it
seems that sudo is always run in front.

opts.wrapInInitScript(false).runAsRoot(true).overrideAuthenticateSudo(false)
.overrideLoginCredentials(getLoginForCommandExecution(os));

Result:
`sudo sh <<'RUN_SCRIPT_AS_ROOT_SSH' ...

which means I can't run as root.

3. So I decided to run as regular user (not root and not sudo) and just
switch to root manually. Works if I do it by hand:
su root
pkg install -y sudo
pkg install -y bash

Opts:
.wrapInInitScript(false).runAsRoot(false)
.overrideLoginCredentials(getLoginForCommandExecution(os))

But for some reason, it seems that jclouds puts me in some weird
restricted shell where I can't execute anything.
su returns "Sorry" and trying to run a simple command like "whoami"
results in: not found. It's clear to me that this is not the same
environment as the one
when I manually ssh into the machine so what is the difference?


Best regards, Klemen

On 27. 01. 2016 11:06, Ignasi Barrera wrote:
> No. You can use the one in the template builder to run the first one and
> the run the rest by calling computeService.runScriptOnNode afterhaving
> created the node.
> 
> El 27/1/2016 11:03 a. m., "Klemen Ferjančič"  > escribió:
> 
> If I add additional scripts on top of template builder are they going to
> be executed in order or is the solution more complex than this?
> 
> Something like:
> 
> templateBuilder.wrapInInitScript(false).runScript(installBashAndSudo())
> .wrapInitScript(true).runScript(everythingElse())
> 
> On 27. 01. 2016 10:56, Ignasi Barrera wrote:
> > Yes, the RunScriptOptions provide options to run the scripts as root,
> > and also to explicitly disable sudo.
> >
> > Currently the wrapper script requires bash, so you'll have to install
> > it first if you need to use it. You could split the runscript in two
> > operations: a first one with the wrapInitScript(false) to just install
> > bash, and a second one to run the desired script.
> >
> > That wrapper script, as said, contains several helpers to let jclouds
> > query the status of the script execution: see if it is still running,
> > allow to abort it, etc. If you are running scripts that take time, I'd
> > recommend you run them with the wrapper script.
> >
> > I.
> >
> > On 27 January 2016 at 10:48, Klemen Ferjančič  > wrote:
> >> Good advice, I suspect I know what is going on.
> >>
> >> Home directory contains bootstrap and /tmp/init-bootstrap exists.
> >> However, if I run ./init-bootstrap it says "not found". I checked the
> >> init script and it seems that bash is expected to exist on the system
> >> (#!/bin/bash), however, bash does not come preinstalled on FreeBSD. I
> >> could install bash with runScript but I don't think personal
> script is
> >> executed before the init one? Another interesting note: sudo is not
> >> preinstalled either so I probably need to run my script as root. I'll
> >> play around with this and see what I can do.
> >>
> >> Code snippet:
> >>
> >> templateBuilder.osFamily(OsFamily.FREEBSD);
> >> templateBuilder.imageId(REGION + "/ami-9f5549f3");
> >> EC2TemplateOptions o =
> EC2TemplateOptions.Builder.keyPair("mykeypair")
> >>
> 
> .overrideLoginCredentials(getLoginForCommandExecution(os)).runScript(getBootInstructions(os));
> >> templateBuilder.locationId(REGION);
> >> templateBuilder.hardwareId(INSTANCE);
> >> templateBuilder.options(o);
> >>
> >>
> >> On 27. 01. 2016 10:23, Ignasi Barrera wrote:
> >>> Hi Klemen,
> >>>
> >>> jclouds generates and uploads that script to the node. It is a
> wrapper
> >>> script that provides some helpers to get the status of the
> configured
> >>> script, and to properly capture the output, stderr, and make it
> >>> possible to abort its execution.
> >>>
> >>> After the failure, can you log in to the instance and see which
> files
> >>> do you have in the user's home directory and in /tmp?
> >>>
> >>> You can also disable the wrapper script by configuring the
> >>> "wrapInitScript(false)" in the RunScriptOptions, and see if the
> >>> problem persists.
> >>>
> >>> Could you also share a small code snippet of the code you're
> using to
> >>> bootstrap the node?
> >>>
> >>> I.
> >>>
> >>> On 27 January 2016 at 09:39, Klemen Ferjančič  > wrote:
>  Hi
> 
>  When I create a new Ec2 inst