Re: @ActionEvent generation doesn't work when method is called within the same manager context

[Darren Shepherd]

Any call to this.method() will not process @ActionEvent annotation.  That is 
expected after the spring changes that were done.  There's no change that can 
be done to change that behavior, it's just how spring AOP work because it's 
proxy based.

If you want @ActionEvent to be processed you must cross beans, or just 
programmatically record the action event.

Darren

On May 7, 2014, at 1:57 PM, "Alena Prokharchyk" 
mailto:alena.prokharc...@citrix.com>> wrote:

Kelven/Darren,

While working on the fix for  CLOUDSTACK-6595, I’ve noticed that the methods 
annotated with @ActionEvents sometimes don’t generate the events. It happens 
when the annotated method is called within the same manager. When its called 
from another manager, the event is generated and inserted to cloud.events 
table. Kelven said its related to the Spring changes done by Darren.

Can you please take a look at it? Kelven, I’ve reassigned CLOUDSTACK-6595 to 
you (fix has to be submitted to 4.4-forward first, then cherry-picked to 4.4)

-Alena.

From: Alena Prokharchyk 
mailto:alena.prokharc...@citrix.com>>
Date: Wednesday, May 7, 2014 at 1:53 PM
To: Daan Hoogland mailto:daan.hoogl...@gmail.com>>, 
"dev@cloudstack.apache.org" 
mailto:dev@cloudstack.apache.org>>
Subject: [ACS44] Cherry pick for CLOUDSTACK-6595

Daan, can you please cherry-pick the commit to 4.4 branch:

commit b9284db857cd001e75926cb7c49b6aff4ab63695
Author: Alena Prokharchyk 
mailto:alena.prokharc...@citrix.com>>
Date:   Wed May 7 13:44:26 2014 -0700

CLOUDSTACK-6595: call expunge method marked with @ActionEvent from expunge 
thread context. So the action even EXPUNGE.VM is generated when vm is expunged 
by the thread.

Thank you,
Alena.

Re: Event Interceptor fix - contrail plugin : Review Request

[Darren Shepherd]

I'm sorry, I thought I did this weeks ago.  Do you not see the commit in 
master?  I'll double check, maybe I forgot to push.

Darren




From: Suresh Balineni 
Date: 2014-02-24 19:19:57 +
To: Darren Shepherd ,dev@cloudstack.apache.org 

Subject: Re: Event Interceptor fix - contrail plugin : Review Request
Cc: Sheng Yang ,Hugo ,Animesh Chaturvedi 
,Sunil Bakhru 

Hi Darren,

Can  you please review and ship these changes (Master branch)? Without this, 
contrail implementation won't work.

Thanks,
Suresh

From: Suresh Balineni 
mailto:sbalin...@junipernetworks.onmicrosoft.com>>
Date: Monday, February 10, 2014 12:10 PM
To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" 
mailto:dev@cloudstack.apache.org>>, Darren Shepherd 
mailto:darren.sheph...@citrix.com>>
Cc: Darren Shepherd 
mailto:darren.sheph...@citrix.com>>, Hugo 
mailto:h...@trippaers.nl>>, Animesh Chaturvedi 
mailto:animesh.chaturv...@citrix.com>>, Sheng 
Yang mailto:sh...@yasker.org>>, Sunil Bakhru 
mailto:sbak...@juniper.net>>
Subject: Re: Event Interceptor fix - contrail plugin : Review Request

Hi Darren,

Can you please review this ?

Thanks,
Suresh

From: Suresh Balineni mailto:sbalin...@juniper.net>>
Date: Monday, February 3, 2014 3:24 PM
Cc: Darren Shepherd 
mailto:darren.sheph...@citrix.com>>, Hugo 
mailto:h...@trippaers.nl>>, Animesh Chaturvedi 
mailto:animesh.chaturv...@citrix.com>>, Sheng 
Yang mailto:sh...@yasker.org>>, Sunil Bakhru 
mailto:sbak...@juniper.net>>
Subject: Re: Event Interceptor fix - contrail plugin : Review Request

Hi Darren,

Can you review this?

Thanks,
Suresh


On Jan 30, 2014, at 4:23 PM, "Suresh Balineni" 
mailto:sbalin...@juniper.net>> wrote:

Hi Darren,

Can you please review and ship the following event interceptor fix in CS master 
branch?
I have implemented this as per your suggestion and it works fine.

https://reviews.apache.org/r/17337/

This review is pending for while

Thanks,
Suresh

Re: Review Request 17337: Contrail Event Interceptor fix

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17337/#review34170
---

Ship it!


Commit c1cfaa2d1c553d86ae35443d15100b20c2d1b69f

- Darren Shepherd


On Jan. 24, 2014, 11:23 p.m., Suresh Balineni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17337/
> ---
> 
> (Updated Jan. 24, 2014, 11:23 p.m.)
> 
> 
> Review request for cloudstack.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> When project/domains are created,  our plugin code won't get invoked. Our 
> contrail plugin uses Event infrastructure provided by cloudstack to receive 
> these events and handle them accordingly. It is must to create 
> domains/projects before creating a virtual network/vm object in contrail 
> implementation. Hence our plugin must need a way to get notified about 
> project/domain events.
> 
> 
> Diffs
> -
> 
>   
> plugins/network-elements/juniper-contrail/resources/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml
>  PRE-CREATION 
>   
> plugins/network-elements/juniper-contrail/src/org/apache/cloudstack/network/contrail/management/EventUtils.java
>  29b58fb 
> 
> Diff: https://reviews.apache.org/r/17337/diff/
> 
> 
> Testing
> ---
> 
> Verified locally
> 
> 
> Thanks,
> 
> Suresh Balineni
> 
>

Re: Upgrade to newer version of gson?

[Darren Shepherd]

If you stay with the 1.x it should be safe.

Darren

> On Jan 3, 2014, at 6:56 PM, "Alex Huang"  wrote:
> 
> I tried to bump up to a later version before.  Had horrible trouble with it.  
> It was a problem with in how the API was using gson to render.  I forget 
> whether this is the version I attempted.
> 
> --Alex
> 
>> -Original Message-
>> From: Edison Su
>> Sent: Friday, January 3, 2014 5:53 PM
>> To: dev@cloudstack.apache.org
>> Cc: Alex Huang; Darren Shepherd
>> Subject: RE: Upgrade to newer version of gson?
>> 
>> Oh, a minor upgrade should be enough:
>> https://code.google.com/p/google-gson/issues/detail?id=354
>> Just need to bump up to 1.7.2.
>> 
>>> -Original Message-
>>> From: Edison Su [mailto:edison...@citrix.com]
>>> Sent: Friday, January 03, 2014 5:48 PM
>>> To: dev@cloudstack.apache.org
>>> Cc: Alex Huang; Darren Shepherd
>>> Subject: Upgrade to newer version of gson?
>>> 
>>> Hit a bug in gson:
>>> https://groups.google.com/forum/#!msg/google-
>>> gson/3OdgAAUOi3c/UMTmNt1XnG0J
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-5573
>>> It's fixed since gson-2.0. Should we upgrade to gson-2.x?

Re: Loading order of Adapter components

[Darren Shepherd]

I think it's documented on the wiki but the name comes from the getName() of 
the class.  If getName isn't available the it's getClass().getSimpleName().  
The default behavior of getName is to grab the class name, so for most 
practical cases it's the class name.  

Darren

> On Dec 19, 2013, at 3:41 PM, Prachi Damle  wrote:
> 
> That worked. But initially I had tried with the small case bean id 
> ‘roleBasedEntityAccessChecker’ that matched the DB entry and that did not 
> load the order correctly.
>  
> So should it be the class name  and not the ‘id’ in the bean definition in 
> the order config?
>  
> Thanks,
> Prachi
>  
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] 
> Sent: Thursday, December 19, 2013 2:30 PM
> To: Prachi Damle
> Cc: dev@cloudstack.apache.org
> Subject: Re: Loading order of Adapter components
>  
> The names are cases sensitive.  Is there a reason you put the name as 
> lowercase in the global configuration?
> 
> Darren
> 
> On Dec 19, 2013, at 3:12 PM, "Prachi Damle"  wrote:
> 
> Hi Darren,
>  
> I am trying to add a new implementation of SecurityChecker interface which is 
> an Adapter.  I am adding  a plugin – RoleBasedChecker,  and I want to add it 
> to the list of SecurityCheckers at first position.
> However the new checker gets added to the list at the last position. The 
> order specified as below is not getting followed when the components are  
> loaded.
>  
> Do you think I am missing anything to be changed?
>  
> Changes I did:
>  
> 1)  I added my component in the global config:
>  
> 'Advanced', 'DEFAULT', 'ExtensionRegistry', 'security.checkers.order', 
> 'roleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker', 'The 
> order of precedence for the extensions', 
> 'roleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker', 
> '2013-12-18 01:45:31', 'Global', '0'
>  
>  
> 2)  I added it to the spring-core-registry-core-context.xml under 
> cloud-core
>  
> class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
> 
> 
>  
> value="RoleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker" 
> />
> 
>  
>  
> 3)  The bean itself is defined in the 
> spring-acl-role-based-access-checkers-context.xml under the plugin
>  class="org.apache.cloudstack.acl.entity.RoleBasedEntityAccessChecker" />
>  
>  
> Thanks,
> Prachi

Re: Loading order of Adapter components

[Darren Shepherd]

The names are cases sensitive.  Is there a reason you put the name as lowercase 
in the global configuration?

Darren

> On Dec 19, 2013, at 3:12 PM, "Prachi Damle"  wrote:
> 
> Hi Darren,
>  
> I am trying to add a new implementation of SecurityChecker interface which is 
> an Adapter.  I am adding  a plugin – RoleBasedChecker,  and I want to add it 
> to the list of SecurityCheckers at first position.
> However the new checker gets added to the list at the last position. The 
> order specified as below is not getting followed when the components are  
> loaded.
>  
> Do you think I am missing anything to be changed?
>  
> Changes I did:
>  
> 1)  I added my component in the global config:
>  
> 'Advanced', 'DEFAULT', 'ExtensionRegistry', 'security.checkers.order', 
> 'roleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker', 'The 
> order of precedence for the extensions', 
> 'roleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker', 
> '2013-12-18 01:45:31', 'Global', '0'
>  
>  
> 2)  I added it to the spring-core-registry-core-context.xml under 
> cloud-core
>  
> class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
> 
> 
>  
> value="RoleBasedEntityAccessChecker,AffinityGroupAccessChecker,DomainChecker" 
> />
> 
>  
>  
> 3)  The bean itself is defined in the 
> spring-acl-role-based-access-checkers-context.xml under the plugin
>  class="org.apache.cloudstack.acl.entity.RoleBasedEntityAccessChecker" />
>  
>  
> Thanks,
> Prachi

Re: cloudstack no longer monitoring log4j-cloud.xml?

[Darren Shepherd]

I'll look at it, but that util method that loads the log4j-cloud.xml is still 
being called, I can't remember the api off the top of my head.

My first guess would actually be that the file is getting loaded from a 
different place than /etc.  ACS packages a lot of files twice and I often see 
the scripts files being loaded from web-inf/classes and not the other location 
in /usr.  

Darren

> On Dec 17, 2013, at 5:03 PM, "Alex Huang"  wrote:
> 
> Hey Darren,
> 
> I've just been alerted that cloudstack doesn't seem to be dynamically loading 
> the log4j configurations any more.  This means you need to restart the server 
> every time someone changes the logging level.  This was done by the 
> componentlocator previously.  Was this lost during the spring changes?  Can 
> you bring the functionality back?  
> 
> I saw this in master but my guess is it's in 4.3 as well.
> 
> Thanks.
> 
> --Alex

Re: cloudstack no longer monitoring log4j-cloud.xml?

[Darren Shepherd]

I just tried on my desktop and it works.  LogUtils from ACS just calls
DOMConfigurator.configureAndWatch() which spawns a thread called
Filewatchdog.  It just sleep for 60 seconds and then checks the
modified time on the log4j-cloud.xml.  I haven't tested on a really
production setup though.  Can you tell me if you see the line with
"log4j configuration found at "

Darren

On Tue, Dec 17, 2013 at 10:20 PM, Alex Huang  wrote:
> Darren,
>
> Iirc, all I did was make a thread available to log4j to monitor the config 
> file that it had loaded.
>
> I checked the open file handles with Sheng and couldn't find any open file 
> handles to any log4j-cloud.xml, which if this was done, will always have an 
> open file handle.
>
> --Alex
>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Tuesday, December 17, 2013 9:16 PM
>> To: Alex Huang
>> Cc: dev@cloudstack.apache.org
>> Subject: Re: cloudstack no longer monitoring log4j-cloud.xml?
>>
>> Cc'ing dev list from the right email
>>
>> Darren
>>
>> > On Dec 17, 2013, at 10:14 PM, "Darren Shepherd"
>>  wrote:
>> >
>> > I'll look at it, but that util method that loads the log4j-cloud.xml is 
>> > still being
>> called, I can't remember the api off the top of my head.
>> >
>> > My first guess would actually be that the file is getting loaded from a
>> different place than /etc.  ACS packages a lot of files twice and I often 
>> see the
>> scripts files being loaded from web-inf/classes and not the other location in
>> /usr.
>> >
>> > Darren
>> >
>> >> On Dec 17, 2013, at 5:03 PM, "Alex Huang" 
>> wrote:
>> >>
>> >> Hey Darren,
>> >>
>> >> I've just been alerted that cloudstack doesn't seem to be dynamically
>> loading the log4j configurations any more.  This means you need to restart
>> the server every time someone changes the logging level.  This was done by
>> the componentlocator previously.  Was this lost during the spring changes?
>> Can you bring the functionality back?
>> >>
>> >> I saw this in master but my guess is it's in 4.3 as well.
>> >>
>> >> Thanks.
>> >>
>> >> --Alex

Re: cloudstack no longer monitoring log4j-cloud.xml?

[Darren Shepherd]

Cc'ing dev list from the right email

Darren

> On Dec 17, 2013, at 10:14 PM, "Darren Shepherd"  
> wrote:
> 
> I'll look at it, but that util method that loads the log4j-cloud.xml is still 
> being called, I can't remember the api off the top of my head.
> 
> My first guess would actually be that the file is getting loaded from a 
> different place than /etc.  ACS packages a lot of files twice and I often see 
> the scripts files being loaded from web-inf/classes and not the other 
> location in /usr.  
> 
> Darren
> 
>> On Dec 17, 2013, at 5:03 PM, "Alex Huang"  wrote:
>> 
>> Hey Darren,
>> 
>> I've just been alerted that cloudstack doesn't seem to be dynamically 
>> loading the log4j configurations any more.  This means you need to restart 
>> the server every time someone changes the logging level.  This was done by 
>> the componentlocator previously.  Was this lost during the spring changes?  
>> Can you bring the functionality back?  
>> 
>> I saw this in master but my guess is it's in 4.3 as well.
>> 
>> Thanks.
>> 
>> --Alex

Re: persistence layer

[Darren Shepherd]

Alright, this is long….

I have to say after the 10+ years of struggling with JDBC, Hibernate,
JbdcTemplate, JPA, QueryDSL, and countless custom solutions, I've have
never found a database access solution that has impressed me as much as
jooq. If I was starting from scratch today I'd definitely choose jooq. Now
all of that doesn't matter though. Evaluating database access solutions in
isolation is about as productive as discussing if c++ or java is a better
language. The answer being "it depends."  We really need to ground this
discussion in the realities of what ACS is today, what are the issues and
where do we want to go. Nobody is going to sign on to a complete gut and
rewrite of the data acces layer (DAL), so any solution must allow a slow
migration to it.

Current solution

ACS currently has a completely custom DAL. While the VOs are annotated with
JPA annotations it is dangerous to compare the custom ACS framework to JPA
as the differences are substantial. It is also important to note that ACS
does not have a ORM today. There is no relational mapping.

The ACS DAL can be broken down to five parts. 1) DAO framework (GenericDao)
2) object mapping 3) query api 4) crud w/ change tracking 5) transaction
management

DAO framework

ACS has a strict 1-to-1 mapping of VO to DAO. The generic DAO provides the
common methods like findById that you see in most java DAOs. The generic
DAO uses reflection to query the VOs’ JPA annotations to build up the
metadata for the crud operations and query building.

Object mapping
===
The metadata obtained through the GenericDao is used to map sql to the VOs.
If you use the GenericDaos findX or listByX methods or the query api it
will map the result to a VO. Additionally if you update a VO and persist it
the mapper will generate SQL from the VO.

Query API

ACS has a search builder api that allows you to build sql queries in a type
safe manner. The functionality is mostly limited to basically conditions
and joins. The api works for most of the use cases in ACS but still there a
quite a lot of raw sql statements in ACS because the api is not functional
enough.

CRUD with changing tracking
=
When POJOs are created by the GenericDao they are enhanced with cglib. All
changes to the pojos are tracked as updates. When persist is called the
updated fields are used to construct an update.

Transaction management
===
Transactions are managed programmatically through the transaction api.
Nested transactions and transaction propagation are not supported.
Isolation level is determined at the DB pool level, so all connections have
the same level which is read committed.

Problems

The general problems of the dal in ACS is that it's 100% custom. The APIs,
especially around transactions and querying, are often a stumbling block
for people developing in ACS. Additionally, since this is all custom, the
maintenance and enhancements of the DAL is solely the responsibility of the
ACS community. Finally, very little documentation exists around all of
this, and there is realistically about 3 people who understand the core
implementation.

You have to consider that ACS was started almost 5 years ago. I can
understand the creators not wanting to use hibernate or spring and instead
rolling their own solutions. The problem with rolling your own solution
typically ends up being that your use cases start off simple and your
custom framework does too. As the product matures and more complex use
cases are tackled things are no longer simple. You are then forced to
enhance your simple framework, but then the warts start to really show. Or,
you start bypassing it because it is too limiting.

DAO framework

I don't have major fundamental complaints with the DAOs. The API exposed is
fairly consistent with most DAOs you would see in any java application. The
main complaint I have is the strict 1-to-1 mapping between VO and DAO. In
general I don't like that design pattern and prefer to group DAOs according
to function and not tables. That's an opinion and I think the larger java
community tends to disagree with me. So I concede on that point. The bigger
problem with the 1-to-1 model is extensibility. Today, if you are writing a
new plugin and need to execute some query that doesn't exist today, you
really don't have much choice but to change the existing DAO in the core.
This leads to people instead of doing DB logic in a DAO, they end up doing
it in the services classes which is just bad.

Object mapping
===
The object mapping in ACS is quite simple. Since there is no relational
mapping, it just basically does the equivalent of BeanUtils.setProperty().
There is additional logic for database encryption. One of the down side of
the current mapping is that it just works for VOs. If you want to do some
more complex sql query that returns arbitrary output, you are forced to
fall back to raw JDBC.  The Query API,

Re: Resource Management/Locking [was: Re: What would be your ideal solution?]

[Darren Shepherd]

Okay, I'll have to stew over this for a bit.  My one general comment is
that it seems complicated.  Such a system seems like it would take a good
amount of effort to construct properly and as such it's a risky endeavour.

Darren


On Mon, Nov 25, 2013 at 12:10 PM, John Burwell  wrote:

> Darren,
>
> In a peer-to-peer model such as I describe, active-active is and is not a
> concept.  The supervision tree is responsible for identifying failure, and
> initiating process re-allocation for failed resources.  For example, if a
> pod’s management process crashed, it would also crash all of the processes
> managing the hosts in that pod.  The zone would then attempt to restart the
> pod’s management process (either local to the zone supervisor or on a
> remote instance which could be configurable) until it was able to start
> “ready” process for the child resource.
>
> This model requires a “special” root supervisor that is controlled by the
> orchestration tier which can identify when a zone supervisor becomes
> unavailable, and attempts to restart it.  The ownership of this “special”
> supervisor will require a consensus mechanism amongst the orchestration
> tier processes to elect an owner of the process and determine when a new
> owner needs to be elected (e.g. a Raft implementation such as barge [1]).
>  Given the orchestration tier is designed as an AP system, an orchestration
> tier process should be able to be an owner (i.e. the operator is not
> required to identify a “master” node).  There are likely other potential
> topologies (e.g. a root supervisor per zone rather than one for all zones),
> but in all cases ownership election would be the same.  Most importantly,
> there are no data durability requirements in this claim model.  When an
> orchestration process becomes unable to continue owning a root supervisor,
> the other orchestration processes recognize the missing owner and initiate
> ownership claim the process for the partition.
>
> In all failure scenarios, the supervision tree must be rebuilt from the
> point of failure downward using the process allocation process I previously
> described.  For an initial implementation, I would recommend taking simply
> throwing any parts of the supervision tree that are already running in the
> event of a widespread failure (e.g. a zone with many pods).  Dependent on
> the recovery time and SLAs, a future optimization may be to re-attach
> “orphaned” branches of the previous tree to the tree being built as part of
> the recovery process (e.g. loss a zone supervisor due to a switch failure).
>  Additionally, the system would also need a mechanism to hand-off ownership
> of the root supervisor for planned outages (hardware
> upgrades/decommissioning, maintenance windows, etc).
>
> Again, caveated with a a few hand waves, the idea is to build up a
> peer-to-peer management model that provides strict serialization
> guarantees.  Fundamentally, it utilizes a tree of processes to provide
> exclusive access, distribute work, and ensure availability requirements
> when partitions occur.  Details would need to be worked out for the best
> application to CloudStack (e.g root node ownership and orchestration tier
> gossip), but we would be implementing well-trod distributed systems
> concepts in the context cloud orchestration (sounds like a fun thing to do
> …).
>
> Thanks,
> -John
>
> [1]: https://github.com/mgodave/barge
>
> P.S. I see the libraries/frameworks referenced as the building blocks to a
> solution, but none of them (in whole or combination) solves the problem
> completely.
>
> On Nov 25, 2013, at 12:29 PM, Darren Shepherd 
> wrote:
>
> I will ask one basic question.  How do you forsee managing one mailbox per
> resource.  If I have multiple servers running in an active-active mode, how
> do you determine which server has the mailbox?  Do you create actors on
> demand?  How do you synchronize that operation?
>
> Darren
>
>
> On Mon, Nov 25, 2013 at 10:16 AM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> You bring up some interesting points.  I really need to digest this
>> further.  From a high level I think I agree, but there are a lot of implied
>> details of what you've said.
>>
>> Darren
>>
>>
>> On Mon, Nov 25, 2013 at 8:39 AM, John Burwell  wrote:
>>
>>> Darren,
>>>
>>> I originally presented my thoughts on this subject at CCC13 [1].
>>>  Fundamentally, I see CloudStack as having two distinct tiers —
>>> orchestration management and automation control.  The orchestration tier
>>> coordinates the automation control layer to fulfill user goals (e.g. create
>>> a VM instance, alter

Re: Unable to add NetScaler in 4.3 branch builds - Automation blocker

[Darren Shepherd]

It appears that I didn't commit any of the noredist spring configuration
for network-elements.  I will get that added in a bit once I just do some
quick validation.

Darren


On Wed, Nov 20, 2013 at 7:41 PM, Rayees Namathponnan <
rayees.namathpon...@citrix.com> wrote:

> Created below defect, EIP / ELB automation blocked due to this
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-5224
>
>
> Regards,
> Rayees
>
> -Original Message-
> From: Rayees Namathponnan [mailto:rayees.namathpon...@citrix.com]
> Sent: Wednesday, November 20, 2013 2:29 PM
> To: dev@cloudstack.apache.org
> Cc: Darren Shepherd  (
> darren.s.sheph...@gmail.com)
> Subject: Unable to add NetScaler in 4.3 branch builds
>
> Hi,
>
> I created noredist (non oss  build from 4.3 branch ;  and  created
> advanced zone in KVM.
>
> I am trying to add NetScaler as service provider;  but its failed with
> below error
>
>
> 2013-11-20 00:13:57,532 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-2de1fb83) ===START===  10.223.240.194 -- GET
>  physicalnetworkid=05ef4a90-f9ba-449f-b1b6
>
> -a437e6c4d4dd&apiKey=a8WrP3KUsp4G9e4xsseUEgqRJF0hoZ8uZwtIL5tM7fnSNgZ-uez5ht7x0GvH8fnVzI59gjnq93VRZzazazy8dQ&name=Netscaler&command=addNetworkServiceProvider&s
> ignature=Kz%2FM3E60UlpWJg0VbjEs%2FdHpIUE%3D&response=json
> 2013-11-20 00:13:57,553 INFO  [c.c.a.ApiServer]
> (catalina-exec-24:ctx-2de1fb83 ctx-0bbc33a1 ctx-e0da0b07) Unable to find
> the Network Element implementing the Service Provider 'Netscaler'
> 2013-11-20 00:13:57,554 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-2de1fb83 ctx-0bbc33a1 ctx-e0da0b07) ===END===
>  10.223.240.194 -- GET
>  
> physicalnetworkid=05ef4a90-f9ba-449f-b1b6-a437e6c4d4dd&apiKey=a8WrP3KUsp4G9e4xsseUEgqRJF0hoZ8uZwtIL5tM7fnSNgZ-uez5ht7x0GvH8fnVzI59gjnq93VRZzazazy8dQ&name=Netscaler&command=addNetworkServiceProvider&signature=Kz%2FM3E60UlpWJg0VbjEs%2FdHpIUE%3D&response=json
>
>
> Regards,
> Rayees
>

what's the procedure for committing to 4.3?

[Darren Shepherd]

What's the procedure for making changes to 4.3?  Obviously commit to master
and then cherry-pick but at this point is there any other control around
it?  If I need to commit something in 4.3 do I just do it myself?

Darren

Re: Enabling AMQP/RabbitMQ Events on master

[Darren Shepherd]

Just create a file on the classpath
META-INF/cloudstack/core/spring-event-bus-context.xml with the below
contents (change server, port, username, etc)

http://www.springframework.org/schema/beans";
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
   xmlns:context="http://www.springframework.org/schema/context";
   xmlns:aop="http://www.springframework.org/schema/aop";
   xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
  http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
  http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-3.0.xsd";
  >











You can put that file at
/etc/cloudstack/management/META-INF/cloudstack/core/spring-event-bus-context.xml

Darren


On Mon, Nov 25, 2013 at 8:24 AM, David Grizzanti <
david.grizza...@sungard.com> wrote:

> Murali,
>
> Would you be able to comment on how to enable the event message bus
> notifications on master?
>
> Thanks!
>
> --
> David Grizzanti
> Software Engineer
> Sungard Availability Services
>
> e: david.grizza...@sungard.com
> w: 215.446.1431
> c: 570.575.0315
>
> On November 21, 2013 at 12:35:31 PM, Alena Prokharchyk (
> alena.prokharc...@citrix.com) wrote:
>
> Murali might help you with that as he developed the feature.
>
> -Alena.
>
> From: Min Chen mailto:min.c...@citrix.com>>
> Date: Thursday, November 21, 2013 9:30 AM
> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>, Alena
> Prokharchyk  alena.prokharc...@citrix.com>>
> Cc: Darren Shepherd  darren.sheph...@citrix.com>>
> Subject: Re: Enabling AMQP/RabbitMQ Events on master
>
> CC Darren here, I am having the same question on current master.
>
> Thanks
> -min
>
> On 11/21/13 5:00 AM, "David Grizzanti"  <mailto:david.grizza...@sungard.com>> wrote:
>
> Alena,
>
> Do you or anyone else on the list have any updated information about
> enabling the events on master?
>
> Thanks!
>
> On Thursday, November 7, 2013, David Grizzanti wrote:
>
> Alena,
>
> I don't think these steps will work on master (not installing packages
> of
> cloudstack), I'm building from source. The componentContext XML file
> doesn't seem to exist anymore since some of the Spring refactoring was
> done.
>
> Thanks
>
>
> On Thu, Nov 7, 2013 at 12:42 PM, Alena Prokharchyk <
> alena.prokharc...@citrix.com<mailto:alena.prokharc...@citrix.com>> wrote:
>
> David,
>
> Here are the instructions that I've got from one of the CS QA
> engineers,
> hope it helps.
>
> FS -
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.1.0/html/Admi
> n_Guide/events.html#event-framework
>
>
>
> Test cases relating to this feature was covered as part of Regions
> Feature
> testing -
> https://cwiki.apache.org/confluence/download/attachments/30757955/Regions
> -Test-Execution-42.xlsx
>
>
>
>
>
> Steps to set up RabbitMQ Server:
>
>
>
> Have a RabbitMQ server set up.
>
> Enable rabbitmq_management plugin
>
> C:\Program Files\RabbitMQ
> Server\rabbitmq_server-3.0.3\sbin>rabbitmq-plugins enable
> rabbitmq_management
>
> Restart RabbitMQ service.
>
> In management server :
> Added the following in
> /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/component
> Context.xml
>
>
>  class="org.apache.cloudstack.mom.rabbitmq.RabbitMQEventBus">
> 
> 
> 
>
> 
> 
> 
> 
>
> Restart management server.
>
>
> -Alena.
>
> From: David Grizzanti  david.grizza...@sungard.com>>
> Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> Date: Thursday, November 7, 2013 5:04 AM
> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> Subject: Enabling AMQP/RabbitMQ Events on master
>
> Hi,
>
> I was looking for some help in enabling the AMQP/RabbitMQ events in
> CloudStack. I'm familiar with enabling these events in 4.2, however,
> I'm
> not all the familiar with Spring and given the new modularized changes
> I'm
> not really sure where the XML snippet belongs for the
> eventNotificationBus.
> Previously I had been placing this in applicationContext.
>
>
>
> --
> David Grizzanti
> Software Engineer
> Sungard Availability Services
>
> e: david.grizza...@sungard.com<mailto:david.grizza...@sungard.com>
> w: 215.446.1431
> c: 570.575.0315
>
>
>

Re: Resource Management/Locking [was: Re: What would be your ideal solution?]

[Darren Shepherd]

I will ask one basic question.  How do you forsee managing one mailbox per
resource.  If I have multiple servers running in an active-active mode, how
do you determine which server has the mailbox?  Do you create actors on
demand?  How do you synchronize that operation?

Darren


On Mon, Nov 25, 2013 at 10:16 AM, Darren Shepherd <
darren.s.sheph...@gmail.com> wrote:

> You bring up some interesting points.  I really need to digest this
> further.  From a high level I think I agree, but there are a lot of implied
> details of what you've said.
>
> Darren
>
>
> On Mon, Nov 25, 2013 at 8:39 AM, John Burwell  wrote:
>
>> Darren,
>>
>> I originally presented my thoughts on this subject at CCC13 [1].
>>  Fundamentally, I see CloudStack as having two distinct tiers —
>> orchestration management and automation control.  The orchestration tier
>> coordinates the automation control layer to fulfill user goals (e.g. create
>> a VM instance, alter a network route, snapshot a volume, etc) constrained
>> by policies defined by the operator (e.g. multi-tenacy boundaries, ACLs,
>> quotas, etc).  This layer must always be available to take new requests,
>> and to report the best available infrastructure state information.  Since
>> execution of work is guaranteed on completion of a request, this layer may
>> pend work to be completed when the appropriate devices become available.
>>
>> The automation control tier translates logical units of work to
>> underlying infrastructure component APIs.  Upon completion of unit of
>> work’s execution, the state of a device (e.g. hypervisor, storage device,
>> network switch, router, etc) matches the state managed by the orchestration
>> tier at the time unit of work was created.  In order to ensure that the
>> state of the underlying devices remains consistent, these units of work
>> must be executed serially.  Permitting concurrent changes to resources
>> creates race conditions that lead to resource overcommitment and state
>> divergence.   A symptom of this phenomenon are the myriad of scripts
>> operators write to “synchronize” state between the CloudStack database and
>> their hypervisors.  Another is the example provided below is the rapid
>> create-destroy which can (and often does) leave dangling resources due to
>> race conditions between the two operations.
>>
>> In order to provide reliability, CloudStack vertically partitions the
>> infrastructure into zones (independent power source/network uplink
>> combination) sub-divided into pods (racks).  At this time, regions are
>> largely notional, as such, as are not partitions at this time.  Between the
>> user’s zone selection and our allocators distribution of resources across
>> pods, the system attempts to distribute resources widely as possible across
>> these partitions to provide resilience against a variety infrastructure
>> failures (e.g. power loss, network uplink disruption, switch failures,
>> etc).  In order maximize this resilience, the control plane (orchestration
>> + automation tiers) must be to operate on all available partitions.  For
>> example, if we have two (2) zones (A & B) and twenty (20) pods per zone, we
>> should be able to take and execute work in Zone A when one or more pods is
>> lost, as well as, when taking and executing work in Zone B when Zone B has
>> failed.
>>
>> CloudStack is an eventually consistent system in that the state reflected
>> in the orchestration tier will (optimistically) differ from the state of
>> the underlying infrastructure (managed by the automation tier).
>>  Furthermore, the system has a partitioning model to provide resilience in
>> the face of a variety of logical and physical failures.  However, the
>> automation control tier requires strictly consistent operations.  Based on
>> these definitions, the system appears to violate the CAP theorem [2]
>> (Brewer!).  The separation of the system into two distinct tiers isolates
>> these characteristics, but the boundary between them must be carefully
>> implemented to ensure that the consistency requirements of the automation
>> tier are not leaked to the orchestration tier.
>>
>> To properly implement this boundary, I think we should split the
>> orchestration and automation control tiers into separate physical processes
>> communicating via an RPC mechanism — allowing the automation control tier
>> to completely encapsulate its work distribution model.  In my mind, the
>> tricky wicket is providing serialization and partition tolerance in the
>> automation control tier.  Realistically, there two options — explicit and
>&

Re: Resource Management/Locking [was: Re: What would be your ideal solution?]

[Darren Shepherd]

ation for proper collection of instrumentation information and process
> control/monitoring across data centers.
>
> Admittedly, I have hand waved some significant issues that would beed to
> be resolved.  I believe they are all resolvable, but it would take
> discussion to determine the best approach to them.  Transforming CloudStack
> to such a model would not be trivial, but I believe it would be worth the
> (significant) effort as it would make CloudStack one of the most scalable
> and resilient cloud orchestration/management platforms available.
>
> Thanks,
> -John
>
> [1]:
> http://www.slideshare.net/JohnBurwell1/how-to-run-from-a-zombie-cloud-stack-distributed-process-management
> [2]: http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf
> [3]: http://akka.io
> [4]: https://github.com/puniverse/quasar
> [5]: https://github.com/trifork/erjang/wiki
>
> P.S.  I have CC’ed the developer mailing list.  All conversations at this
> level of detail should be initiated and occur on the mailing list to ensure
> transparency with the community.
>
> On Nov 22, 2013, at 3:49 PM, Darren Shepherd 
> wrote:
>
> 140 characters are not productive.
>
> What would be your idea way to do distributed concurrency control?  Simple
> use case.  Server 1 receives a request to start a VM 1.  Server 2 receives
> a request to delete VM 1.  What do you do?
>
> Darren
>
>
>

Re: persistence layer

[Darren Shepherd]

Going to an ORM is not as simple as you would expect.  First, one can make a 
strong argument that ORM is not the right solution, but that can be ignored 
right now.  

You have to look at the context of ACS and figure out what technology is the 
most practical to adopt.  ACS does not have ORM today.  It has a custom query 
api, object mapping, and change tracking for simple CRUD.   Honestly these 
features are quite sufficient for ACS needs.  The problem, and why we should 
change it, is that the current framework is custom, limited in functionality, 
undocumented, and generally a barrier to people developing on ACS.  So jOOQ is 
a somewhat similar approach but it is just far far better, has a community of 
users that have developed over 3-4 years, is well documented, and honestly just 
a very well thought out framework.  

Darren

> On Nov 22, 2013, at 6:50 PM, Alex Ough  wrote:
> 
> All,
> 
> I'm very interested in converting the current DAO framework to an ORM. I
> didn't have any experience with java related ORMs, but I've done quite lots
> of works with Django and LINQ. So can you add me if this project is started?
> 
> Thanks
> Alex Ough
> 
> 
> On Fri, Nov 22, 2013 at 7:06 AM, Daan Hoogland wrote:
> 
>> Had a quick look, It looks alright. One question/doubt: will we thigh
>> ourselves more to mysql if we code sql more directly instead of
>> abstracting away from it so we can leave db choice to the operator in
>> the future!?!?
>> 
>> On Thu, Nov 21, 2013 at 7:03 AM, Darren Shepherd
>>  wrote:
>>> I've done a lot of analysis on the data access layer, but just haven't
>> had time to put together a discuss/recommendation.  In the end I'd propose
>> we move to jOOQ.  It's an excellent framework that will be very natural to
>> the style of data access that CloudStack uses and we can slowly migrate to
>> it.  I've hacked up some code and proven that I can get the two frameworks
>> to seamlessly interoperate.  So you can select from a custom DAO and commit
>> with jOOQ or vice versa.  Additionally jOOQ will work with the existing
>> pojos we have today.
>>> 
>>> Check out jOOQ and let me know what you think of it.  I know for most
>> people the immediate thought would be to move to JPA, but the way we
>> managed "session" is completely incompatible with JPA and will require
>> constant merging.  Additionally mixing our custom DAO framework with a JPA
>> solution looks darn near impossible.
>>> 
>>> Darren
>>> 
>>>> On Nov 11, 2013, at 8:33 PM, Laszlo Hornyak 
>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> What are the general directions with the persistence system?
>>>> What I know about it is:
>>>> - It works with JPA (javax.persistence) annotations
>>>> - But rather than integrating a general JPA implementation such us
>>>> hibernate, eclipselink or OpenJPA it uses its own query generator and
>> DAO
>>>> classes to generate SQL statements.
>>>> 
>>>> Questions:
>>>> - Are you planing to use JPA? What is the motivation behind the custom
>> DAO
>>>> system?
>>>> - There are some capabilities in the DAO system that are not used.
>> Should
>>>> these capabilities be maintained or is it ok to remove the support for
>>>> unused features in small steps?
>>>> 
>>>> --
>>>> 
>>>> EOF
>> 
>> 

Re: persistence layer

[Darren Shepherd]

Jooq does a good job of handling the differences in dialects so as long as you 
don't use a feature specific to a database your are fine.  More info at 
http://www.jooq.org/doc/3.2/manual/sql-building/dsl-context/sql-dialects/

I've used jooq across databases and it's been fine.  The sql we use in ACS is 
quite simple so supporting multiple DBs will be fine. 

Darren

> On Nov 22, 2013, at 2:06 PM, Daan Hoogland  wrote:
> 
> Had a quick look, It looks alright. One question/doubt: will we thigh
> ourselves more to mysql if we code sql more directly instead of
> abstracting away from it so we can leave db choice to the operator in
> the future!?!?
> 
> On Thu, Nov 21, 2013 at 7:03 AM, Darren Shepherd
>  wrote:
>> I've done a lot of analysis on the data access layer, but just haven't had 
>> time to put together a discuss/recommendation.  In the end I'd propose we 
>> move to jOOQ.  It's an excellent framework that will be very natural to the 
>> style of data access that CloudStack uses and we can slowly migrate to it.  
>> I've hacked up some code and proven that I can get the two frameworks to 
>> seamlessly interoperate.  So you can select from a custom DAO and commit 
>> with jOOQ or vice versa.  Additionally jOOQ will work with the existing 
>> pojos we have today.
>> 
>> Check out jOOQ and let me know what you think of it.  I know for most people 
>> the immediate thought would be to move to JPA, but the way we managed 
>> "session" is completely incompatible with JPA and will require constant 
>> merging.  Additionally mixing our custom DAO framework with a JPA solution 
>> looks darn near impossible.
>> 
>> Darren
>> 
>>> On Nov 11, 2013, at 8:33 PM, Laszlo Hornyak  
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> What are the general directions with the persistence system?
>>> What I know about it is:
>>> - It works with JPA (javax.persistence) annotations
>>> - But rather than integrating a general JPA implementation such us
>>> hibernate, eclipselink or OpenJPA it uses its own query generator and DAO
>>> classes to generate SQL statements.
>>> 
>>> Questions:
>>> - Are you planing to use JPA? What is the motivation behind the custom DAO
>>> system?
>>> - There are some capabilities in the DAO system that are not used. Should
>>> these capabilities be maintained or is it ok to remove the support for
>>> unused features in small steps?
>>> 
>>> --
>>> 
>>> EOF

Re: persistence layer

[Darren Shepherd]

I've done a lot of analysis on the data access layer, but just haven't had time 
to put together a discuss/recommendation.  In the end I'd propose we move to 
jOOQ.  It's an excellent framework that will be very natural to the style of 
data access that CloudStack uses and we can slowly migrate to it.  I've hacked 
up some code and proven that I can get the two frameworks to seamlessly 
interoperate.  So you can select from a custom DAO and commit with jOOQ or vice 
versa.  Additionally jOOQ will work with the existing pojos we have today.  

Check out jOOQ and let me know what you think of it.  I know for most people 
the immediate thought would be to move to JPA, but the way we managed "session" 
is completely incompatible with JPA and will require constant merging.  
Additionally mixing our custom DAO framework with a JPA solution looks darn 
near impossible. 

Darren

> On Nov 11, 2013, at 8:33 PM, Laszlo Hornyak  wrote:
> 
> Hi,
> 
> What are the general directions with the persistence system?
> What I know about it is:
> - It works with JPA (javax.persistence) annotations
> - But rather than integrating a general JPA implementation such us
> hibernate, eclipselink or OpenJPA it uses its own query generator and DAO
> classes to generate SQL statements.
> 
> Questions:
> - Are you planing to use JPA? What is the motivation behind the custom DAO
> system?
> - There are some capabilities in the DAO system that are not used. Should
> these capabilities be maintained or is it ok to remove the support for
> unused features in small steps?
> 
> -- 
> 
> EOF

Re: Review Request 14605: Fill the creationMonitors based on priority

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14605/#review28546
---

Ship it!


Laszlo can you commit this?

- Darren Shepherd


On Oct. 11, 2013, 7:14 p.m., Laszlo Hornyak wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14605/
> ---
> 
> (Updated Oct. 11, 2013, 7:14 p.m.)
> 
> 
> Review request for cloudstack and Darren Shepherd.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> If not priotity, append to the end of the list.
> 
> 
> Diffs
> -
> 
>   engine/orchestration/src/com/cloud/agent/manager/AgentManagerImpl.java 
> 4202015 
> 
> Diff: https://reviews.apache.org/r/14605/diff/
> 
> 
> Testing
> ---
> 
> not really, just guessing this is how it was meant (please review)
> 
> 
> Thanks,
> 
> Laszlo Hornyak
> 
>

Re: Review Request 15344: Enabling DB HA for cloud stack data bases using mysql active-active solution

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15344/#review28545
---


I don't agree with this approach.  Following this logic for every possible 
option one would want to add to the jdbc url we would need to write code to 
manage it.  I would prefer that we just provide a way for the operator to fully 
override the URL.  So they just add "db.cloud.url=jdbc:.." to the db.properties 
and that will override the code generated url

- Darren Shepherd


On Nov. 8, 2013, 8:48 a.m., Damodar Reddy Talakanti wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15344/
> ---
> 
> (Updated Nov. 8, 2013, 8:48 a.m.)
> 
> 
> Review request for cloudstack, Abhinandan Prateek and Darren Shepherd.
> 
> 
> Bugs: https://issues.apache.org/jira/browse/CLOUDSTACK-4916
> 
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/CLOUDSTACK-4916
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> To enable Data base High Availability for cloud stack management server 
> database. We have used Mysql's connector parameters to switch between master 
> and slaves along with mysql's 2-way replication and chain way of replication 
> for more than 2 nodes
> 
> https://wikis.oracle.com/display/CommSuite/MySQL+High+Availability+and+Replication+Information+For+Calendar+Server#MySQLHighAvailabilityandReplicationInformationForCalendarServer-MySQLServerTwoWayReplicationExample
> 
> Due ot this defect http://bugs.mysql.com/bug.php?id=68141 changed 2 files to 
> have synchronized for on block of statements which were cauing Thread 
> BLOCKING..
> 
> 
> Diffs
> -
> 
>   client/tomcatconf/db.properties.in b0cb22b 
>   framework/db/src/com/cloud/utils/db/ConnectionConcierge.java acb9cc6 
>   framework/db/src/com/cloud/utils/db/Merovingian2.java 8544aab 
>   framework/db/src/com/cloud/utils/db/TransactionLegacy.java a874ce1 
> 
> Diff: https://reviews.apache.org/r/15344/diff/
> 
> 
> Testing
> ---
> 
> Basic sanity testing is done using 2 nodes. If any transaction(api request) 
> is in between and if server goes down then that api request will fail
> 
> 
> Thanks,
> 
> Damodar Reddy Talakanti
> 
>

Re: ComponentContext :: getComponent() ?

[Darren Shepherd]

Not really.  There is a away but it's intended for only framework code.  The 
API framework uses it.  For regular business logic, I've yet to see any good 
reason on why you need to look up things in the application context.  There is 
typically a cleaner way.

Darren

> On Nov 6, 2013, at 2:19 PM, Prachi Damle  wrote:
> 
> But in general, is there a way to get hold of a particular bean by name or 
> type?
> 
> -Original Message-
> From: Prachi Damle 
> Sent: Wednesday, November 06, 2013 1:19 PM
> To: dev@cloudstack.apache.org
> Subject: RE: ComponentContext :: getComponent() ?
> 
> I used to call getComponent() to get the current list of 
> AffinityGroupProcessors enabled by the admin. The injected processor list was 
> added later to the class, I think I can use that for this purpose.
> 
> Thanks,
> Prachi
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Tuesday, November 05, 2013 10:09 PM
> To: dev@cloudstack.apache.org
> Subject: Re: ComponentContext :: getComponent() ?
> 
> All other uses of getComponent() you listed should work fine.
> 
> In org.apache.cloudstack.affinity.AffinityGroupServiceImpl, why do you even 
> call getComponent()?  You already have a list of AffinityGroupProcessors 
> injected to the class.  Can't you just loop through those?
> 
> Darren
> 
> 
> On Tue, Nov 5, 2013 at 5:49 PM, Prachi Damle wrote:
> 
>> Thanks Darren, it is the listAffinityGroupTypes API, specific code is 
>> in AffinityGroupServiceImpl :: listAffinityGroupTypes() Code lists the 
>> beans implementing 'AffinityGroupProcessor' adapter interface.
>> 
>> 
>> I also dug up few other callers that rely on ComponentContext ::
>> getComponent()...
>> 
>> com.cloud.baremetal.networkservice.BareMetalResourceBase.fullSync()
>> com.cloud.network.NetworkStateListener.pubishOnEventBus(String,
>> String, Network, State, State) 
>> com.cloud.storage.listener.SnapshotStateListener.pubishOnEventBus(Stri
>> ng,
>> String, Snapshot, State, State)
>> com.cloud.vm.UserVmStateListener.pubishOnEventBus(String, String, 
>> VirtualMachine, State, State) 
>> com.cloud.storage.listener.VolumeStateListener.pubishOnEventBus(String
>> ,
>> String, Volume, State, State)
>> com.cloud.event.AlertGenerator.publishAlertOnEventBus(String, long, 
>> Long, String, String) 
>> com.cloud.event.ActionEventUtils.publishOnEventBus(long, long, String, 
>> String, State, String) 
>> com.cloud.event.UsageEventUtils.publishUsageEvent(String, Long, Long, 
>> String, String)
>> 
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Tuesday, November 05, 2013 4:40 PM
>> To: dev@cloudstack.apache.org
>> Cc: dev@cloudstack.apache.org
>> Subject: Re: ComponentContext :: getComponent() ?
>> 
>> Can you point me to the code that is failing.  The short of it is that 
>> you shouldn't use ComponentContext.  If you show me the failing code I 
>> can offer a better solution or if needed a work around.
>> 
>> Darren
>> 
>>>> On Nov 5, 2013, at 4:18 PM, Prachi Damle 
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> Before the Spring modularization, we have some usecases that were 
>>> using
>> the com.cloud.utils.component.ComponentContext class directly to 
>> inject or to find a bean of given type or name by invoking ComponentContext  
>> ::
>> getComponent().
>>> How should these usecases be ported now after the modularization
>> changes? What should we use instead of ComponentContext, to say find 
>> beans of certain type.
>>> 
>>> This is the root cause for bug
>> https://issues.apache.org/jira/browse/CLOUDSTACK-5045 as the code is 
>> still relying on ComponentContext
>>> 
>>> Thanks,
>>> Prachi
>> 

Re: ComponentContext :: getComponent() ?

[Darren Shepherd]

All other uses of getComponent() you listed should work fine.

In org.apache.cloudstack.affinity.AffinityGroupServiceImpl, why do you even
call getComponent()?  You already have a list of AffinityGroupProcessors
injected to the class.  Can't you just loop through those?

Darren


On Tue, Nov 5, 2013 at 5:49 PM, Prachi Damle wrote:

> Thanks Darren, it is the listAffinityGroupTypes API, specific code is in
> AffinityGroupServiceImpl :: listAffinityGroupTypes()
> Code lists the beans implementing 'AffinityGroupProcessor' adapter
> interface.
>
>
> I also dug up few other callers that rely on ComponentContext ::
> getComponent()...
>
> com.cloud.baremetal.networkservice.BareMetalResourceBase.fullSync()
> com.cloud.network.NetworkStateListener.pubishOnEventBus(String, String,
> Network, State, State)
> com.cloud.storage.listener.SnapshotStateListener.pubishOnEventBus(String,
> String, Snapshot, State, State)
> com.cloud.vm.UserVmStateListener.pubishOnEventBus(String, String,
> VirtualMachine, State, State)
> com.cloud.storage.listener.VolumeStateListener.pubishOnEventBus(String,
> String, Volume, State, State)
> com.cloud.event.AlertGenerator.publishAlertOnEventBus(String, long, Long,
> String, String)
> com.cloud.event.ActionEventUtils.publishOnEventBus(long, long, String,
> String, State, String)
> com.cloud.event.UsageEventUtils.publishUsageEvent(String, Long, Long,
> String, String)
>
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Tuesday, November 05, 2013 4:40 PM
> To: dev@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: ComponentContext :: getComponent() ?
>
> Can you point me to the code that is failing.  The short of it is that you
> shouldn't use ComponentContext.  If you show me the failing code I can
> offer a better solution or if needed a work around.
>
> Darren
>
> > On Nov 5, 2013, at 4:18 PM, Prachi Damle 
> wrote:
> >
> > Hi,
> >
> > Before the Spring modularization, we have some usecases that were using
> the com.cloud.utils.component.ComponentContext class directly to inject or
> to find a bean of given type or name by invoking ComponentContext  ::
> getComponent().
> > How should these usecases be ported now after the modularization
> changes? What should we use instead of ComponentContext, to say find beans
> of certain type.
> >
> > This is the root cause for bug
> https://issues.apache.org/jira/browse/CLOUDSTACK-5045 as the code is
> still relying on ComponentContext
> >
> > Thanks,
> > Prachi
>

Re: ComponentContext :: getComponent() ?

[Darren Shepherd]

Can you point me to the code that is failing.  The short of it is that you 
shouldn't use ComponentContext.  If you show me the failing code I can offer a 
better solution or if needed a work around.  

Darren

> On Nov 5, 2013, at 4:18 PM, Prachi Damle  wrote:
> 
> Hi,
> 
> Before the Spring modularization, we have some usecases that were using the 
> com.cloud.utils.component.ComponentContext class directly to inject or to 
> find a bean of given type or name by invoking ComponentContext  :: 
> getComponent().
> How should these usecases be ported now after the modularization changes? 
> What should we use instead of ComponentContext, to say find beans of certain 
> type.
> 
> This is the root cause for bug 
> https://issues.apache.org/jira/browse/CLOUDSTACK-5045 as the code is still 
> relying on ComponentContext
> 
> Thanks,
> Prachi

Re: does master even work? Can't start SSVM/CP

[Darren Shepherd]

Yep, thanks.  Its working now.

Darren


On Mon, Nov 4, 2013 at 4:40 PM, Min Chen  wrote:

> If you are not using S3 as secondary storage, it should work. For S3, it
> still requires mount on MS to start ssvm though to copy from S3 to NFS
> cache.
>
> Thanks
> -min
>
> On 11/4/13 3:38 PM, "Min Chen"  wrote:
>
> >The last commit is eb9cfffd246267bbb5fe4befeb9afeaaf366874a
> >
> >Thanks
> >-min
> >
> >On 11/4/13 3:36 PM, "Darren Shepherd" 
> wrote:
> >
> >>What is the commit?
> >>
> >>Darren
> >>
> >>
> >>On Mon, Nov 4, 2013 at 4:35 PM, Min Chen  wrote:
> >>
> >>> Darren, your issue should be resolved. I checked in a fix for that.
> >>>Also
> >>> tested and able to start SSVM using Xen and NFS (or S3).
> >>>
> >>> Thanks
> >>> -min
> >>>
> >>> From: Mike Tutkowski  >>> mike.tutkow...@solidfire.com>>
> >>> Date: Monday, November 4, 2013 3:33 PM
> >>> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> >>> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> >>> Cc: Min Chen mailto:min.c...@citrix.com>>, Edison
> >>>Su
> >>> mailto:edison...@citrix.com>>
> >>> Subject: Re: does master even work? Can't start SSVM/CP
> >>>
> >>> For what it's worth, I rebased on top of master today and was able to
> >>> create an environment from scratch using a XenServer host.
> >>>
> >>>
> >>> On Mon, Nov 4, 2013 at 4:30 PM, Darren Shepherd <
> >>> darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>>
> wrote:
> >>> I can't get an environment setup on master, basically everytime it goes
> >>>to
> >>> start the SSVM or CP it tries to stage the template, then that fails
> >>> because the command is routed to LocalHostEndpoint and my mgmt server
> >>>can't
> >>> mount NFS.  Isn't there some special logic somewhere that knows the
> >>>system
> >>> VM template is already pre-staged on NFS?
> >>>
> >>> Darren
> >>>
> >>>
> >>>
> >>> --
> >>> Mike Tutkowski
> >>> Senior CloudStack Developer, SolidFire Inc.
> >>> e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com>
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the cloud<
> >>> http://solidfire.com/solution/overview/?video=play>
> >>>
> >
>
>

Re: does master even work? Can't start SSVM/CP

[Darren Shepherd]

What is the commit?

Darren


On Mon, Nov 4, 2013 at 4:35 PM, Min Chen  wrote:

> Darren, your issue should be resolved. I checked in a fix for that. Also
> tested and able to start SSVM using Xen and NFS (or S3).
>
> Thanks
> -min
>
> From: Mike Tutkowski  mike.tutkow...@solidfire.com>>
> Date: Monday, November 4, 2013 3:33 PM
> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> Cc: Min Chen mailto:min.c...@citrix.com>>, Edison Su
> mailto:edison...@citrix.com>>
> Subject: Re: does master even work? Can't start SSVM/CP
>
> For what it's worth, I rebased on top of master today and was able to
> create an environment from scratch using a XenServer host.
>
>
> On Mon, Nov 4, 2013 at 4:30 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com<mailto:darren.s.sheph...@gmail.com>> wrote:
> I can't get an environment setup on master, basically everytime it goes to
> start the SSVM or CP it tries to stage the template, then that fails
> because the command is routed to LocalHostEndpoint and my mgmt server can't
> mount NFS.  Isn't there some special logic somewhere that knows the system
> VM template is already pre-staged on NFS?
>
> Darren
>
>
>
> --
> Mike Tutkowski
> Senior CloudStack Developer, SolidFire Inc.
> e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com>
> o: 303.746.7302
> Advancing the way the world uses the cloud<
> http://solidfire.com/solution/overview/?video=play>™
>

Re: does master even work? Can't start SSVM/CP

[Darren Shepherd]

The big thing that is different about my environment is that my mgmt server
can't mount NFS.  Otherwise it would work.

Darren


On Mon, Nov 4, 2013 at 4:33 PM, Mike Tutkowski  wrote:

> For what it's worth, I rebased on top of master today and was able to
> create an environment from scratch using a XenServer host.
>
>
> On Mon, Nov 4, 2013 at 4:30 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com
> > wrote:
>
> > I can't get an environment setup on master, basically everytime it goes
> to
> > start the SSVM or CP it tries to stage the template, then that fails
> > because the command is routed to LocalHostEndpoint and my mgmt server
> can't
> > mount NFS.  Isn't there some special logic somewhere that knows the
> system
> > VM template is already pre-staged on NFS?
> >
> > Darren
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>

does master even work? Can't start SSVM/CP

[Darren Shepherd]

I can't get an environment setup on master, basically everytime it goes to
start the SSVM or CP it tries to stage the template, then that fails
because the command is routed to LocalHostEndpoint and my mgmt server can't
mount NFS.  Isn't there some special logic somewhere that knows the system
VM template is already pre-staged on NFS?

Darren

opencontrail in master

[Darren Shepherd]

I just noticed that opencontrail is in master.  There is some small work
that needs to be done to make
org.apache.cloudstack.network.contrail.management.EvenUtils.EventInterceptor
work in the new Spring model.  Is somebody working on this?  Additionally
the new spring config needs to be setup for that plugin too.  Let me know
if you need help.

Darren

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

Yeah, I see there was a commit a3cec3802c4c56173c33ccc1e08af059547fe90e
that broke it again.  I'll reverted that commit and ran the following to
sanity test and its seems to be working now

mvn -T 1.5C -DskipTests clean install
chmod +x
plugins/network-elements/juniper-contrail/test/resources/mysql_db_start.sh
plugins/network-elements/juniper-contrail/test/resources/mysql_db_start.sh
5000
sed s/3306/5000/g utils/conf/db.properties >
utils/conf/db.properties.override
mvn -P developer  -pl developer -Ddeploydb

Make sure you run "mvn clean install" once after you update.

Darren


On Fri, Nov 1, 2013 at 6:34 PM, Pedro Roque Marques <
pedro.r.marq...@gmail.com> wrote:

> Darren,
>
> On Oct 31, 2013, at 9:27 AM, Darren Shepherd 
> wrote:
>
> > This is fixed now.
>
> Maybe i'm doing something really dumb... but i still see the problem.
> I do see the commit afaf637 in the branch history.
>
> Steps to reproduce:
> 1.
> plugins/network-elements/juniper-contrail/test/resources/mysql_db_start.sh
> 5
> 2 copy utils/conf/db.properties to db.properties.override
> 3 modify
>
> # CloudStack database settings
> db.cloud.username=${USER}
> db.cloud.password=
> db.root.password=
> db.cloud.host=localhost
> db.cloud.port=5
> db.cloud.name=cloud
>
> 4. mvn -P developer  -pl developer -Ddeploydb
> > Initializing database=cloud with host=localhost port=3306
> username=cloud password=cloud
>
> It should display port=5 and username=${USER)
>
>
> I'll dig into this further... most likely it is being caused by something
> really stupid i'm doing. But anyway i thought i would suggest a way to
> start a local mysql as a way to test. The contrail plugin test harness is
> allocating a random port and then trying to create a DB that is used for
> the test.
>
>   Pedro.

Re: parallel mvn builds

[Darren Shepherd]

I use it all the time with -T 1.5C and it makes my build twice as fast.
But I do it with the assumption that "it may void your warranty."
Sometimes things blow up, and then I go back to single threaded.  So it
kinda works, but it does blow up.

I wouldn't recommend most people use it.  For people who are closer to full
time ACS developers it can be helpful.  For people that don't use ACS much,
its not worth the hassle because its not guaranteed to work and I have seen
it fail.

Darren


On Thu, Oct 31, 2013 at 3:36 PM, John Kinsella  wrote:

> I've been tinkering with adding -T to builds on my laptop - e.g. "mvn -T
> 2C install" (Means run two build threads on each CPU core the system has)
>
> I'm seeing a 20-30% improvement in build time vs a single-thread build,
> and things seem to work so far, but I do get this warning a few times
> during the build:
>
> [WARNING] *
> [WARNING] * Your build is requesting parallel execution, but project  *
> [WARNING] * contains the following plugin(s) that are not marked as   *
> [WARNING] * @threadSafe to support parallel building. *
> [WARNING] * While this /may/ work fine, please look for plugin updates*
> [WARNING] * and/or request plugins be made thread-safe.   *
> [WARNING] * If reporting an issue, report it against the plugin in*
> [WARNING] * question, not against maven-core  *
> [WARNING] *
> [WARNING] The following plugins are not marked @threadSafe in Apache
> CloudStack:
> [WARNING] org.apache.maven.plugins:maven-site-plugin:3.1
> [WARNING] *
>
> Curious if anybody else has tried this or if we should recommend it to
> folks?
>
>

Re: Review Request 15108: Grab MacAddress in a more deterministic fashion

[Darren Shepherd]

> On Oct. 31, 2013, 8:29 a.m., Laszlo Hornyak wrote:
> > I haven't had a chance to test it yet, but I prefer this solution over my 
> > own patchset, since I was only trying to fix some broken functionality 
> > without radically changing the behavior.
> > One minor problem I see in there is that still lots of code in static 
> > initializer block, it is hard to write unit tests for it. I think we can 
> > solve that later.

I can't remove the static initialization atm but we can move the contents of 
the static block to a method findDefaultMacAddress that returns the MacAddress. 
 Then the static block would just be the assignment.  That should help with 
unit testing. I wasn't too sure how to unit test this code.  If you want to 
move around the code and see if you can write some tests that would be great.


- Darren


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15108/#review27875
-------


On Oct. 31, 2013, 12:18 a.m., Darren Shepherd wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15108/
> ---
> 
> (Updated Oct. 31, 2013, 12:18 a.m.)
> 
> 
> Review request for cloudstack, Alex Huang and Laszlo Hornyak.
> 
> 
> Repository: cloudstack-git
> 
> 
> Description
> ---
> 
> The MAC address is determined in the following order
> 
> 1. Look for MAC that matches interface of cluster.node.IP
> 2. Look for first interface alphabetically that has a non-linklocal address
> 3. Look for first interface alphabetically
> 4. Randomly generate a mac address
> 
> 
> Diffs
> -
> 
>   utils/src/com/cloud/utils/net/MacAddress.java 15350c8 
> 
> Diff: https://reviews.apache.org/r/15108/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Darren Shepherd
> 
>

Re: ACS + Riak-CS, works?

[Darren Shepherd]

Memory is 4gb and disk is about 1gb free (kinda low on disk space).  But the 
system is overtaxed and slow and swapping a bit.  I'll give the vagrant setup a 
try.  

Darren

> On Oct 31, 2013, at 11:29 AM, John Burwell  wrote:
> 
> Darren,
> 
> One silly question, how much disk space is free in the container?  
> Additionally, how much RAM have allocated?  I recommend a minimum of 2 GB for 
> testing (memory is consumed largely by the bitcask backend holding keys in 
> memory).  When disk space or memory runs out, odd errors can result.  
> 
> If you can, I would highly suggest given the vagrant project a go (increase 
> the memory in vagrant-overrides.conf to 2 GB).  In addition to be a known 
> good configuration, it is a common reference point for support.
> 
> Thanks,
> -John
> 
>> On Oct 31, 2013, at 2:14 PM, Darren Shepherd  
>> wrote:
>> 
>> This is actually running in docker and I followed the quickstart [1] to set 
>> it up.  I'll send the config tonight, I don't have access to the machine at 
>> the moment.  I know the 403 seems to indicate the setup is wrong, but all 
>> parts prior to part 18 got a 200 (as you can see in the access log).  So it 
>> would seem that the config is right.  I just don't get why it would randomly 
>> send a 403.  But based on the error message it would seem like it lost 
>> connection to riak to get to user info.
>> 
>> Darren
>> 
>> [1] http://docs.basho.com/riakcs/latest/tutorials/quick-start-riak-cs/
>> 
>>> On Thu, Oct 31, 2013 at 10:48 AM, John Burwell  wrote:
>>> Darren,
>>> 
>>> Your are receiving a 403 which indicates that the ACLs/user is not properly 
>>> configured.  Which version of Riak CS are you using?  Can you send your 
>>> app.config and vm.args for Riak and Riak CS?  Personally, I use the 
>>> vagrant-riak-cs-cluster [1] project to build local instances for 
>>> development and testing.  
>>> 
>>> Thanks,
>>> -John
>>> 
>>> [1]: https://github.com/basho/vagrant-riak-cs-cluster
>>> 
>>> 
>>>> On Oct 31, 2013, at 1:23 PM, Darren Shepherd  
>>>> wrote:
>>>> 
>>>> My template is less than 5gb and it still uses multipart.  I actually 
>>>> don't think this is a CloudStack issue but more of an issue with Riak-CS.  
>>>> I was just wondering if anybody has specifically ran the setup or knows 
>>>> more about Riak-CS and can tell me whats up.  I've been trying off and on 
>>>> for the last week to try to get this working with 4.2 but it just keeps 
>>>> failing.  I think I'll just try ceph rados gateway if I can't get this 
>>>> working.
>>>> 
>>>> Darren
>>>> 
>>>> 
>>>>> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu 
>>>>>  wrote:
>>>>> Hi,
>>>>> 
>>>>> Cloudstack with Riak-CS works fine. If we register template more than 5GB 
>>>>> we use multipart upload to push the files to Riak. But this does not work 
>>>>> for snapshots and volumes. This will be fixed in CS4.2.1 . But for 
>>>>> template registration it should work fine.
>>>>> 
>>>>> Thanks,
>>>>> Sanjeev
>>>>> 
>>>>> -Original Message-
>>>>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>>>>> Sent: Thursday, October 31, 2013 11:18 AM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: ACS + Riak-CS, works?
>>>>> 
>>>>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter 
>>>>> what I try when it goes to download and save the template, after its 
>>>>> posted a series of multi part uploads it ends up sending a 403.  The 
>>>>> access log looks like
>>>>> 
>>>>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>>>>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>>>>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic 
>>>>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>>>>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>>>>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>>>

Re: why system.vm.use.local.storage?

[Darren Shepherd]

I just find it annoying setting up a zone with local storage that I have to 
turn this setting on.  If I have no shared storage, then system VMs won't 
deploy with out this parameter.  It's just seems like a useless setting that 
might have been added because there is something else in the system that isn't 
working right.

Darren

> On Oct 31, 2013, at 10:52 AM, Kelcey Jamison Damage 
>  wrote:
> 
> That's a good point. I know the system VMs are auto created if the system is 
> aware of their state being either 'Destroyed', 'Expunged'. If the state is 
> 'Starting', or 'Started'/'Running' then no actions are taken. 
> 
> In your scenario, the system VM would still be 'Started' in the database even 
> though communication is lost to the host, so no automatic action would be 
> taken. 
> 
> - Original Message -
> 
> From: "Darren Shepherd"  
> To: dev@cloudstack.apache.org 
> Sent: Thursday, October 31, 2013 10:47:41 AM 
> Subject: why system.vm.use.local.storage? 
> 
> Why do we have the configuration system.vm.use.local.storage? Why would 
> somebody care if they systemvm is on local storage? I'm guessing the idea 
> here is if it's on local storage we can't bring it back in a host failure 
> situation? But don't we just recreate the systemvm? 
> 
> Darren 
> 

race conditions in VolumeServiceImpl.createBaseImageAsync() creates NPE

[Darren Shepherd]

The following code results in a NPE in bad situations

templatePoolRef =
_tmpltPoolDao.acquireInLockTable(templatePoolRefId,
storagePoolMaxWaitSeconds);

if (templatePoolRef == null) {
if (s_logger.isDebugEnabled()) {
s_logger.info("Unable to acquire lock on
VMTemplateStoragePool " + templatePoolRefId);
}
templatePoolRef =
_tmpltPoolDao.findByPoolTemplate(dataStore.getId(), template.getId());
if (templatePoolRef.getState() ==
ObjectInDataStoreStateMachine.State.Ready ) {
s_logger.info("Unable to acquire lock on
VMTemplateStoragePool " + templatePoolRefId + ", But Template " +
template.getUniqueName() + " is already copied to primary storage, skip
copying");
createVolumeFromBaseImageAsync(volume,
templateOnPrimaryStoreObj, dataStore, future);
return;
}
throw new CloudRuntimeException("Unable to acquire lock on
VMTemplateStoragePool: " + templatePoolRefId);
}

If two threads are trying to stage the same template thread one gets the
lock, thread two will wait.  If thread one fails to stage the template it
will delete the templatePoolRef from the database.  Thread two will now get
the lock in op_lock, but the internal findById will not find a
templatePoolRef because it has been deleted and return null from
acquireInLockTable().  Technically thread two has the lock, but the ref
templatePoolRef wasn't found.  The subsequent line "templatePoolRef =
_tmpltPoolDao.findByPoolTemplate(...)" will return null, because it doesn't
exist and then on the next line templatePoolRef.getState() will throw a NPE.

Darren

Re: ACS + Riak-CS, works?

[Darren Shepherd]

This is actually running in docker and I followed the quickstart [1] to set
it up.  I'll send the config tonight, I don't have access to the machine at
the moment.  I know the 403 seems to indicate the setup is wrong, but all
parts prior to part 18 got a 200 (as you can see in the access log).  So it
would seem that the config is right.  I just don't get why it would
randomly send a 403.  But based on the error message it would seem like it
lost connection to riak to get to user info.

Darren

[1] http://docs.basho.com/riakcs/latest/tutorials/quick-start-riak-cs/

On Thu, Oct 31, 2013 at 10:48 AM, John Burwell  wrote:

> Darren,
>
> Your are receiving a 403 which indicates that the ACLs/user is not
> properly configured.  Which version of Riak CS are you using?  Can you send
> your app.config and vm.args for Riak and Riak CS?  Personally, I use the
> vagrant-riak-cs-cluster [1] project to build local instances for
> development and testing.
>
> Thanks,
> -John
>
> [1]: https://github.com/basho/vagrant-riak-cs-cluster
>
>
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
> wrote:
>
> My template is less than 5gb and it still uses multipart.  I actually
> don't think this is a CloudStack issue but more of an issue with Riak-CS.
> I was just wondering if anybody has specifically ran the setup or knows
> more about Riak-CS and can tell me whats up.  I've been trying off and on
> for the last week to try to get this working with 4.2 but it just keeps
> failing.  I think I'll just try ceph rados gateway if I can't get this
> working.
>
> Darren
>
>
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
> sanjeev.neelar...@citrix.com> wrote:
>
>> Hi,
>>
>> Cloudstack with Riak-CS works fine. If we register template more than 5GB
>> we use multipart upload to push the files to Riak. But this does not work
>> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
>> registration it should work fine.
>>
>> Thanks,
>> Sanjeev
>>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>>
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>> what I try when it goes to download and save the template, after its posted
>> a series of multi part uploads it ends up sending a 403.  The access log
>> looks like
>>
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>>
>> And then in the error log I always get
>>
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
>> record for s3 failed. Reason: disconnected
>>
>> I'm able to interact with Riak-CS with s3cmd just fine but its not using
>> multipart. I think I'm going to try to download a new s3cmd that does
>> multipart and see if even that will work.
>>
>> Darren
>>
>
>
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

Does this issue exist in 4.2/4.2.1?  I'm going to put the bug as blocker,
if you disagree we can lower it.  I really think it is quite a bad scenario
to mount NFS to the mgmt server.

Darren


On Thu, Oct 31, 2013 at 10:48 AM, Min Chen  wrote:

> Delaying them until ssvm is up may not that easy to handle in current
> CloudStack. We may be able to send an error in this case by distinguishing
> S3 register template case from other scenarios. Would you mind filing a
> JIRA bug on this? Your issue seems different from the issue filed by Daan
> (https://issues.apache.org/jira/browse/CLOUDSTACK-4759).
>
> Thanks
> -min
>
>
> On 10/31/13 10:38 AM, "Darren Shepherd" 
> wrote:
>
> >The behaviour should be to delay until there is a SSVM available.  I don't
> >know if that is even possible in CloudStack, I don't want you to sleep().
> >Is there any way to delay?  Something else in the system should be
> >ensuring
> >that a SSVM is running and available (what I don't know...).  If you can't
> >delay, it is far better in my mind to send an error to the user than to
> >try
> >to do the operation on the mgmt server.
> >
> >I think it is very dangerous to have this code that routes to localhost
> >because in a production setup ACS will have the sudo access to
> >mount/mkdir.  But the server may not have network access, so you run the
> >chance of mounting the NFS and having it hang almost indefinitely which
> >can
> >cause system instability for the mgmt server.
> >
> >The specific issue I have right now is that for some reason on master, the
> >allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
> >restart the mgmt stack, on host connect it tries to download templates so
> >it sends a command and it goes to LocalHostEndPoint.
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:
> >
> >> Yes, that is true, it is a bug in that case. The logic of choosing
> >> endpoint to send Command will not work properly for destroying/Stopping
> >>VM
> >> case. Our assumption is that registering template is done when ssvm is
> >>up
> >> and running. Otherwise, it will send to LocalHost to execute command. In
> >> your opinion, what should be expected behavior if user is trying to
> >> register template when ssvm is crashed? Directly reporting error instead
> >> of trying anything.
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 10:18 AM, "Darren Shepherd" 
> >> wrote:
> >>
> >> >It's happening when I first create the zone.  I'll debug it further to
> >> >tell
> >> >you exactly what its happening.  But I don't understand how the
> >>following
> >> >code in DefaultEndpointSelector could be correct.  If I have a zone,
> >>and I
> >> >delete/stop/crash the SSVM, and at the same time register a new
> >>template,
> >> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >> >
> >> >protected EndPoint findEndpointForImageStorage(DataStore store) {
> >> >Long dcId = null;
> >> >Scope storeScope = store.getScope();
> >> >if (storeScope.getScopeType() == ScopeType.ZONE) {
> >> >dcId = storeScope.getScopeId();
> >> >}
> >> >// find ssvm that can be used to download data to store. For
> >> >zone-wide
> >> >// image store, use SSVM for that zone. For region-wide store,
> >> >// we can arbitrarily pick one ssvm to do that task
> >> >List ssAHosts =
> >> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> >> >if (ssAHosts == null || ssAHosts.isEmpty()) {
> >> >s_logger.info("No running ssvm is found, so command will
> be
> >> >sent to LocalHostEndPoint");
> >> >return LocalHostEndpoint.getEndpoint(); // use local host
> >>as
> >> >endpoint in
> >> >// case of no ssvm existing
> >> >}
> >> >Collections.shuffle(ssAHosts);
> >> >HostVO host = ssAHosts.get(0);
> >> >return
> >>RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >> >host.getPrivateIpAddress(),
> >> >host.getPublicIpAddress());
> >> >}
> >> >
> >> >Darren
> >> >
> 

why system.vm.use.local.storage?

[Darren Shepherd]

Why do we have the configuration system.vm.use.local.storage?  Why would
somebody care if they systemvm is on local storage?  I'm guessing the idea
here is if it's on local storage we can't bring it back in a host failure
situation?  But don't we just recreate the systemvm?

Darren

Re: ACS + Riak-CS, works?

[Darren Shepherd]

Uploading without multipart works with s3cmd.  The s3cmd I was using didn't
have multipart support, so I'll download a newer one and try it tonight.

Would you happen to know what the below means?  I get a 200 response for
parts 1,2,3,4,etc and then randomly one part will fail with the below.

riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3
failed. Reason: disconnected

Darren



On Thu, Oct 31, 2013 at 10:30 AM, John Burwell  wrote:

> Darren,
>
> Since S3/Riak CS can receive the contents of a file out of order and
> re-assemble upon completion of transfer of all parts, the underlying driver
> uses the TransferManager which automatically breaks files up into
> multi-part uploads to increase parallelism to decrease the time required to
> upload to the object store.  Therefore, multi-part uploads are not employed
> solely to upload files larger than 5 GB.
>
> Have you tried to upload the file to Riak CS using s3cmd outside of
> CloudStack?
>
> Thanks,
> -John
>
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd 
> wrote:
>
> My template is less than 5gb and it still uses multipart.  I actually
> don't think this is a CloudStack issue but more of an issue with Riak-CS.
> I was just wondering if anybody has specifically ran the setup or knows
> more about Riak-CS and can tell me whats up.  I've been trying off and on
> for the last week to try to get this working with 4.2 but it just keeps
> failing.  I think I'll just try ceph rados gateway if I can't get this
> working.
>
> Darren
>
>
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
> sanjeev.neelar...@citrix.com> wrote:
>
>> Hi,
>>
>> Cloudstack with Riak-CS works fine. If we register template more than 5GB
>> we use multipart upload to push the files to Riak. But this does not work
>> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
>> registration it should work fine.
>>
>> Thanks,
>> Sanjeev
>>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org
>> Subject: ACS + Riak-CS, works?
>>
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>> what I try when it goes to download and save the template, after its posted
>> a series of multi part uploads it ends up sending a 403.  The access log
>> looks like
>>
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>>
>> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>> OpenJDK_64-Bit_Server_VM/20.0-b12"
>>
>> And then in the error log I always get
>>
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
>> record for s3 failed. Reason: disconnected
>>
>> I'm able to interact with Riak-CS with s3cmd just fine but its not using
>> multipart. I think I'm going to try to download a new s3cmd that does
>> multipart and see if even that will work.
>>
>> Darren
>>
>
>
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

The behaviour should be to delay until there is a SSVM available.  I don't
know if that is even possible in CloudStack, I don't want you to sleep().
Is there any way to delay?  Something else in the system should be ensuring
that a SSVM is running and available (what I don't know...).  If you can't
delay, it is far better in my mind to send an error to the user than to try
to do the operation on the mgmt server.

I think it is very dangerous to have this code that routes to localhost
because in a production setup ACS will have the sudo access to
mount/mkdir.  But the server may not have network access, so you run the
chance of mounting the NFS and having it hang almost indefinitely which can
cause system instability for the mgmt server.

The specific issue I have right now is that for some reason on master, the
allocator is not working for me anymore.  So SSVM doesn't deploy.  If I
restart the mgmt stack, on host connect it tries to download templates so
it sends a command and it goes to LocalHostEndPoint.

Darren


On Thu, Oct 31, 2013 at 10:28 AM, Min Chen  wrote:

> Yes, that is true, it is a bug in that case. The logic of choosing
> endpoint to send Command will not work properly for destroying/Stopping VM
> case. Our assumption is that registering template is done when ssvm is up
> and running. Otherwise, it will send to LocalHost to execute command. In
> your opinion, what should be expected behavior if user is trying to
> register template when ssvm is crashed? Directly reporting error instead
> of trying anything.
>
> Thanks
> -min
>
> On 10/31/13 10:18 AM, "Darren Shepherd" 
> wrote:
>
> >It's happening when I first create the zone.  I'll debug it further to
> >tell
> >you exactly what its happening.  But I don't understand how the following
> >code in DefaultEndpointSelector could be correct.  If I have a zone, and I
> >delete/stop/crash the SSVM, and at the same time register a new template,
> >it seems the DownloadCommand will go to LocalHostEndPoint.
> >
> >protected EndPoint findEndpointForImageStorage(DataStore store) {
> >Long dcId = null;
> >Scope storeScope = store.getScope();
> >if (storeScope.getScopeType() == ScopeType.ZONE) {
> >dcId = storeScope.getScopeId();
> >}
> >// find ssvm that can be used to download data to store. For
> >zone-wide
> >// image store, use SSVM for that zone. For region-wide store,
> >// we can arbitrarily pick one ssvm to do that task
> >List ssAHosts =
> >listUpAndConnectingSecondaryStorageVmHost(dcId);
> >if (ssAHosts == null || ssAHosts.isEmpty()) {
> >s_logger.info("No running ssvm is found, so command will be
> >sent to LocalHostEndPoint");
> >return LocalHostEndpoint.getEndpoint(); // use local host as
> >endpoint in
> >// case of no ssvm existing
> >}
> >Collections.shuffle(ssAHosts);
> >HostVO host = ssAHosts.get(0);
> >return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
> >host.getPrivateIpAddress(),
> >host.getPublicIpAddress());
> >}
> >
> >Darren
> >
> >
> >On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:
> >
> >> Hi Darren,
> >>
> >> The logic of sending command to Local in case of no SSVM is up
> >>is
> >> to
> >> support S3 as secondary storage. In case of S3 as secondary storage,
> >> system vm template is automatically downloaded to S3 when S3 is added
> >>into
> >> CloudStack, unlike NFS secondary storage case where we used a script to
> >> pre-populate system vm template. The thing I don't get is: what scenario
> >> are you trying to do to trigger DownloadCommand when SSVM is not up?
> >>
> >> Thanks
> >> -min
> >>
> >> On 10/31/13 9:32 AM, "Darren Shepherd" 
> >> wrote:
> >>
> >> >No, that's not the problem.  The problem is that the DownloadCommand
> >> >is supposed to go to the SSVM, but it sending it to the
> >> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
> >> >not create /var/cloudstack/mnt/
> >> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
> >> >so that it does not have sudo access.  A standard cloudstack install
> >> >has sudo access so it will create those folders and mount on the mgmt
> >> >server, but it shouldn't.
> >> >
> >> >Please don't tell me we made a change so that the mgmt server is
> >> >expected to have NFS access to all secondary storages.
> >> >
> >> >Darren
> >> >
> >> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU 
> >>wrote:
> >> >> Please check if you can
> >> >> create
> >> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> >> and mount nfs point.
> >>
> >>
>
>

Re: ACS + Riak-CS, works?

[Darren Shepherd]

My template is less than 5gb and it still uses multipart.  I actually don't
think this is a CloudStack issue but more of an issue with Riak-CS.  I was
just wondering if anybody has specifically ran the setup or knows more
about Riak-CS and can tell me whats up.  I've been trying off and on for
the last week to try to get this working with 4.2 but it just keeps
failing.  I think I'll just try ceph rados gateway if I can't get this
working.

Darren


On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <
sanjeev.neelar...@citrix.com> wrote:

> Hi,
>
> Cloudstack with Riak-CS works fine. If we register template more than 5GB
> we use multipart upload to push the files to Riak. But this does not work
> for snapshots and volumes. This will be fixed in CS4.2.1 . But for template
> registration it should work fine.
>
> Thanks,
> Sanjeev
>
> -Original Message-
> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org
> Subject: ACS + Riak-CS, works?
>
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
> what I try when it goes to download and save the template, after its posted
> a series of multi part uploads it ends up sending a 403.  The access log
> looks like
>
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
>
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
> OpenJDK_64-Bit_Server_VM/20.0-b12"
>
> And then in the error log I always get
>
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
> record for s3 failed. Reason: disconnected
>
> I'm able to interact with Riak-CS with s3cmd just fine but its not using
> multipart. I think I'm going to try to download a new s3cmd that does
> multipart and see if even that will work.
>
> Darren
>

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

It's happening when I first create the zone.  I'll debug it further to tell
you exactly what its happening.  But I don't understand how the following
code in DefaultEndpointSelector could be correct.  If I have a zone, and I
delete/stop/crash the SSVM, and at the same time register a new template,
it seems the DownloadCommand will go to LocalHostEndPoint.

protected EndPoint findEndpointForImageStorage(DataStore store) {
Long dcId = null;
Scope storeScope = store.getScope();
if (storeScope.getScopeType() == ScopeType.ZONE) {
dcId = storeScope.getScopeId();
}
// find ssvm that can be used to download data to store. For
zone-wide
// image store, use SSVM for that zone. For region-wide store,
// we can arbitrarily pick one ssvm to do that task
List ssAHosts =
listUpAndConnectingSecondaryStorageVmHost(dcId);
if (ssAHosts == null || ssAHosts.isEmpty()) {
s_logger.info("No running ssvm is found, so command will be
sent to LocalHostEndPoint");
return LocalHostEndpoint.getEndpoint(); // use local host as
endpoint in
// case of no ssvm existing
}
Collections.shuffle(ssAHosts);
HostVO host = ssAHosts.get(0);
return RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
host.getPrivateIpAddress(),
host.getPublicIpAddress());
}

Darren


On Thu, Oct 31, 2013 at 9:42 AM, Min Chen  wrote:

> Hi Darren,
>
> The logic of sending command to Local in case of no SSVM is up is
> to
> support S3 as secondary storage. In case of S3 as secondary storage,
> system vm template is automatically downloaded to S3 when S3 is added into
> CloudStack, unlike NFS secondary storage case where we used a script to
> pre-populate system vm template. The thing I don't get is: what scenario
> are you trying to do to trigger DownloadCommand when SSVM is not up?
>
> Thanks
> -min
>
> On 10/31/13 9:32 AM, "Darren Shepherd" 
> wrote:
>
> >No, that's not the problem.  The problem is that the DownloadCommand
> >is supposed to go to the SSVM, but it sending it to the
> >LocalHostEndPoint which is the mgmt server.  So the mgmt server can
> >not create /var/cloudstack/mnt/
> >secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
> >so that it does not have sudo access.  A standard cloudstack install
> >has sudo access so it will create those folders and mount on the mgmt
> >server, but it shouldn't.
> >
> >Please don't tell me we made a change so that the mgmt server is
> >expected to have NFS access to all secondary storages.
> >
> >Darren
> >
> >On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU  wrote:
> >> Please check if you can
> >> create
> >>/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> >> and mount nfs point.
>
>

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

Yeah I think it would be great to talk about this at CCC.  I'm
hesitant to further narrow down the definition of the network.  For
example, I think OpenStack's Neutron is fundamentally flawed because
they defined a network as a L2 segment.  During my time that I was
away from CloudStack the physical network concept was added as a first
class object in CloudStack.  I don't know if I fully agree with it.  I
have to dig deeper and just haven't had the time to.  My inclination
is that is was one step too far and that it isn't really needed.  I
know you need to associate traffic with essentially a bridge on the
hypervisor, but that can be done with just labelling the network
offering that created the network.  So I need to dig into this
further.

In the world of SDN, I think its even more important to keep the
definition of the a network loose.  SDN has the capability of
completely changing the way we look at L2 and L3.  Currently in
networking we group things by L3 and L2 concepts as that is how
routers and switches are laid out today.  As SDN matures and you see
more flow oriented design it won't make sense to group things using L2
and L3 concepts (as those become more a physical fabric technology),
the groups becomes more loose and thus the definition of a network
should be loose.

Now that's not to say that a network can't provide L2 and L3
information.  You should be able to create a network in CloudStack and
based on the configuration you know that it is a single L2 or L3.  It
is just that the core orchestration system can't make that fundamental
assumption.  I'd be interested in furthering the model and maybe
adding a concept of a L2 network such that a network guru when
designing a network, can define multiple l2networks and associate them
with the generic network that was created.  That idea I'm still
toiling with.

For example, when configuring DHCP on the systemvm.  DHCP is a L2
based serviced.  So to configure DHCP you really need to know for each
nic, what is the L2 its attached to and what are the VMs associated
with that L2.  Today, since there is no first class concept of a L2
network, you have to look at the implied definition of L2.  For basic
networks, the L2 is the Pod, so you need to list all VMs in that Pod.
For guest/VPC networks, the L2 is the network object, so you need to
list all VMs associated with the network.  It would be nice if when
the guru designed the network, it also defined the l2networks, and
then when a VM starts the guru the reserve() method could associate
the l2network to the nic.  So the nic object would have a network_id
and a l2_network_id.

I've gone off on a total tangent, but yeah, we should talk.

Darren

On Thu, Oct 31, 2013 at 2:53 AM, Hugo Trippaers  wrote:
>
> On 31 okt. 2013, at 03:48, Darren Shepherd  
> wrote:
>
>> Its important to remember that the definition of a Network in
>> CloudStack is quite purposefully very loose.  A network in the
>> simplest terms is just a group of VMs.  A guru creates and manages
>> that network.  A network is not strictly defined as a union of
>> traffictype/isolationtype/networktype/guesttype.  So, for example,
>> there need not be a 1-to-1 mapping between Guru and IsolationType.
>> You should be able to have multiple gurus per any context.  That's the
>> theory and vision.
>
> I think that vision was put down somewhere before we entered the era of 
> software defined networking. At the moment it is simply not true as long as 
> the guru is responsible for creating the overlay networks there is a 1 on 1 
> relation with the guru and the supported isolation types in several cases.
>
>>
>> Now the reality is that the current selection logic of the guru in the
>> orchestrator doesn't fully back up that vision because the selection
>> logic too easily allows two Guru's to implement the same network.
>>
>> So I think the current behavior of setupNetwork is flawed.  This
>> change, I feel, muddies to a certain degree the definition of a
>> network in CloudStack and attempts to say a network is strict union of
>> traffic/isolation/network/guest-type.  If we merge this change, does
>> it hurt anything?  No, its more just conceptual disagreement.  When I
>> get to the point that I feel setupNetwork is limiting what I want to
>> do, I'll propose a change.  Right now, I just accept that its not
>> right in my mind.
>
> I’ve fixed the definition of setupNetwork so the function returns a single 
> network when called. This was already the effective behavior, but now its put 
> down in code as well. I’ll push that change to the branch today.
>
>
>>
>> So I'd have to say I'm neutral on this change.  I don't care too much
>> either way as I think down the

Re: Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

No, that's not the problem.  The problem is that the DownloadCommand
is supposed to go to the SSVM, but it sending it to the
LocalHostEndPoint which is the mgmt server.  So the mgmt server can
not create /var/cloudstack/mnt/
secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011.  I have my mgmt setup
so that it does not have sudo access.  A standard cloudstack install
has sudo access so it will create those folders and mount on the mgmt
server, but it shouldn't.

Please don't tell me we made a change so that the mgmt server is
expected to have NFS access to all secondary storages.

Darren

On Thu, Oct 31, 2013 at 8:54 AM, Wei ZHOU  wrote:
> Please check if you can
> create /var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011,
> and mount nfs point.

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

This is fixed now.  Sorry about that.  I really do go through extra
length to test my changes.  I'm finding it difficult to ensure quality
because there is so much variation.  You have to test everything in so
many contexts.  It shouldn't be like that I hope to help reduced the
amount of variation in this platform.  This was part of why I made the
db.properties change to begin with.  I wanted to change MacAddress to
read from db.properties, but then I found out that that properties
file is loaded in probably 10 different places in all slightly
different ways.  Some places would decrypt, some wouldn't, some
ignored if the file was not found, some would just throw a NPE.  So I
consolidated all that code.

Darren


On Thu, Oct 31, 2013 at 9:07 AM, Darren Shepherd
 wrote:
> Ugh, I see what I did now.  When I tested that code I changed the code
> I didn't do a "mvn install" before running "mvn ... deploydb."  That's
> annoying, now I want to figure out why maven doesn't pickup changes
> unless I do an install.  So I effectively tested with the old code.
>
> Darren
>
> On Thu, Oct 31, 2013 at 8:52 AM, Darren Shepherd
>  wrote:
>> I'll fix that.  Gimme ten mintues.  I specifically looked at that code
>> and thought I didn't change the behavior, but I guess I screwed it up.
>>
>> Just a general comment.  There's too many ways to do the same thing in
>> CloudStack.  Especially the database.  The way databases are setup for
>> developers shouldn't be so different from production.  The way we
>> manage the DB at the moment is so problematic in my mind.
>>
>> Darren
>>
>>
>> On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
>>> Heya,
>>>
>>> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
>>> loading of the db.properties file, however this also affected the 
>>> DatabaseCreator used by the deploydb procedure to refresh the database. 
>>> Effectively the db.properies.override is ignored at the moment and the 
>>> db.proeprties file in utils/conf/ will be used instead. We need to come up 
>>> with a nice solution for this in the near future, but in the meantime make 
>>> any custom setting there. Just don’t commit them ;-)
>>>
>>> Cheers,
>>>
>>> Hugo

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

Ugh, I see what I did now.  When I tested that code I changed the code
I didn't do a "mvn install" before running "mvn ... deploydb."  That's
annoying, now I want to figure out why maven doesn't pickup changes
unless I do an install.  So I effectively tested with the old code.

Darren

On Thu, Oct 31, 2013 at 8:52 AM, Darren Shepherd
 wrote:
> I'll fix that.  Gimme ten mintues.  I specifically looked at that code
> and thought I didn't change the behavior, but I guess I screwed it up.
>
> Just a general comment.  There's too many ways to do the same thing in
> CloudStack.  Especially the database.  The way databases are setup for
> developers shouldn't be so different from production.  The way we
> manage the DB at the moment is so problematic in my mind.
>
> Darren
>
>
> On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
>> Heya,
>>
>> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
>> loading of the db.properties file, however this also affected the 
>> DatabaseCreator used by the deploydb procedure to refresh the database. 
>> Effectively the db.properies.override is ignored at the moment and the 
>> db.proeprties file in utils/conf/ will be used instead. We need to come up 
>> with a nice solution for this in the near future, but in the meantime make 
>> any custom setting there. Just don’t commit them ;-)
>>
>> Cheers,
>>
>> Hugo

Re: note to devcloud users on db.properties.override

[Darren Shepherd]

I'll fix that.  Gimme ten mintues.  I specifically looked at that code
and thought I didn't change the behavior, but I guess I screwed it up.

Just a general comment.  There's too many ways to do the same thing in
CloudStack.  Especially the database.  The way databases are setup for
developers shouldn't be so different from production.  The way we
manage the DB at the moment is so problematic in my mind.

Darren


On Thu, Oct 31, 2013 at 7:18 AM, Hugo Trippaers  wrote:
> Heya,
>
> With commit 1460196496d73e0db25c7beb2392cfaf9d591ed7 Darren improved the 
> loading of the db.properties file, however this also affected the 
> DatabaseCreator used by the deploydb procedure to refresh the database. 
> Effectively the db.properies.override is ignored at the moment and the 
> db.proeprties file in utils/conf/ will be used instead. We need to come up 
> with a nice solution for this in the near future, but in the meantime make 
> any custom setting there. Just don’t commit them ;-)
>
> Cheers,
>
> Hugo

Sending to LocalHostEndpoint, Is there any resolution for this?

[Darren Shepherd]

Edison, Min,

When creating a Zone and there is no SSVM so it sends to
LocalHostEndPoint.  I always get the below error.  Is there some fix
for this?

ERROR [o.a.c.s.r.NfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) Unable to create local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
ERROR [o.a.c.s.r.LocalNfsSecondaryStorageResource]
(pool-4-thread-1:ctx-fb095de0) GetRootDir for
nfs://192.168.3.134:/exports/secondary1 failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
com.cloud.utils.exception.CloudRuntimeException: Unable to create
local folder for:
/var/cloudstack/mnt/secStorage/64d6e26f-e656-3ba3-908f-ce6610ede011 in
order to mount nfs://192.168.3.134:/exports/secondary1
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.ensureLocalRootPathExists(NfsSecondaryStorageResource.java:2456)
at 
org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.mount(LocalNfsSecondaryStorageResource.java:96)
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.mountUri(NfsSecondaryStorageResource.java:2293)
at 
org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.getRootDir(LocalNfsSecondaryStorageResource.java:85)
at 
org.apache.cloudstack.storage.template.DownloadManagerImpl.handleDownloadCommand(DownloadManagerImpl.java:695)

Darren

ACS + Riak-CS, works?

[Darren Shepherd]

Has anybody gotten CloudStack and Riak-CS to work together.  No matter what
I try when it goes to download and save the template, after its posted a
series of multi part uploads it ends up sending a 403.  The access log
looks like

127.0.0.1 - - [31/Oct/2013:05:41:11 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:28 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:42 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
OpenJDK_64-Bit_Server_VM/20.0-b12"

And then in the error log I always get

2013-10-31 05:41:51.867 [error]
<0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record
for s3 failed. Reason: disconnected

I'm able to interact with Riak-CS with s3cmd just fine but its not using
multipart. I think I'm going to try to download a new s3cmd that does
multipart and see if even that will work.

Darren

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

On Wed, Oct 30, 2013 at 11:36 AM, Pedro Roque Marques
 wrote:
>
> Why ? That seems to me the most obvious way to do it.
> There are different networking solutions: e.g. VLANs and overlays such as 
> OpenContrail that assume an L3 switching topology.  For a deployment one 
> would tend to choose a solution associated with the physical network.

Sure that would be simple.  The problem is that Zone also implies
other things.  Secondary/Primary storage for example.  Zone puts other
limitations on things.  So by tying networking to a zone you can get
into the situation where somebody has a CloudStack installation, and
they want to add OpenContrail.  If we tie it to the zone, that may
mean that in order for them to use OpenContrail they may need to get
another NFS secondary storage, primary storage, compute nodes, etc and
then copy all the templates from one zone to another, etc, etc.

So currently you can't have Basic and Advanced networking in the same
zone.  I think it should be possible.  Imagine I had a smaller
dev/test cloud.  Today if I want basic and advance networking (because
those are two distinict work loads), I have to have servers dedicated
to each type.  So 10 for basic, 10 or advanced.  Now the networking
under the hood for basic and advance can mix quite easily but just
because we decided to implement it as it is today, I'm suddenly forced
to managed two pools of physical resources, when it would have been
must easier and more cost effective to have one pool.

I think this is general thing that should change with CloudStack over
time.  Zone and Pod should not be so closely tied to network concepts.
 CloudStack should be capable of being looser for people who want more
hybrid work loads.  We too easily say that a service provider is just
going to choose one or the other so why support mixed configurations.
But there are more people that could be using CloudStack but are
turned off by the strict models and concepts that CloudStack enforces.
 I have to say I was one of those people.  CloudStack forced me to
manage my infrastructure in a certain way that I didn't particularly
care for.

Darren

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

Its important to remember that the definition of a Network in
CloudStack is quite purposefully very loose.  A network in the
simplest terms is just a group of VMs.  A guru creates and manages
that network.  A network is not strictly defined as a union of
traffictype/isolationtype/networktype/guesttype.  So, for example,
there need not be a 1-to-1 mapping between Guru and IsolationType.
You should be able to have multiple gurus per any context.  That's the
theory and vision.

Now the reality is that the current selection logic of the guru in the
orchestrator doesn't fully back up that vision because the selection
logic too easily allows two Guru's to implement the same network.

So I think the current behavior of setupNetwork is flawed.  This
change, I feel, muddies to a certain degree the definition of a
network in CloudStack and attempts to say a network is strict union of
traffic/isolation/network/guest-type.  If we merge this change, does
it hurt anything?  No, its more just conceptual disagreement.  When I
get to the point that I feel setupNetwork is limiting what I want to
do, I'll propose a change.  Right now, I just accept that its not
right in my mind.

So I'd have to say I'm neutral on this change.  I don't care too much
either way as I think down the line this all needs to change.

Darren

On Wed, Oct 30, 2013 at 11:36 AM, Pedro Roque Marques
 wrote:
>
> On Oct 30, 2013, at 11:08 AM, Darren Shepherd  
> wrote:
>
>> I definitely don't want the setup to be done at a zone level.
>
> Why ? That seems to me the most obvious way to do it.
> There are different networking solutions: e.g. VLANs and overlays such as 
> OpenContrail that assume an L3 switching topology.  For a deployment one 
> would tend to choose a solution associated with the physical network.
>
>> Specifically, I'd like to get rid of the notion of basic and advanced
>> zones which would mean that you could be mixing gurus in the same
>> zone.  If anything, I'd say the simplest would be to just specify the
>> Guru in the network offering.
>
> Basic and advanced zones is rather unhelpful, i agree. But you still can't 
> Mix Guru's unless they can interoperate... The way for them to interop (i.e. 
> minimum common denominator) is to connect at L3 in an router between zones.
>
>>
>> I like the work that Hugo has done, but the implications of that
>> change are 2 things.  First, we only want one guru to match.  If that
>> is the case, then lets change the NetworkOrchestrator.setupNetwork()
>> to just return one network.  I have a hard time understanding why you
>> would want to return two or more.  Even if the guru created two
>> networks internally, only one should be return to the caller.  There's
>> not a good way to invoke setupNetwork() and handle cases where more
>> than one network comes back.
>>
>> Second, we are saying the only way to choose a guru is based on
>> networkType/trafficType/guestType/isolationmethods.  I think the core
>> logic should be more flexible than that.  We can first check those 4
>> items, and then if we find more than one guru that matches, use the
>> best one (before what I said about adding canHandle to the interface).
>> But the check shouldn't be only those four criteria.  Hugo's change
>> does not strictly enforce that, but it is essentially implied.
>>
>> After saying all that, if we just put the Guru in the network
>> offering.  Would that work for everything?  That would be a really
>> simple and explicit way to configure this all.
>
> My preference would be for a NetworkManager per zone.
>
>>
>> Darren
>>
>>
>> On Wed, Oct 30, 2013 at 10:31 AM, Alex Huang  wrote:
>>> I agree with Hugo that the current way of ask NetworkGuru to decide itself 
>>> is not desirable.  It makes absolute sense to change that but I wonder why 
>>> do it through capabilities.  Why not just ask the system admin to choose 
>>> when they setup the zone?   When I originally designed NetworkGuru, I 
>>> figured there's a difference in the L2-L3 technology deployed and the 
>>> network services (L4-L7) provided on top of it.  So I separated out 
>>> NetworkGuru and NetworkElement.  However, I didn't think it's likely that 
>>> there would be two or three different type of L2-L3 technologies deployed 
>>> within the same zone.  I figured we can leave it to the system admin to 
>>> just pick which set of NetworkGurus should be deployed in a certain zone 
>>> and that's good enough.
>>>
>>> I do think there should be more tie in between the NetworkElements and 
>>> NetworkGurus

Re: ifconfig and MacAddress

[Darren Shepherd]

Shoot, I didn't see your patch.  I just created a patch that
supersedes yours.  https://reviews.apache.org/r/15108/  The logic is
as follows

1. Look for MAC that matches interface of cluster.node.IP
2. Look for first interface alphabetically that has a non-linklocal address
3. Look for first interface alphabetically
4. Randomly generate a mac address

Darren

On Wed, Oct 30, 2013 at 4:27 PM, Laszlo Hornyak
 wrote:
> Hi,
>
> A patch to make MacAddress eat the Windows ipconfig output.
> https://reviews.apache.org/r/15106/
>
> I have one more question about how MacAddress should work. It seems it is
> returning the mac address of the first interface of the ipconfig/ifconfig
> output. Therefore if you configure up a new network, e.g. let libvirt start
> a new network or just plug in a usb wifi adapter, it may get something
> else. Does this meet the expectations?
>
>
>
>
> On Tue, Oct 29, 2013 at 8:18 PM, Laszlo Hornyak 
> wrote:
>
>> Hi,
>>
>> I will update my patch with a fix, but I also like Darran's solution: just
>> delete the class and/or use InetAddress
>>
>>
>>
>>
>> On Tue, Oct 29, 2013 at 6:42 PM, Kelven Yang wrote:
>>
>>>  Windows ipconfig is not supported, a fix is welcome.
>>>
>>>  To run a CS management server cluster on Windows. manual configuration
>>> is required. cluster.node.ip needs to configured correctly in db.properties
>>>
>>>  Kelven
>>>
>>>   From: Alex Huang 
>>> Date: Tuesday, October 29, 2013 at 8:43 AM
>>> To: Laszlo Hornyak , "dev@cloudstack.apache.org"
>>> 
>>> Cc: Kelven Yang 
>>> Subject: RE: ifconfig and MacAddress
>>>
>>>   I can confirm the code’s only there because Java didn’t have it before.
>>> 
>>>
>>> ** **
>>>
>>> --Alex
>>>
>>> ** **
>>>
>>> *From:* Laszlo Hornyak 
>>> [mailto:laszlo.horn...@gmail.com]
>>>
>>> *Sent:* Monday, October 28, 2013 11:50 PM
>>> *To:* dev@cloudstack.apache.org
>>> *Cc:* Alex Huang; Kelven Yang
>>> *Subject:* Re: ifconfig and MacAddress
>>>
>>> ** **
>>>
>>> Hi,
>>>
>>> I noticed other problem with the MacAddress, it appears that it does not
>>> handle the Windows ipconfig output.
>>>
>>> (see related patch) https://reviews.apache.org/r/14514/ I tested it with
>>> win8, 7, and XP outputs
>>>
>>> As far as I remember this may be a problem when running CS cluster on
>>> windows, since each node determines the default as id.
>>>
>>> ** **
>>>
>>> On Sun, Oct 27, 2013 at 1:11 AM, Darren Shepherd <
>>> darren.s.sheph...@gmail.com> wrote:
>>>
>>> Is there a specific reason that in MacAddress it uses "ifconfig" or
>>> other utilities to grab the Mac Address as opposed to the java API
>>> java.net.NetworkInterface?  There's a comment in that code that says
>>> that code was copied from some public domain utility.  So I'm guessing
>>> its there just because that's what somebody wrote back before Java 6
>>> introduced NetworkInteface.getHardwareAddress().  I'd like to rewrite
>>> this code to use the standard API, grab the Mac deterministically and
>>> also not use ifconfig which was deprecated 3 years ago.
>>>
>>> Darren
>>>
>>>
>>>
>>>
>>> --
>>>
>>> EOF 
>>>
>>
>>
>>
>> --
>>
>> EOF
>>
>
>
>
> --
>
> EOF

Review Request 15108: Grab MacAddress in a more deterministic fashion

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15108/
---

Review request for cloudstack, Alex Huang and Laszlo Hornyak.


Repository: cloudstack-git


Description
---

The MAC address is determined in the following order

1. Look for MAC that matches interface of cluster.node.IP
2. Look for first interface alphabetically that has a non-linklocal address
3. Look for first interface alphabetically
4. Randomly generate a mac address


Diffs
-

  utils/src/com/cloud/utils/net/MacAddress.java 15350c8 

Diff: https://reviews.apache.org/r/15108/diff/


Testing
---


Thanks,

Darren Shepherd

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

I definitely don't want the setup to be done at a zone level.
Specifically, I'd like to get rid of the notion of basic and advanced
zones which would mean that you could be mixing gurus in the same
zone.  If anything, I'd say the simplest would be to just specify the
Guru in the network offering.

I like the work that Hugo has done, but the implications of that
change are 2 things.  First, we only want one guru to match.  If that
is the case, then lets change the NetworkOrchestrator.setupNetwork()
to just return one network.  I have a hard time understanding why you
would want to return two or more.  Even if the guru created two
networks internally, only one should be return to the caller.  There's
not a good way to invoke setupNetwork() and handle cases where more
than one network comes back.

Second, we are saying the only way to choose a guru is based on
networkType/trafficType/guestType/isolationmethods.  I think the core
logic should be more flexible than that.  We can first check those 4
items, and then if we find more than one guru that matches, use the
best one (before what I said about adding canHandle to the interface).
 But the check shouldn't be only those four criteria.  Hugo's change
does not strictly enforce that, but it is essentially implied.

After saying all that, if we just put the Guru in the network
offering.  Would that work for everything?  That would be a really
simple and explicit way to configure this all.

Darren


On Wed, Oct 30, 2013 at 10:31 AM, Alex Huang  wrote:
> I agree with Hugo that the current way of ask NetworkGuru to decide itself is 
> not desirable.  It makes absolute sense to change that but I wonder why do it 
> through capabilities.  Why not just ask the system admin to choose when they 
> setup the zone?   When I originally designed NetworkGuru, I figured there's a 
> difference in the L2-L3 technology deployed and the network services (L4-L7) 
> provided on top of it.  So I separated out NetworkGuru and NetworkElement.  
> However, I didn't think it's likely that there would be two or three 
> different type of L2-L3 technologies deployed within the same zone.  I 
> figured we can leave it to the system admin to just pick which set of 
> NetworkGurus should be deployed in a certain zone and that's good enough.
>
> I do think there should be more tie in between the NetworkElements and 
> NetworkGurus.  For example, whether a certain NetworkElement can work with 
> the L2-L3 technology deployed.
>
> --Alex
>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Wednesday, October 30, 2013 10:11 AM
>> To: dev@cloudstack.apache.org; Alex Huang; Chiradeep Vittal
>> Subject: Re: [MERGE] network-guru-orchestration into master
>>
>> I don't particular like saying that picking a Guru is based solely on that 
>> criteria.
>> It should be much looser than that.  If the problem you are trying to fix is 
>> two
>> Gurus implement the same network then we need to set back a bit.  I'd like
>> Alex or Chiradeep to weigh in on this.  Currently setupNetwork returns a list
>> of networks that were created.  I've looked at the code and every single
>> invocation to setupNetwork expects (and mostly hard code) a response of
>> one network.
>>
>> So for a more proper fix I'd say that the API of setupNetwork should just
>> return Network.  Additionally NetworkGuru should have a
>> canHandle/canDesign method on the interface that returns a priority/score
>> (similar to what was just done in the storage strategies).  Then the
>> orchestrator can find the best match and then use the correct guru.  Now the
>> consolidated logic that you've done should be in a abstract base class that
>> handles the basic cases of traffic type, network type, etc.
>>
>> Darren
>>
>> On Wed, Oct 30, 2013 at 6:10 AM, Hugo Trippaers  wrote:
>> >
>> > On 30 okt. 2013, at 02:09, Darren Shepherd 
>> wrote:
>> >
>> >> First, I like the idea of consolidating logic.  I could see also
>> >> implementing this as just an abstract base class that handles this
>> >> common logic.  I'm not sure which approach I prefer.  Exactly what is
>> >> the problem that you are trying to solve?  Without more details, I'd
>> >> lean towards implementing this logic in an abstract base class that
>> >> all NetworkGurus can choose to extend.
>> >>
>> >
>> > Not as much a problem as a design choice. It is my intention to use
>> > capabilities to select the NetworkGuru instead of asking each network
>> > guru in turn if it wants to design 

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

I don't particular like saying that picking a Guru is based solely on
that criteria.  It should be much looser than that.  If the problem
you are trying to fix is two Gurus implement the same network then we
need to set back a bit.  I'd like Alex or Chiradeep to weigh in on
this.  Currently setupNetwork returns a list of networks that were
created.  I've looked at the code and every single invocation to
setupNetwork expects (and mostly hard code) a response of one network.

So for a more proper fix I'd say that the API of setupNetwork should
just return Network.  Additionally NetworkGuru should have a
canHandle/canDesign method on the interface that returns a
priority/score (similar to what was just done in the storage
strategies).  Then the orchestrator can find the best match and then
use the correct guru.  Now the consolidated logic that you've done
should be in a abstract base class that handles the basic cases of
traffic type, network type, etc.

Darren

On Wed, Oct 30, 2013 at 6:10 AM, Hugo Trippaers  wrote:
>
> On 30 okt. 2013, at 02:09, Darren Shepherd  
> wrote:
>
>> First, I like the idea of consolidating logic.  I could see also
>> implementing this as just an abstract base class that handles this
>> common logic.  I'm not sure which approach I prefer.  Exactly what is
>> the problem that you are trying to solve?  Without more details, I'd
>> lean towards implementing this logic in an abstract base class that
>> all NetworkGurus can choose to extend.
>>
>
> Not as much a problem as a design choice. It is my intention to use 
> capabilities to select the NetworkGuru instead of asking each network guru in 
> turn if it wants to design the network. The idea here is that new network 
> gurus only need to supply a list of capabilities to be integrated into 
> cloudstack. Like i can handle isolation type X in advanced networks for 
> traffic type Guest. The network orchestrator can make an informed decision on 
> who to give the task (and warn if there is more than one capable)
>
>
>> Darren
>>
>> On Tue, Oct 29, 2013 at 9:42 AM, Hugo Trippaers  wrote:
>>> Hey guys,
>>>
>>> This is something i had on my wish list for some time. The current way 
>>> network gurus are handled is that each guru is asked to design a network 
>>> and will decide by itself to either do it or don’t. Most gurus have sane 
>>> checks on which types of networks it can handle, but we have seen issues 
>>> there in the past.
>>>
>>> With these changes the network orchestrator will check the capabilities of 
>>> a guru and based on that ask one or more gurus to design the network. With 
>>> this the power is where is should new, the network orchestrator.
>>>
>>> This also means that new networking plugins with gurus will have an easier 
>>> integration, just list the capabilities. It will also save some database 
>>> calls (once i clean out all canHandle functions) as gurus will have to 
>>> lookup less to decide if they should to their job.
>>>
>>> What do you guys think?
>>>
>>> Current branch is tested with devcloud in a manual test, so there is more 
>>> work to do there. I wanted to get some opinions while performing more tests.
>>>
>>> Cheers,
>>>
>>> Hugo
>

Re: Docker?

[Darren Shepherd]

This would be a great thing to try at the CCC hackathon.  Docker poses
some interesting challenges and to do a useful integration is going to
take some careful design.  There's two things to consider.  First,
docker's #1 feature in my mind is that it is a paradigm shift.  You
interact with docker in a different way than you would EC2.  If you
attempt to make docker just another hypervisor, it just starts looking
like LXC and then has very little benefit over LXC integration.  You
need to determine how to integrate docker while still keeping its
features (like Dockerfile).

Second, docker is very limited in the networking area.  Docker as it
currently stand manages all of networking aspect which includes
assigning its own IP, random port mappings, etc.

I have a lot of ideas about this and I'd love to chat with some people
at CCC about it.  We could probably get it working at CCC I'll
probably come prepared with stuff.

Darren

On Wed, Oct 30, 2013 at 6:32 AM, Alex Hitchins
 wrote:
> I'd be interested in looking more into Docker - are you planning to host a
> talk on the subject?
>
>
> Alexander Hitchins
> -
> Personal Email : a...@alexhitchins.com
> Apache Email   : a...@alexhitchins.com
> Website: http://alexhitchins.com
> Mobile : 07788 423 969
>
> -Original Message-
> From: Nguyen Anh Tu [mailto:ng.t...@gmail.com]
> Sent: 30 October 2013 13:18
> To: Sebastien Goasguen
> Cc: dev@cloudstack.apache.org
> Subject: Re: Docker?
>
> Hi,
>
> I'm starting look into docker's architecture and figure out how it plays
> with lxc and openstack at that moment. Will try to figure some of key points
> before the CCC13 start, so we can discuss directly at there.
>
> Sent from my GT-N7000
> On Oct 30, 2013 4:14 PM, "Sebastien Goasguen"  wrote:
>
>>
>> On Oct 30, 2013, at 4:56 AM, Nux!  wrote:
>>
>> > Hi,
>> >
>> > Docker[1] seems to be all the rage these days and has breathed new
>> > life
>> in LXC. Is there any interest whatsoever from the devs to support for it?
>> > I noticed it has already landed in the other stack.
>> >
>>
>> The answer is yes :)
>>
>> Nugyen has started looking at it, so as Darren.
>>
>> Can you start a wiki page to dump some ideas, there are some
>> interesting architectural decision to be had.
>>
>> -sebastien
>>
>> > Lucian
>> >
>> > [1] - http://www.docker.io/
>> >
>> > --
>> > Sent from the Delta quadrant using Borg technology!
>> >
>> > Nux!
>> > www.nux.ro
>>
>>
>

Re: [MERGE] network-guru-orchestration into master

[Darren Shepherd]

First, I like the idea of consolidating logic.  I could see also
implementing this as just an abstract base class that handles this
common logic.  I'm not sure which approach I prefer.  Exactly what is
the problem that you are trying to solve?  Without more details, I'd
lean towards implementing this logic in an abstract base class that
all NetworkGurus can choose to extend.

Darren

On Tue, Oct 29, 2013 at 9:42 AM, Hugo Trippaers  wrote:
> Hey guys,
>
> This is something i had on my wish list for some time. The current way 
> network gurus are handled is that each guru is asked to design a network and 
> will decide by itself to either do it or don’t. Most gurus have sane checks 
> on which types of networks it can handle, but we have seen issues there in 
> the past.
>
> With these changes the network orchestrator will check the capabilities of a 
> guru and based on that ask one or more gurus to design the network. With this 
> the power is where is should new, the network orchestrator.
>
> This also means that new networking plugins with gurus will have an easier 
> integration, just list the capabilities. It will also save some database 
> calls (once i clean out all canHandle functions) as gurus will have to lookup 
> less to decide if they should to their job.
>
> What do you guys think?
>
> Current branch is tested with devcloud in a manual test, so there is more 
> work to do there. I wanted to get some opinions while performing more tests.
>
> Cheers,
>
> Hugo

Re: cloudmonkey exit value

[Darren Shepherd]

First, I care more about the exit code working. 

I want just the value.  So currently I have

cloudmonkey set color false
cloudmonkey createVM ... | grep ^id | awk -F' = ' '{print $2}'

Darren

> On Oct 27, 2013, at 9:48 PM, Prasanna Santhanam  wrote:
> 
>> On Sun, Oct 27, 2013 at 04:49:35PM -0700, Darren Shepherd wrote:
>> It would be really nice if the API failed if cloudmonkey exited with
>> !=0 exit code.  I've been scripting some stuff with cloud monkey and
>> this makes it difficult.  It would also be nice to somehow specific
>> which field you want printed.  So when I do a create command I can say
>> just echo id value only.  Would make my scripts much easier.
> you can pipe with grep?
> $ cloudmonkey createCommand | grep id
> 
>> Darren
> 
> -- 
> Prasanna.,
> 
> 
> Powered by BigRock.com
> 

Re: [Merge] hyperv branch to master

[Darren Shepherd]

+1

When you merge you will get a conflict in the spring configuration.
componentContext.xml and others are gone now in master.  You will need
to update to the new style.  The easiest will be to refer to XenServer
spring config for an example of how to do it.  You can largely copy
and paste the contents of
plugins/hypervisors/xen/resources/META-INF/cloudstack and replace
xenserver with hyperv.  You can also refer to
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions
for more in depth explaination of the changes.

Darren

On Tue, Oct 29, 2013 at 9:35 AM, David Nalley  wrote:
> On Mon, Oct 28, 2013 at 7:41 AM, Devdeep Singh  
> wrote:
>> Hi,
>>
>> I would like to merge the support for Hyperv to the master branch. 
>> Development for this has been done by Donal, Rajesh, Anshul and I on branch 
>> [1]. The feature was proposed for merge earlier [3] but unit tests for 
>> hyperv agent code were requested [4].
>>
>
> I know this has been discussed several times, but what was the
> decision about making the agent code live in its own repo?
> It's C#, might need to be signed to actually run easily on Windows, etc.
>
> --David

Re: Tiered Quality

[Darren Shepherd]

Starting with the honor system might be good.  It's not so easy some times to 
relate lines of code to functionality.  Also just because it hits a line of 
code doesn't mean it's really tested.

Can't we just get people to just put a check mark on some table in the wiki?

Darren

> On Oct 28, 2013, at 12:08 PM, Santhosh Edukulla 
>  wrote:
> 
> 1.It seems we already have a code coverage numbers using sonar as below. It 
> currently shows only the numbers for unit tests.
> 
> https://analysis.apache.org/dashboard/index/100206
> 
> 2. The below link has an explanation for using it for both integration and 
> unit tests.
> 
> http://docs.codehaus.org/display/SONAR/Code+Coverage+by+Integration+Tests+for+Java+Project
> 
> 3. Many links suggests it has good decision coverage facility compared to 
> other coverage tools.
> 
> http://onlysoftware.wordpress.com/2012/12/19/code-coverage-tools-jacoco-cobertura-emma-comparison-in-sonar/
> 
> Regards,
> Santhosh
> 
> From: Laszlo Hornyak [laszlo.horn...@gmail.com]
> Sent: Monday, October 28, 2013 1:43 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Tiered Quality
> 
> Sonar already tracks the unit test coverage. It is also able to track the
> integration test coverage, however this might be a bit more sophisticated
> in CS since not all hardware/software requirements are available in the
> jenkins environment. However, this could be a problem in any environment.
> 
> 
>> On Mon, Oct 28, 2013 at 5:53 AM, Prasanna Santhanam  wrote:
>> 
>> We need a way to check coverage of (unit+integration) tests. How many
>> lines of code hit on a deployed system that corresponds to the
>> component donated/committed. We don't have that for existing tests so
>> it makes it hard to judge if a feature that comes with tests covers
>> enough of itself.
>> 
>>> On Sun, Oct 27, 2013 at 11:00:46PM +0100, Laszlo Hornyak wrote:
>>> Ok, makes sense, but that sounds like even more work :) Can you share the
>>> plan on how will this work?
>>> 
>>> 
>>> On Sun, Oct 27, 2013 at 7:54 PM, Darren Shepherd <
>>> darren.s.sheph...@gmail.com> wrote:
>>> 
>>>> I think it can't be at a component level because components are too
>> large.
>>>> It needs to be at a feature for implementation level.  For example,
>> live
>>>> storage migration for xen and live storage migration for kvm (don't
>> know if
>>>> that's a real thing) would be two separate items.
>>>> 
>>>> Darren
>>>> 
>>>>> On Oct 27, 2013, at 10:57 AM, Laszlo Hornyak <
>> laszlo.horn...@gmail.com>
>>>> wrote:
>>>>> 
>>>>> I believe this will be very useful for users.
>>>>> As far as I understand someone will have to qualify components. What
>> will
>>>>> be the method for qualification? I do not think simply the test
>> coverage
>>>>> would be right. But then if you want to go deeper, then you need a
>> bigger
>>>>> effort testing the components.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sun, Oct 27, 2013 at 4:51 PM, Darren Shepherd <
>>>>> darren.s.sheph...@gmail.com> wrote:
>>>>> 
>>>>>> I don't know if a similar thing has been talked about before but I
>>>>>> thought I'd just throws this out there.  The ultimate way to ensure
>>>>>> quality is that we have unit test and integration test coverage on
>> all
>>>>>> functionality.  That way somebody authors some code, commits to, for
>>>>>> example, 4.2, but then when we release 4.3, 4.4, etc they aren't on
>>>>>> the hook to manually tests the functionality with each release.  The
>>>>>> obvious nature of a community project is that people come and go.
>> If
>>>>>> a contributor wants to ensure the long term viability of the
>>>>>> component, they should ensure that there are unit+integration tests.
>>>>>> 
>>>>>> Now, for whatever reason whether good or bad, it's not always
>> possible
>>>>>> to have full integration tests.  I don't want to throw down the
>> gamut
>>>>>> and say everything must have coverage because that will mean some
>>>>>> useful code/feature will not get in because of some coverage wasn't
>>>>>> possible at the time.
>>>>>> 
>>>>>> What I propose is that for every feature or function we put it in a
>>>>>> tier of what is the quality of it (very similar to how OpenStack
>>>>>> qualifies their hypervisor integration).  Tier A means unit test and
>>>>>> integration test coverage gates the release.  Tier B means unit test
>>>>>> coverage gates the release.  Tier C mean who knows, it compiled.  We
>>>>>> can go through and classify the components and then as a community
>> we
>>>>>> can try to get as much into Tier A as possible.
>>>>>> 
>>>>>> Darren
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> EOF
>>> 
>>> 
>>> 
>>> --
>>> 
>>> EOF
>> 
>> --
>> Prasanna.,
>> 
>> 
>> Powered by BigRock.com
> 
> 
> --
> 
> EOF

cloudmonkey exit value

[Darren Shepherd]

It would be really nice if the API failed if cloudmonkey exited with
!=0 exit code.  I've been scripting some stuff with cloud monkey and
this makes it difficult.  It would also be nice to somehow specific
which field you want printed.  So when I do a create command I can say
just echo id value only.  Would make my scripts much easier.

Darren

Re: Tiered Quality

[Darren Shepherd]

I think it can't be at a component level because components are too large.  It 
needs to be at a feature for implementation level.  For example, live storage 
migration for xen and live storage migration for kvm (don't know if that's a 
real thing) would be two separate items.  

Darren

> On Oct 27, 2013, at 10:57 AM, Laszlo Hornyak  wrote:
> 
> I believe this will be very useful for users.
> As far as I understand someone will have to qualify components. What will
> be the method for qualification? I do not think simply the test coverage
> would be right. But then if you want to go deeper, then you need a bigger
> effort testing the components.
> 
> 
> 
> On Sun, Oct 27, 2013 at 4:51 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
> 
>> I don't know if a similar thing has been talked about before but I
>> thought I'd just throws this out there.  The ultimate way to ensure
>> quality is that we have unit test and integration test coverage on all
>> functionality.  That way somebody authors some code, commits to, for
>> example, 4.2, but then when we release 4.3, 4.4, etc they aren't on
>> the hook to manually tests the functionality with each release.  The
>> obvious nature of a community project is that people come and go.  If
>> a contributor wants to ensure the long term viability of the
>> component, they should ensure that there are unit+integration tests.
>> 
>> Now, for whatever reason whether good or bad, it's not always possible
>> to have full integration tests.  I don't want to throw down the gamut
>> and say everything must have coverage because that will mean some
>> useful code/feature will not get in because of some coverage wasn't
>> possible at the time.
>> 
>> What I propose is that for every feature or function we put it in a
>> tier of what is the quality of it (very similar to how OpenStack
>> qualifies their hypervisor integration).  Tier A means unit test and
>> integration test coverage gates the release.  Tier B means unit test
>> coverage gates the release.  Tier C mean who knows, it compiled.  We
>> can go through and classify the components and then as a community we
>> can try to get as much into Tier A as possible.
>> 
>> Darren
> 
> 
> 
> -- 
> 
> EOF

Tiered Quality

[Darren Shepherd]

I don't know if a similar thing has been talked about before but I
thought I'd just throws this out there.  The ultimate way to ensure
quality is that we have unit test and integration test coverage on all
functionality.  That way somebody authors some code, commits to, for
example, 4.2, but then when we release 4.3, 4.4, etc they aren't on
the hook to manually tests the functionality with each release.  The
obvious nature of a community project is that people come and go.  If
a contributor wants to ensure the long term viability of the
component, they should ensure that there are unit+integration tests.

Now, for whatever reason whether good or bad, it's not always possible
to have full integration tests.  I don't want to throw down the gamut
and say everything must have coverage because that will mean some
useful code/feature will not get in because of some coverage wasn't
possible at the time.

What I propose is that for every feature or function we put it in a
tier of what is the quality of it (very similar to how OpenStack
qualifies their hypervisor integration).  Tier A means unit test and
integration test coverage gates the release.  Tier B means unit test
coverage gates the release.  Tier C mean who knows, it compiled.  We
can go through and classify the components and then as a community we
can try to get as much into Tier A as possible.

Darren

Re: ifconfig and MacAddress

[Darren Shepherd]

Just want to clarify something for people reading this thread.  I'm not looking 
at code right now so I could be slightly wrong about the following.  'host' and 
'cluster.node.IP' are slightly different.  Cluster.node.ip is recorded in the 
mshost table and is used to tell cluster peers what IP the current node is 
addressable by.  The 'host' property is used to tell the systemvms what IP to 
use to get to the mgmt servers.  The host property in a production setup should 
actually be bound to a VIP so that you can handle failover between mgmt 
servers.  So cluster.node.IP != host in most setups.  If you don't set the 
cluster.node.IP (I don't know why) it seems to pick localhost.  If you are only 
running one mgmt server (like a dev setup) that is actually fine. Even though 
it's localhost it will still listen on *.  So in a dev setup the 
cluster.node.IP will be localhost, but you just need to set the host property 
to your real IP.  

This is working for me on master, so I'm not too sure what's going on in your 
setup.  Are you seeing that Marvin is not setting the prop in the config table. 
 Or are you seeing the prop in the config table but that it's not getting set 
on cmdline of the systemvms.  I typically set the host prop with SQL, so never 
tried through Marvin.  

Darren

> On Oct 26, 2013, at 9:30 PM, Marcus Sorensen  wrote:
> 
> there's a 'host' global config property that I believe is supposed to
> be the master mgmt server. For some reason this is also not working
> quite right for me on master, it used to set correctly per my marvin
> zone config, going from 4.2 to master the 'host' property always ends
> up as 'localhost' rather than the IP. This messes things up because it
> passes 'localhost' as the mgmt server to the system vms.
> 
> On Sat, Oct 26, 2013 at 9:44 PM, Darren Shepherd
>  wrote:
>> I completely agree and part of why I want to change it.  On my machine my 
>> wifi and Ethernet devices sometimes swap as being first in the list and it 
>> doesn't seem to like that at all.  I don't know if it's because my mgmt IP 
>> is localhost or what, but the new mgmt host entry never takes over ownership 
>> of the previous hosts.
>> 
>> I'd like to make it more deterministic.  Not completely sure what the 
>> algorithm should be.  I was thinking that I'd sort the interfaces by Mac or 
>> name and then look for the non-link local nic.
>> 
>> Actually, I can just look for the Mac of the IP set as the cluster.node.IP.  
>> Some stupid reason I was thinking that was in the DB, but it's (obviously?) 
>> in the db.propeties.
>> 
>> Darren
>> 
>>> On Oct 26, 2013, at 4:27 PM, Marcus Sorensen  wrote:
>>> 
>>> Tangent, if that's the code that comes up with mgmt server id (by Mac
>>> address conversion to short), its bitten me in the past. It doesn't
>>> tolerate new NICs well, for example in development when a mgmt server is
>>> also a host and a link local bridge is created. Seems bad form to just run
>>> ifconfig and grab the first Mac you find.
>>> On Oct 26, 2013 5:12 PM, "Darren Shepherd" 
>>> wrote:
>>> 
>>>> Is there a specific reason that in MacAddress it uses "ifconfig" or
>>>> other utilities to grab the Mac Address as opposed to the java API
>>>> java.net.NetworkInterface?  There's a comment in that code that says
>>>> that code was copied from some public domain utility.  So I'm guessing
>>>> its there just because that's what somebody wrote back before Java 6
>>>> introduced NetworkInteface.getHardwareAddress().  I'd like to rewrite
>>>> this code to use the standard API, grab the Mac deterministically and
>>>> also not use ifconfig which was deprecated 3 years ago.
>>>> 
>>>> Darren
>>>> 

Re: ifconfig and MacAddress

[Darren Shepherd]

I completely agree and part of why I want to change it.  On my machine my wifi 
and Ethernet devices sometimes swap as being first in the list and it doesn't 
seem to like that at all.  I don't know if it's because my mgmt IP is localhost 
or what, but the new mgmt host entry never takes over ownership of the previous 
hosts.  

I'd like to make it more deterministic.  Not completely sure what the algorithm 
should be.  I was thinking that I'd sort the interfaces by Mac or name and then 
look for the non-link local nic.

Actually, I can just look for the Mac of the IP set as the cluster.node.IP.  
Some stupid reason I was thinking that was in the DB, but it's (obviously?) in 
the db.propeties.  

Darren

> On Oct 26, 2013, at 4:27 PM, Marcus Sorensen  wrote:
> 
> Tangent, if that's the code that comes up with mgmt server id (by Mac
> address conversion to short), its bitten me in the past. It doesn't
> tolerate new NICs well, for example in development when a mgmt server is
> also a host and a link local bridge is created. Seems bad form to just run
> ifconfig and grab the first Mac you find.
> On Oct 26, 2013 5:12 PM, "Darren Shepherd" 
> wrote:
> 
>> Is there a specific reason that in MacAddress it uses "ifconfig" or
>> other utilities to grab the Mac Address as opposed to the java API
>> java.net.NetworkInterface?  There's a comment in that code that says
>> that code was copied from some public domain utility.  So I'm guessing
>> its there just because that's what somebody wrote back before Java 6
>> introduced NetworkInteface.getHardwareAddress().  I'd like to rewrite
>> this code to use the standard API, grab the Mac deterministically and
>> also not use ifconfig which was deprecated 3 years ago.
>> 
>> Darren
>> 

ifconfig and MacAddress

[Darren Shepherd]

Is there a specific reason that in MacAddress it uses "ifconfig" or
other utilities to grab the Mac Address as opposed to the java API
java.net.NetworkInterface?  There's a comment in that code that says
that code was copied from some public domain utility.  So I'm guessing
its there just because that's what somebody wrote back before Java 6
introduced NetworkInteface.getHardwareAddress().  I'd like to rewrite
this code to use the standard API, grab the Mac deterministically and
also not use ifconfig which was deprecated 3 years ago.

Darren

Re: failing unit tests....

[Darren Shepherd]

There is also a lot of static initialization.  I don't know exactly
what was the issue that broke the build, but I assume its because
Transaction(Legacy) got loaded, and that class in a static block loads
db.properties, which then call some encryption utilities that will
fail if db.properties is not there.  I'd rather remove the static
initializer (or atleast make it not blow up if its not there).  The
tricky part is that there are so many "main class" utilities that
depend on Transaction class.  I really want to standardize all
utilities that they initialize in the same way so we can clean this
stuff up, but that just takes time.

Darren

On Sat, Oct 26, 2013 at 11:17 AM, Laszlo Hornyak
 wrote:
> Hi Alex,
>
> The build was failing again today so I have sent another quick fix with
> commit id
> 7902315287c268ff81e3b6664df6ddee7351716a looks like the build is back to
> normal. The prevous fix was reverted as after Darren's latest changes the
> jdbc driver was no longer needed for tests in that project.
>
> The tests are a bit fragile and in my opinion the reason is that the
> components are building a bit too much on the environment they are running
> in. So writing good tests is a bit difficult in some cases, in some other
> cases almost impossible.
>
>
>
>
> On Sat, Oct 26, 2013 at 12:57 AM, Alex Huang  wrote:
>
>> Hi Laszlo,
>>
>> Thanks for the fix.  It does fix this problem.  I took a quick look.
>>
>> As I understand it from Alena, Darren had to move the initialization of
>> the LockMasterListener higher due to some problems Mike experienced.  My
>> guess is that it had to be higher because Spring cannot work with the
>> ordered initialization that we used to have for the components (new,
>> configure, start) so it was moved into static which relies on jvm
>> initialization order.  So moving the initialization of LockMasterListener
>> back into start() method of ManagementServer basically will break things
>> for Mike again.  I'll leave it to Darren to resolve this for now.  I don't
>> know enough about Spring loading to resolve this correctly for both cases.
>>
>> --Alex
>>
>> > -Original Message-
>> > From: Laszlo Hornyak [mailto:laszlo.horn...@gmail.com]
>> > Sent: Friday, October 25, 2013 3:19 PM
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: failing unit tests
>> >
>> > I have just sent a fix for that, looks like everything is green now.
>> Please check!
>> >
>> >
>> > On Fri, Oct 25, 2013 at 11:52 PM, Darren Shepherd <
>> > darren.s.sheph...@gmail.com> wrote:
>> >
>> > > I'll fix that.
>> > >
>> > > Darren
>> > >
>> > > On Fri, Oct 25, 2013 at 1:54 PM, Alex Huang 
>> > wrote:
>> > > > I'm getting this failing unit test when building with the latest
>> > > > from
>> > > master.  Anyone working on it or know what it is already?  From the
>> > > stack, it looks like it's a problem location the jdbc driver.  This
>> > > was working just yesterday.
>> > > >
>> > > > Test set: com.cloud.alert.AlertControlsUnitTest
>> > > >
>> > > --
>> > > -
>> > > > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> > > > 0.175
>> > > sec <<< FAILURE!
>> > > > warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<<
>> > > FAILURE!
>> > > > junit.framework.AssertionFailedError: Exception in constructor:
>> > > testInjected (java.lang.ExceptionInInitializerError
>> > > > at
>> > > com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.jav
>> > > a:46)
>> > > > at
>> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > > Method)
>> > > > at
>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
>> > > rAccessorImpl.java:57)
>> > > > at
>> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
>> > > nstructorAccessorImpl.java:45)
>> > > > at
>> > > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> > > > at junit.framework.TestSuite.createTest(TestSuite.java:61)
>> > > > at
>> junit.framework.TestSuite.addTestMethod(Te

Re: CS-Management HA Networking

[Darren Shepherd]

I was thinking the same thing.  Right now there's really not much logic around 
it, just grab the first one.  

One problem that may arise has to do with initialization order.  It may be that 
the time that the MAC address is read  database access may not be available.  
Regardless it should be deterministic regardless of OS interface order.  I'll 
look at the code probably sometime next week. 

Darren

> On Oct 26, 2013, at 11:09 AM, Marty Sweet  wrote:
> 
> Possibly, I would say it makes more sense to find what interface the
> cluster.node.ip (I think) is using, then get the MAC address from that
> interface?
> Then if users add interfaces and change the order through udev it will be
> persistent (to an extent).
> 
> The only problem I can think of is if it's using loopback, which it
> shouldn't do if management setup is run?
> 
> Marty
> 
> 
> On Sat, Oct 26, 2013 at 6:08 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
> 
>> Glad that helped.  Seems that we should change CloudStack to ignore mac
>> addresses that are 00:..:00.  If you want to put in a bug you can assign it
>> me and I'll look into changing that.
>> 
>> Darren
>> 
>>> On Oct 26, 2013, at 5:35 AM, Marty Sweet  wrote:
>>> 
>>> Hi Darren, thanks for the heads up about that script.
>>> 
>>> Old Networking Setup:
>>> eth0 eth1 -> management0
>>> management0.11 -> vlan11
>>> management0.12 -> vlan12
>>> 
>>> Turns out in true Ubuntu Networking fashion bond0 was being created for
>> no
>>> reason and was appearing in ifconfig -a (so the script was pulling out
>> the
>>> first mac address it found), although it was not active and could not be
>>> downed.
>>> 
>>> bond0 Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>>> BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
>>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>> collisions:0 txqueuelen:0
>>> RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>>> 
>>> Under this configuration the script returned:
>>> addr in integer is 0
>>> addr in bytes is  0 0 0 0 0 0
>>> addr in char is 00:00:00:00:00:00
>>> 
>>> 
>>> Once I used bond0 as my bond name, opposed to management0, it started
>>> working, as the bond was now in use.
>>> Old Networking Setup:
>>> eth0 eth1 -> bond0
>>> bond0.11 -> vlan11
>>> bond0.12 -> vlan12
>>> 
>>> Many thanks,
>>> Marty
>>> 
>>> 
>>>> On Wed, Oct 23, 2013 at 7:44 AM, Marty Sweet 
>> wrote:
>>>> 
>>>> Hi Darren,
>>>> 
>>>> Thanks for getting back to me. I will set the networking config up again
>>>> and run the commands you sent me over the next couple of days.
>>>> 
>>>> Thanks,
>>>> Marty
>>>> 
>>>> 
>>>> On Tue, Oct 22, 2013 at 11:39 PM, Darren Shepherd <
>>>> darren.s.sheph...@gmail.com> wrote:
>>>> 
>>>>> Well that wasn't very useful message.  If you can find the cloud-utils
>>>>> jar on your server run
>>>>> 
>>>>> java -cp /cloud-utils-4.1.1.jar com.cloud.utils.net.MacAddress
>>>>> 
>>>>> That will output what its finding for the mac address.  Also run an
>>>>> "ifconfig -a" from the command line.  If you won't mind sending the
>>>>> output of "ifconfig -a" that would be helpful to see what's going
>>>>> wrong.
>>>>> 
>>>>> Darren
>>>>> 
>>>>> On Tue, Oct 22, 2013 at 2:48 PM, Marty Sweet 
>>>>> wrote:
>>>>>> Just noticed I didn't include the log:
>>>>>> 
>>>>>> http://pastebin.com/wUtCsSAb
>>>>>> 
>>>>>> Marty
>>>>>> 
>>>>>> 
>>>>>>> On Tue, Oct 22, 2013 at 10:38 PM, Marty Sweet 
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi Darren,
>>>>>>> 
>>>>>>> Maybe I'm getting confused with an issue I had with the Agents around
>>>>> that
>>>>>>> time!
>>>>>>> The error message I got was very cryptic. Having a fresh look at the
>>>>>>> source code

Re: CS-Management HA Networking

[Darren Shepherd]

Glad that helped.  Seems that we should change CloudStack to ignore mac 
addresses that are 00:..:00.  If you want to put in a bug you can assign it me 
and I'll look into changing that.

Darren

> On Oct 26, 2013, at 5:35 AM, Marty Sweet  wrote:
> 
> Hi Darren, thanks for the heads up about that script.
> 
> Old Networking Setup:
> eth0 eth1 -> management0
> management0.11 -> vlan11
> management0.12 -> vlan12
> 
> Turns out in true Ubuntu Networking fashion bond0 was being created for no
> reason and was appearing in ifconfig -a (so the script was pulling out the
> first mac address it found), although it was not active and could not be
> downed.
> 
> bond0 Link encap:Ethernet  HWaddr 00:00:00:00:00:00
>  BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
>  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>  collisions:0 txqueuelen:0
>  RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> Under this configuration the script returned:
> addr in integer is 0
> addr in bytes is  0 0 0 0 0 0
> addr in char is 00:00:00:00:00:00
> 
> 
> Once I used bond0 as my bond name, opposed to management0, it started
> working, as the bond was now in use.
> Old Networking Setup:
> eth0 eth1 -> bond0
> bond0.11 -> vlan11
> bond0.12 -> vlan12
> 
> Many thanks,
> Marty
> 
> 
>> On Wed, Oct 23, 2013 at 7:44 AM, Marty Sweet  wrote:
>> 
>> Hi Darren,
>> 
>> Thanks for getting back to me. I will set the networking config up again
>> and run the commands you sent me over the next couple of days.
>> 
>> Thanks,
>> Marty
>> 
>> 
>> On Tue, Oct 22, 2013 at 11:39 PM, Darren Shepherd <
>> darren.s.sheph...@gmail.com> wrote:
>> 
>>> Well that wasn't very useful message.  If you can find the cloud-utils
>>> jar on your server run
>>> 
>>> java -cp /cloud-utils-4.1.1.jar com.cloud.utils.net.MacAddress
>>> 
>>> That will output what its finding for the mac address.  Also run an
>>> "ifconfig -a" from the command line.  If you won't mind sending the
>>> output of "ifconfig -a" that would be helpful to see what's going
>>> wrong.
>>> 
>>> Darren
>>> 
>>> On Tue, Oct 22, 2013 at 2:48 PM, Marty Sweet 
>>> wrote:
>>>> Just noticed I didn't include the log:
>>>> 
>>>> http://pastebin.com/wUtCsSAb
>>>> 
>>>> Marty
>>>> 
>>>> 
>>>>> On Tue, Oct 22, 2013 at 10:38 PM, Marty Sweet 
>>>> wrote:
>>>> 
>>>>> Hi Darren,
>>>>> 
>>>>> Maybe I'm getting confused with an issue I had with the Agents around
>>> that
>>>>> time!
>>>>> The error message I got was very cryptic. Having a fresh look at the
>>>>> source code:
>>> https://github.com/apache/cloudstack/blob/04cdd90a84f4be5ba02778fe0cd352a4b1c39a13/utils/src/org/apache/cloudstack/utils/identity/ManagementServerNode.java
>>>>> 
>>>>> Would suggest that it gets: private static final long s_nodeId =
>>>>> MacAddress.getMacAddress().toLong(); and ensures it's <=0 in the
>>> check()
>>>>> function, which is run by the SystemIntegrityChecker.
>>>>> 
>>>>> Hopefully it is just a MAC Address issue, what would the
>>> IntegrityChecker
>>>>> be looking for?
>>>>> 
>>>>> Thanks,
>>>>> Marty
>>>>> 
>>>>> 
>>>>> On Tue, Oct 22, 2013 at 10:02 PM, Darren Shepherd <
>>>>> darren.s.sheph...@gmail.com> wrote:
>>>>> 
>>>>>> Do you have a specific error from a log?  I was not aware that
>>>>>> CloudStack would look for interfaces w/ eth*, em*.  In the code it
>>>>>> just does "ifconfig -a" to list the devices.  By creating a bond, the
>>>>>> mac address CloudStack finds will probably change then I could imagine
>>>>>> something could possibly fail.
>>>>>> 
>>>>>> Darren
>>>>>> 
>>>>>> On Tue, Oct 22, 2013 at 1:39 PM, Marty Sweet 
>>>>>> wrote:
>>>>>>> Hi Guys.
>>>>>>> 
>>>>>>> I am planning on upgrading my 4.1.1 infrastructure to 4.2 over the
>>>>>> weekend.
>>>>>>> 
>>&g

Re: failing unit tests....

[Darren Shepherd]

Laszlo,

You want to revert your commit.  I caused the original issue.  I moved
the initialization of LockMasterListener to be a class member to work
around a different intiailization issue somebody reported.  It was a
hasty change and I shouldn't have done it like that.  So that broke
the unit tests.  I did the proper change and checked it in.  I'll
watch the builds to make sure everything is green again.

I'd prefer not to have mysql-java at test scope for anything to
specifically avoid things like this.  The way that I ran the unit
tests was not really proper and it added the mysql driver to the
classpath which ended up connecting to the DB, so I didn't notice the
issue.  If I had ran the proper mvn build it would have failed with
the issue saying it can't find the mysql driver.  For unit tests we
obviously don't want them hitting the DB and not having the driver on
the classpath helps prevent accidentally connecting to the DB on
desktops where you might have a local MySQL.

Darren

On Fri, Oct 25, 2013 at 3:19 PM, Laszlo Hornyak
 wrote:
> I have just sent a fix for that, looks like everything is green now. Please
> check!
>
>
> On Fri, Oct 25, 2013 at 11:52 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> I'll fix that.
>>
>> Darren
>>
>> On Fri, Oct 25, 2013 at 1:54 PM, Alex Huang  wrote:
>> > I'm getting this failing unit test when building with the latest from
>> master.  Anyone working on it or know what it is already?  From the stack,
>> it looks like it's a problem location the jdbc driver.  This was working
>> just yesterday.
>> >
>> > Test set: com.cloud.alert.AlertControlsUnitTest
>> >
>> ---
>> > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.175
>> sec <<< FAILURE!
>> > warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<<
>> FAILURE!
>> > junit.framework.AssertionFailedError: Exception in constructor:
>> testInjected (java.lang.ExceptionInInitializerError
>> > at
>> com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.java:46)
>> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> > at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> > at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> > at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> > at junit.framework.TestSuite.createTest(TestSuite.java:61)
>> > at junit.framework.TestSuite.addTestMethod(TestSuite.java:294)
>> > at
>> junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
>> > at junit.framework.TestSuite.(TestSuite.java:129)
>> > at
>> org.junit.internal.runners.JUnit38ClassRunner.(JUnit38ClassRunner.java:71)
>> > at
>> org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:14)
>> > at
>> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>> > at
>> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>> > at
>> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>> > at
>> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>> > at
>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
>> > at
>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>> > at
>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>> > at
>> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>> > at
>> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderF

Re: failing unit tests....

[Darren Shepherd]

I'll fix that.

Darren

On Fri, Oct 25, 2013 at 1:54 PM, Alex Huang  wrote:
> I'm getting this failing unit test when building with the latest from master. 
>  Anyone working on it or know what it is already?  From the stack, it looks 
> like it's a problem location the jdbc driver.  This was working just 
> yesterday.
>
> Test set: com.cloud.alert.AlertControlsUnitTest
> ---
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.175 sec <<< 
> FAILURE!
> warning(junit.framework.TestSuite$1)  Time elapsed: 0.004 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: Exception in constructor: testInjected 
> (java.lang.ExceptionInInitializerError
> at 
> com.cloud.alert.AlertControlsUnitTest.(AlertControlsUnitTest.java:46)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at junit.framework.TestSuite.createTest(TestSuite.java:61)
> at junit.framework.TestSuite.addTestMethod(TestSuite.java:294)
> at junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
> at junit.framework.TestSuite.(TestSuite.java:129)
> at 
> org.junit.internal.runners.JUnit38ClassRunner.(JUnit38ClassRunner.java:71)
> at 
> org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:14)
> at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
> at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
> at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
> at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to 
> initialize a connection to the database for locking purposes:
> at com.cloud.utils.db.Merovingian2.(Merovingian2.java:74)
> at 
> com.cloud.utils.db.Merovingian2.createLockMaster(Merovingian2.java:80)
> at 
> com.cloud.server.LockMasterListener.(LockMasterListener.java:33)
> at 
> com.cloud.server.ManagementServerImpl.(ManagementServerImpl.java:602)
> ... 27 more
> Caused by: java.sql.SQLException: No suitable driver found for 
> jdbc:mysql://localhost:3306/cloud?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096
> at java.sql.DriverManager.getConnection(DriverManager.java:596)
> at java.sql.DriverManager.getConnection(DriverManager.java:215)
> at 
> org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
> at 
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>
> --Alex

Re: Metadata URL location

[Darren Shepherd]

I think it should be fine.  Conceptually, 169.254/16 is not supposed
to be routed.  The standard protocol is that you can randomly assign
yourself an IP as long as nobody responds to ARP.  So the definition
of the subnet is that it's your local subnet.  EC2 is funky though.
If you launch an amazon linux AMI it will come up with the following
route.

169.254.169.254 0.0.0.0 255.255.255.255 UH0  00 eth0

Now it really don't matter if that route exists because ARP is useless
in EC2.  If you look at your ARP cache (or do arping), you'll notice
everything is FE:FF:FF:FF:FF:FF.  So all packets leaving your VM go to
the same place and then EC2 magic ensues

Darren

On Fri, Oct 25, 2013 at 7:51 AM, Chiradeep Vittal
 wrote:
> Hmm. Windows may not work then? Needs testing
>
> On 10/24/13 5:04 PM, "Darren Shepherd"  wrote:
>
>>Chiradeep,
>>
>>Linux distros set 169.254/16 route on the primary interface.  It's just
>>there now.  I'm not sure if that's because of ec2 or if it's always been
>>that way, but all modern distros will assign it if you have a standard
>>base install.
>>
>>In the VPC case I think we would have to use network namespaces to bind
>>the same IP to multiple NICs.  You could bind the IP to loopback, but
>>then it won't ARP.  So since linux distros already have the route, they
>>won't send to the gateway.
>>
>>Yes systemvm will need to be allowed to talk over 169.254, so that's
>>needs to change.
>>
>>Again, I said the reason I thought this wasn't done is because it's hard.
>> But still doable.  I'd like to see us do this change.  At least in the
>>KVM case, it would be real nice to be able to pickup the standard ubuntu
>>(or fedora) cloud qcow and have it run in CloudStack.
>>
>>Darren
>>
>>> On Oct 24, 2013, at 3:57 PM, Chiradeep Vittal
>>> wrote:
>>>
>>> For the VPC virtual router case this would this have to be done on all
>>> guest interfaces?
>>> Could we alias localhost on the VR to 169.254.169.254?
>>> For shared networks, basic zone and networks where the VR is not the
>>> default gateway, we would have to send another (169.254.0.0/16) route in
>>> the DHCP response to point to the VR.
>>>
>>> Additionally in basic zone the anti-spoof filters in dom0 would have to
>>>be
>>> adjusted to let 169.254.169.254 addresses
>>>
>>>> On 10/24/13 8:51 AM, "Darren Shepherd" 
>>>>wrote:
>>>>
>>>> So additionally you need to do
>>>>
>>>> ip addr add dev eth0 169.254.169.254/0
>>>>
>>>> On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren
>>>>
>>>> wrote:
>>>>> You would also need to supernet 169.254.169.254 on the virtual router
>>>>> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it
>>>>> will
>>>>> still arp to servers that are calling it that have real ip addresses.
>>>>>
>>>>> Additionally we had some other iptables rules in there that would
>>>>>change
>>>>> the the ip address of the incoming request to metadata based upon the
>>>>> mac
>>>>> address that was hitting it.  This was to prevent spoofing of another
>>>>> vm's
>>>>> IP and getting someone else's metadata (at least in our metadata
>>>>> implementation we keyed off of the VM IP calling into metadata).  This
>>>>> also allowed a user to set whatever ipaddress they wanted, but as long
>>>>> as
>>>>> the mac address was the same and they still had a zeroconfig route on
>>>>> the
>>>>> VM, they still got only their metadata.
>>>>> 
>>>>>
>>>>> Kris Lindgren
>>>>> Senior Linux Systems Engineer
>>>>> GoDaddy, LLC.
>>>>>
>>>>>
>>>>> This email message and any attachment(s) hereto are intended for use
>>>>> only
>>>>> by its intended recipient(s) and may contain confidential information.
>>>>> If
>>>>> you have received this email in error, please immediately notify the
>>>>> sender and permanently delete the original and any copy of this
>>>>>message
>>>>> and its attachments.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>&g

Re: About component.xml and applicationContext.xml.in

[Darren Shepherd]

Refer to
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions

In short your can either add/remove jars from the classpath or use the
*exclude properties to disable elements.  Exclude properties are
usually the easiest as most everything is enabled by default.

Darren

On Thu, Oct 24, 2013 at 5:17 PM, Rayees Namathponnan
 wrote:
> In 4.2,  component.xml and applicationContext.xml.in using to load required 
> plugin in management server.
>
> component.xml and applicationContext.xml.in are removed from master branch ?  
> this case,   where I can enable or disable plugin from cloudstack ?
>
> Regards,
> Rayees
>

Re: Anyone ever get this exception?

[Darren Shepherd]

It was simple enough I committed
d178b25daa484dda44bb34e63cb719d1c3cc1519 for this.

Darren

On Thu, Oct 24, 2013 at 4:07 PM, Mike Tutkowski
 wrote:
> OK, thanks, Darren.
>
> If it becomes larger than a "simple" fix, I can log a JIRA ticket for it,
> too.
>
>
> On Thu, Oct 24, 2013 at 4:19 PM, Darren Shepherd <
> darren.s.sheph...@gmail.com> wrote:
>
>> Never seen that, but yeah that's an issue.  It's possible given the
>> current initialization in ACS to somebody tries to lock something
>> before the lock manager is created.  That should change let me
>> look at that.  I can probably fix it on master.
>>
>> Darren
>>
>> On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
>>  wrote:
>> > Hi,
>> >
>> > I'm running in master.
>> >
>> > I had to reboot the only host (KVM) in my CloudStack dev environment. As
>> > you'd expect, it was running my system VMs.
>> >
>> > At the same time, I had to restart my CS MS.
>> >
>> > I've noticed this exception before when I first start up the CS MS and it
>> > has to restart SSVM. SSVM doesn't start and I have to restart the CS MS
>> to
>> > get it to work (usually starts up SSVM the second time without any
>> > exceptions):
>> >
>> > WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
>> > exception There's no support for locking yet
>> > com.cloud.utils.exception.CloudRuntimeException: There's no support for
>> > locking yet
>> > at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
>> > at
>> >
>> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
>> > at
>> >
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>> > at
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
>> > at
>> >
>> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
>> > at
>> >
>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
>> > at
>> >
>> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
>> > at
>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
>> > at
>> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
>> > at
>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
>> > at
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
>> > at
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > at
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > at
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>

Re: Metadata URL location

[Darren Shepherd]

Chiradeep,

Linux distros set 169.254/16 route on the primary interface.  It's just there 
now.  I'm not sure if that's because of ec2 or if it's always been that way, 
but all modern distros will assign it if you have a standard base install.

In the VPC case I think we would have to use network namespaces to bind the 
same IP to multiple NICs.  You could bind the IP to loopback, but then it won't 
ARP.  So since linux distros already have the route, they won't send to the 
gateway.  

Yes systemvm will need to be allowed to talk over 169.254, so that's needs to 
change.

Again, I said the reason I thought this wasn't done is because it's hard.  But 
still doable.  I'd like to see us do this change.  At least in the KVM case, it 
would be real nice to be able to pickup the standard ubuntu (or fedora) cloud 
qcow and have it run in CloudStack.

Darren

> On Oct 24, 2013, at 3:57 PM, Chiradeep Vittal  
> wrote:
> 
> For the VPC virtual router case this would this have to be done on all
> guest interfaces? 
> Could we alias localhost on the VR to 169.254.169.254?
> For shared networks, basic zone and networks where the VR is not the
> default gateway, we would have to send another (169.254.0.0/16) route in
> the DHCP response to point to the VR.
> 
> Additionally in basic zone the anti-spoof filters in dom0 would have to be
> adjusted to let 169.254.169.254 addresses
> 
>> On 10/24/13 8:51 AM, "Darren Shepherd"  wrote:
>> 
>> So additionally you need to do
>> 
>> ip addr add dev eth0 169.254.169.254/0
>> 
>> On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren 
>> wrote:
>>> You would also need to supernet 169.254.169.254 on the virtual router
>>> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it
>>> will
>>> still arp to servers that are calling it that have real ip addresses.
>>> 
>>> Additionally we had some other iptables rules in there that would change
>>> the the ip address of the incoming request to metadata based upon the
>>> mac
>>> address that was hitting it.  This was to prevent spoofing of another
>>> vm's
>>> IP and getting someone else's metadata (at least in our metadata
>>> implementation we keyed off of the VM IP calling into metadata).  This
>>> also allowed a user to set whatever ipaddress they wanted, but as long
>>> as
>>> the mac address was the same and they still had a zeroconfig route on
>>> the
>>> VM, they still got only their metadata.
>>> 
>>> 
>>> Kris Lindgren
>>> Senior Linux Systems Engineer
>>> GoDaddy, LLC.
>>> 
>>> 
>>> This email message and any attachment(s) hereto are intended for use
>>> only
>>> by its intended recipient(s) and may contain confidential information.
>>> If
>>> you have received this email in error, please immediately notify the
>>> sender and permanently delete the original and any copy of this message
>>> and its attachments.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 10/24/13 9:12 AM, "Darren Shepherd" 
>>> wrote:
>>> 
>>>> My guess, I don't really know, would be because its hard.  The VR uses
>>>> link local for the control network so 169.254/16 is bound to the wrong
>>>> nic.  To fix this you just need some ip routing magic in linux (credit
>>>> goes to Kris Lindgren who showed me how to do this).  Add the below to
>>>> a file, substitute eth0 for the guest network nic, run "ip -b "
>>>> The below effectively creates a routing table dedicated to the IP
>>>> 169.254.169.254 that sets it default route to go out the guest network
>>>> nic.
>>>> 
>>>> rule add from 169.254.169.254 table 70
>>>> rule add to 169.254.169.254 dev eth0 table 70
>>>> route flush table 70
>>>> route add default dev eth0 src 169.254.169.254 table 70
>>>> route flush cache
>>>> 
>>>> Darren
>>>> 
>>>> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>>>>  wrote:
>>>>> Hi Guys,
>>>>> 
>>>>> CloudStack metadata services are on the default gateway while on EC2,
>>>>> its at 169.254.169.254. Am curious to know why CloudStack does not
>>>>> use a link local address for meta data services.
>>>>> 
>>>>> A search of the Wiki
>>>>> (https:

Re: how to use hashes on c.a.o?

[Darren Shepherd]

> Is there a reason not to verify authenticity and validity of the
> tarball with the gpg sig?

Nope, just laziness.

Darren

Re: Anyone ever get this exception?

[Darren Shepherd]

Never seen that, but yeah that's an issue.  It's possible given the
current initialization in ACS to somebody tries to lock something
before the lock manager is created.  That should change let me
look at that.  I can probably fix it on master.

Darren

On Thu, Oct 24, 2013 at 11:19 AM, Mike Tutkowski
 wrote:
> Hi,
>
> I'm running in master.
>
> I had to reboot the only host (KVM) in my CloudStack dev environment. As
> you'd expect, it was running my system VMs.
>
> At the same time, I had to restart my CS MS.
>
> I've noticed this exception before when I first start up the CS MS and it
> has to restart SSVM. SSVM doesn't start and I have to restart the CS MS to
> get it to work (usually starts up SSVM the second time without any
> exceptions):
>
> WARN  [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-efae49e6) Unexpected
> exception There's no support for locking yet
> com.cloud.utils.exception.CloudRuntimeException: There's no support for
> locking yet
> at com.cloud.utils.db.Transaction.lock(Transaction.java:386)
> at
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1001)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:992)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:596)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:588)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.createSecStorageVmInstance(SecondaryStorageManagerImpl.java:561)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startNew(SecondaryStorageManagerImpl.java:489)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:667)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:101)
> at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:78)
> at
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:71)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
> Thanks!
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud
> *™*

Re: is marvin on master broken?

[Darren Shepherd]

Ah thanks, that's my problem.  I removed the logger portion not really
thinking about it.  Oh the joys of DSLs.

Darren

On Thu, Oct 24, 2013 at 11:08 AM, Santhosh Edukulla
 wrote:
> Hi Darren,
>
> 1. I just ran again using  the latest nosetests-2.7 with  marvin-plugin, 
> marvin-config=, it went fine with 
> no issues.
>
> 2. Please check whether the below contents are available under configuration 
> file passed to marvin or nosetests. Check advanced.cfg under ( setup/dev/ ) 
> for reference. In short, it seems as per trace, under loadCfg , 
> testClientLogFile which is getting loaded based upon below values from config 
> is None, leading to further issues and so below trace.
>
> For passing the correct config :
>
>  If we are using nosetests to run the marvin tests, please use 
> --marvin-config=""  ( or  )
>
>  If we are using deployAndRun, please use --config ="".
>
> "logger": [
> {
> "name": "TestClient",
> "file": "/tmp/testclient.log"
> },
> {
> "name": "TestCase",
> "file": "/tmp/testcase.log"
> }
> ],
>
> 3. If issue persists for some reason, please let us know.
>
> 4.  Ideally, marvin should have gracefully handled to continue further or 
> gracefully exited with relevant help text to console if there are any 
> dependencies. It seems there were other checks also missing.  In any case, we 
> are planning to do some changes to logging facility under marvin.
>
> Thanks!
> Santhosh
> 
> From: Darren Shepherd [darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 11:00 AM
> To: dev@cloudstack.apache.org
> Subject: Re: is marvin on master broken?
>
> Traceback (most recent call last):
>   File "tools/marvin/marvin/deployDataCenter.py", line 610, in 
> deploy.deploy()
>   File "tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
> self.loadCfg()
>   File "tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>   File "tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 505, in listUsers
> response = self.connection.marvinRequest(command,
> response_type=response, method=method)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
> line 269, in marvinRequest
> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
> AttributeError: 'NoneType' object has no attribute 'debug'
>
> On Thu, Oct 24, 2013 at 1:21 AM, Santhosh Edukulla
>  wrote:
>> Hello Darren,
>>
>> Some trace is still missing i believe. i could not see the last stack frame 
>> in the below trace as what lead to this trace?
>>
>> I just pulled the latest from master branch and used marvin  to deploy few 
>> cloudstack entities and it worked. Can you please provide the command you 
>> are using to run marvin tests? or what command lead to the below trace?
>>
>> Regards,
>> Santhosh
>> 
>> From: Darren Shepherd [darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 24, 2013 3:48 AM
>> To: dev@cloudstack.apache.org
>> Subject: is marvin on master broken?
>>
>> Whenever I use marvin on master I get
>>
>> Traceback (most recent call last):
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
>> deploy.deploy()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
>> self.loadCfg()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
>> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>>   File "./tools/marvin/marvin/deployDataCenter.py", line 492, in 
>> registerApiKey
>> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>>   File 
>> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
>> line 505, in listUsers
>> response = self.connection.marvinRequest(command,
>> response_type=response, method=method)
>>   File 
>> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
>> line 269, in marvinRequest
>> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
>>
>> So it looks like cloudConnection needs a logger passed to the
>> constructor and its not being passed?  I've just been hacking up
>> ./tools/marvin/marvin/cloudstackConnection.py to work around this.
>>
>> Darren

Review Request 14914: Make commands.properties optional for non-ACS code

[Darren Shepherd]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14914/
---

Review request for cloudstack and Chris Suich.


Repository: cloudstack-git


Description
---

Currently any new API extension to CloudStack must edit
commands.properties to add the appropriate ACLs.  This generally works
fine for ACS as we control the contents of that file and distribute
all the code ourself.  The hang up comes when somebody develops code
outside of ACS and want to add their code to an existing ACS
installation.  The Spring work that has been done has made this much
easier, but you are still required to manually edit
commands.properties.  This change introduces the following logic.

First check commands.properties for ACL info.  If ACL info exists, use
that to authorize the command.  If no ACL information exists (ie
null), then look at the @APICommand annotation.  The defaults of


Diffs
-

  api/src/org/apache/cloudstack/api/APICommand.java 4d024c1 
  
plugins/acl/static-role-based/src/org/apache/cloudstack/acl/StaticRoleBasedAPIAccessChecker.java
 affcf91 

Diff: https://reviews.apache.org/r/14914/diff/


Testing
---


Thanks,

Darren Shepherd

Re: how to use hashes on c.a.o?

[Darren Shepherd]

I ran into the issue while trying to write scripts to get CloudStack
in docker.  Here's my hack-tastic work around

HASH=$(echo $(wget -O - -q
http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
|\
 cut -f2 -d: ) | sed 's/ //g' | tr '[:upper:]' '[:lower:]')

Darren

On Thu, Oct 24, 2013 at 3:05 PM, David Nalley  wrote:
> On Thu, Oct 24, 2013 at 5:43 PM, Darren Shepherd
>  wrote:
>> Chip,
>>
>> Do you care if we switch to GNU coreutils format for the hashes?  The
>> hash value is the same it will just be in the format like
>>
>> file.tbz2 *12b12341b1234b1234b1b2341b234b
>>
>> And then you just run "sha512sum -c "
>>
>> Darren
>>
>
> I just ran into this problem writing an ansible playbook to build RPM
> packages. md5sum -c and sha256sum -c don't seem to work with the
> format that we provide which seems odd.
>
> Incidentally the download page no longer has the documentation for
> verifying hashes.
>
> --David

Re: [PROPOSAL] Enable "non-ACS distributed" API extension to be added CloudStack w/o modifying commands.properties

[Darren Shepherd]

Chris,

This change was simple enough I just put up a review for it
https://reviews.apache.org/r/14914/  You want to download the patch
and see if that works for you?

Darren

On Thu, Oct 24, 2013 at 2:28 PM, Darren Shepherd
 wrote:
> Currently any new API extension to CloudStack must edit
> commands.properties to add the appropriate ACLs.  This generally works
> fine for ACS as we control the contents of that file and distribute
> all the code ourself.  The hang up comes when somebody develops code
> outside of ACS and want to add their code to an existing ACS
> installation.  The Spring work that has been done has made this much
> easier, but you are still required to manually edit
> commands.properties.  I propose that we add the ACL metadata as an
> optional field in @APICommand.  The logic will be as follows.
>
> First check commands.properties for ACL info.  If ACL info exists, use
> that to authorize the command.  If no ACL information exists (ie
> null), then look at the @APICommand annotation.  The defaults of
> @APICommand will provide no ACL info.  If the @APICommand annotation
> provides no ACL info, use that.
>
> Effectively what this means is that for all existing "ACS distributed"
> code @APICommand will provide no ACL info (as the default is none) so
> commands.properties will be used.  If somebody extends ACS, they can
> set the ACL info in @APICommand.
>
> The scope of proposal is focused on "non-ACS" distributed code.  In
> order for ACS to move to the annotations fully and make
> commands.properties optional more things need to change.  Specifically
> the documentation and some python utilities look at
> command.properties.  It would be a nice future enhancement to make
> commands.properties fully optional, but for the time being this
> proposal will address the core issue.
>
> Darren

Re: Metadata URL location

[Darren Shepherd]

Alex,

I don't think that's correct.  The instructions that I gave in the
earlier email was changes to the VR to serve metadata from the VR.
Regardless of hypervisor, it should work.

Darren

On Thu, Oct 24, 2013 at 10:54 AM, Alex Huang  wrote:
> In order to use an link local address inside the end user vm, that metadata 
> service must be setup on every hypervisor's dom0 or it has to be proxied out 
> of the dom0.  That's not doable for VmWare.  Instead, CloudStack uses VR to 
> serve the data, which works for all three hypervisors.
>
> --Alex
>
>> -Original Message-
>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>> Sent: Thursday, October 24, 2013 8:13 AM
>> To: dev@cloudstack.apache.org
>> Cc: klindg...@godaddy.com
>> Subject: Re: Metadata URL location
>>
>> My guess, I don't really know, would be because its hard.  The VR uses link
>> local for the control network so 169.254/16 is bound to the wrong nic.  To 
>> fix
>> this you just need some ip routing magic in linux (credit goes to Kris 
>> Lindgren
>> who showed me how to do this).  Add the below to a file, substitute eth0 for
>> the guest network nic, run "ip -b "
>> The below effectively creates a routing table dedicated to the IP
>> 169.254.169.254 that sets it default route to go out the guest network nic.
>>
>> rule add from 169.254.169.254 table 70
>> rule add to 169.254.169.254 dev eth0 table 70 route flush table 70 route add
>> default dev eth0 src 169.254.169.254 table 70 route flush cache
>>
>> Darren
>>
>> On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>>  wrote:
>> > Hi Guys,
>> >
>> > CloudStack metadata services are on the default gateway while on EC2,
>> > its at 169.254.169.254. Am curious to know why CloudStack does not use
>> > a link local address for meta data services.
>> >
>> > A search of the Wiki
>> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDST
>> ACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&
>> spaceSearch=true&queryString=metadata) didn't seem to list any doc
>> related to the design of this service.
>> >
>> > TIA.
>> >
>> > --
>> > @shankerbalan
>> >
>> > M: +91 98860 60539 | O: +91 (80) 67935867 shanker.ba...@shapeblue.com
>> > | www.shapeblue.com | Twitter:@shapeblue ShapeBlue Services India LLP,
>> > 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
>> >
>> > CloudStack Bootcamp Training on 27/28 November, Bangalore
>> > http://www.shapeblue.com/cloudstack-training/
>> >
>> >
>> >
>> >
>> > This email and any attachments to it may be confidential and are intended
>> solely for the use of the individual to whom it is addressed. Any views or
>> opinions expressed are solely those of the author and do not necessarily
>> represent those of Shape Blue Ltd or related companies. If you are not the
>> intended recipient of this email, you must neither take any action based
>> upon its contents, nor copy or show it to anyone. Please contact the sender 
>> if
>> you believe you have received this email in error. Shape Blue Ltd is a
>> company incorporated in England & Wales. ShapeBlue Services India LLP is a
>> company incorporated in India and is operated under license from Shape
>> Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
>> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a
>> registered trademark.

Re: how to use hashes on c.a.o?

[Darren Shepherd]

Chip,

Do you care if we switch to GNU coreutils format for the hashes?  The
hash value is the same it will just be in the format like

file.tbz2 *12b12341b1234b1234b1b2341b234b

And then you just run "sha512sum -c "

Darren

On Thu, Oct 24, 2013 at 2:34 PM, John Kinsella  wrote:
> Instructions for testing the hash are in the release test page [1]. It is 
> also documented in the install guide.
>
> It is the way it is I believe because Chip took the release build script from 
> CouchDB, as mentioned in the release build page.
>
> 1: 
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+procedure
>
> On Oct 24, 2013, at 12:53 AM, Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>> wrote:
>
> But how does one validate it?  I just wrote a dumb script to
> concatenation, remove whitespace, lowercase and then pass to
> "sha512sum -c."  I've never seen anyone provide SHAs in that format.
> I wouldn't expect many people to know how to use them.  Why can't we
> use the good old GNU coreutils style?
>
> Darren
>
> On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella 
> mailto:j...@stratosec.co>> wrote:
> This is the output of gpg -v --print-md SHA512, generated as part of the 
> release procedure [1] by tools/build/build_asf.sh
>
> 1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure
>
>
> On Oct 17, 2013, at 7:56 PM, Darren Shepherd 
> mailto:darren.s.sheph...@gmail.com>> wrote:
>
> The hashes that are on c.a.o for the releases have a format like
>
> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
>
> apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
>684E0094 F1666F57 5D694916 CF74ED98
>9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
>EA82D42C 78880EDB 04F2532A 61376537
>
> I've never seen this.  Is this some hip new format I'm not aware of,
> and I'm the uncool kid still using GNU coreutils?
>
> Darren
>
>
>
>
> Stratosec<http://stratosec.co/> - Compliance as a Service
> o: 415.315.9385
> @johnlkinsella<http://twitter.com/johnlkinsella>
>

[PROPOSAL] Enable "non-ACS distributed" API extension to be added CloudStack w/o modifying commands.properties

[Darren Shepherd]

Currently any new API extension to CloudStack must edit
commands.properties to add the appropriate ACLs.  This generally works
fine for ACS as we control the contents of that file and distribute
all the code ourself.  The hang up comes when somebody develops code
outside of ACS and want to add their code to an existing ACS
installation.  The Spring work that has been done has made this much
easier, but you are still required to manually edit
commands.properties.  I propose that we add the ACL metadata as an
optional field in @APICommand.  The logic will be as follows.

First check commands.properties for ACL info.  If ACL info exists, use
that to authorize the command.  If no ACL information exists (ie
null), then look at the @APICommand annotation.  The defaults of
@APICommand will provide no ACL info.  If the @APICommand annotation
provides no ACL info, use that.

Effectively what this means is that for all existing "ACS distributed"
code @APICommand will provide no ACL info (as the default is none) so
commands.properties will be used.  If somebody extends ACS, they can
set the ACL info in @APICommand.

The scope of proposal is focused on "non-ACS" distributed code.  In
order for ACS to move to the annotations fully and make
commands.properties optional more things need to change.  Specifically
the documentation and some python utilities look at
command.properties.  It would be a nice future enhancement to make
commands.properties fully optional, but for the time being this
proposal will address the core issue.

Darren

Re: [DISCUSS] make commands.properties the exception, not the rule

[Darren Shepherd]

The spring stuff doesn't help.  This discussion didn't really go
anywhere and there weren't much comments.  I'd say at this point we
just put together a proposal.  Give me 5 minutes and I'll throw up a
proposal in a different thread.

Darren

On Thu, Oct 24, 2013 at 1:47 PM, SuichII, Christopher
 wrote:
> I’d like to see if we can come up with some solution for this issue. Having 
> ‘add these lines to commands.properties’ is not really an acceptable 
> installation step for plugins/extensions/etc.
>
> So, the ideas I’ve seen are:
> -Turn commands.properties into a blacklist instead of a whitelist
> -Dynamically discover additional commands.properties
>
> Darren - is there any chance that the Spring modularization stuff you’ve done 
> would make the latter any easier?
>
> Are there any others?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 9, 2013, at 3:49 PM, SuichII, Christopher  
> wrote:
>
>> I just wanted to add a little clarification from a plugin perspective.
>>
>> Having commands.properties as a whitelist just adds another place that 
>> plugins have to register with CloudStack. For plugins that do not intend on 
>> being a part of the CloudStack source, this is actually quite tricky. 
>> Currently, to add entries to commands.properties, any plugin like this would 
>> either need to tell the CS administrator to manually modify this file (error 
>> prone, laborious and an uncommon installation practice) or develop an 
>> installation script to modify commands.properties when installing, updating 
>> or uninstalling the plugin (also error prone and scary).
>>
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 9, 2013, at 1:08 AM, Darren Shepherd  
>> wrote:
>>
>>> So I'm saying if you want to disable a command you put myBadCmd=0 in
>>> the commands.properties.  So yes, a blacklist over a whitelist.  For
>>> people paranoid about maybe some command exists that they don't know
>>> about, we can even add a "blacklist=false to the command properties.
>>> Then the commands.properites becomes the all mighty master of what is
>>> allowed (a whitelist).  But by default, I think the file should be
>>> empty and default to what is defined by the API annotation.
>>>
>>> Darren
>>>
>>> On Tue, Oct 8, 2013 at 5:45 PM, SuichII, Christopher
>>>  wrote:
>>>> Maybe we could consider switching from a whitelist to a blacklist, then. A 
>>>> whitelist is certainly easier in terms of a one-step configuration, but a 
>>>> blacklist would allow for much easier plugin development, installation and 
>>>> removal. Perhaps we could find write a script that generates the complete 
>>>> list of APIs to create the blacklist from (I know this API exists 
>>>> currently, but not in the format of commands.properties).
>>>>
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 8, 2013, at 7:11 PM, Prachi Damle  wrote:
>>>>
>>>>> I think commands.properties is not just providing ACL on the API - but it 
>>>>> also serves as a whitelist of APIs available on the deployment.
>>>>> It can be a one-step configuration option to disable certain 
>>>>> functionality.
>>>>>
>>>>> Prachi
>>>>>
>>>>>
>>>>> -Original Message-
>>>>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com]
>>>>> Sent: Tuesday, October 08, 2013 3:24 PM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: [DISCUSS] make commands.properties the exception, not the rule
>>>>>
>>>>> I would like to largely remove commands.properties.  I think most API 
>>>>> commands naturally have a default ACL that should be applied.  I think it 
>>>>> makes sense to add to the @APICommand flags for user, domain, admin.  
>>>>> Then, as an override mechanism, people can edit commands.properties to 
>>>>> change the default ACL.  This would make it such that people could add 
>>>>> new commands without the need to edit commands.properties.
>>>>>
>>>>> Thoughts?  How will this play with whatever is being done with rbac?
>>>>>
>>>>> Darren
>>>>
>>
>

Re: Modularized Spring Modules

[Darren Shepherd]

Yes avoid ComponentContext.inject() at all costs.  It is really bad.
The problem with ComponentContext.inject(), besides that it is a bad
programming pattern, is that it doesn't know which spring context you
are in.  So ComponentContext.inject() will inject only beans that are
defined in the "core" context.  So extensions are in a child of "core"
so it will not find beans in the child.  There are some ways to work
around this because API commands heavily rely on
ComponentContext.inject(), but I'd prefer that that not be used.
Basically, just don't use ComponentContext.inject().

This was also why at some point I mentioned that
List can not be inject to a SnapshotObject.  The
SnapshotObject is setup with ComponentContext.inject() which will not
find the SnapshotStrategys because no SnapshotStrategys exist in the
"core" context.

I will have to work with Edison to eventually remove
ComponentContext.inject() from storage in general.

Darren

On Thu, Oct 24, 2013 at 1:40 PM, SuichII, Christopher
 wrote:
> So, I kind of figured it out…
>
> In the past, I was creating my DataStoreLifeCycle with 
> ComponentContext.inject() and it had some members which were @Injecte(d) - 
> those were the guys that were failing to be instantiated. When I switched to 
> simply @Inject my DataStoreLifeCycle into my DataStoreProvider, everything 
> works fine.
>
> So, with this new system, is there some reason that using 
> ComponentContext.inject() would fail to load beans while @Inject would 
> succeed?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 24, 2013, at 4:04 PM, SuichII, Christopher  
> wrote:
>
>> Darren,
>>
>> I’m switching my plugin over to use the new modularized Spring stuff you 
>> just merged and there is something I’m still battling with. I have other 
>> beans that were previously instantiated before my DataStoreProvider which 
>> get injected into the provider, lifecycle, etc. So, those beans need to be 
>> instantiated before the DataStoreProvider. How can I ensure those beans are 
>> created and setup before the DataStoreProvider does?
>>
>> -Chris
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>

Re: Modularized Spring Modules

[Darren Shepherd]

Let me clarify some terms first, this might be confusing.  There is
instantiation and then there is initialization.  Instantiation is
calling the constructor.  Initialization is calling the
@PostConstruct.  Beyond that there is what I've been calling
"CloudStack Extended Lifecycle,"  This for bean implementing
ComponentLifecycle or an extensible interface, like DataStoreProvider.
 Since I'm not sure which aspect your asking about, I'll just explain
the whole process.

Spring is now arranged into a hierarchy of contexts.  The parent
contexts are initialized first, and then the children afterwards.  So
the process goes as follows

1. Instantiate all beans in the current context - There is no way to
deterministically control the instantiation of beans within a context.
 This is just a fundamental fact that can't change.  So beans are
randomly instantiated.
2. Inject all dependencies on all beans in current context - This
entails injecting all the dependencies defined by @Inject.  The order
is not really deterministic which beans get injected first.
3. Initialize all beans in graph order - This entails calling all
methods that have @PostConstruct.  Once the beans are all wired up
they form a graph of dependencies.  Child beans are initialized first,
then the parent.  CloudStack has a lot of circular dependencies in the
beans currently, so while this is intended to be deterministic, you
may find issues.  The storage code is particularly bad in this area,
but if you are defining your beans in your own context then this
doesn't apply.  Only if you add beans to the "core" context will you
find issues.  If a class has an initialization dependency that is not
express through its bean dependencies, you can add "depends-on" to the
bean defintion in the XML.  This will ensure that the specified bean
will be initialized first.
4. CloudStack Extended LifeCycle: Call ComponentLifecycle.configure()
on all ComponentLifecycle implementing beans
5. CloudStack Extended LifeCycle: Register extensible types.  This
phase happens really at the same time as step 4.  When an extensible
type is registered nothing typically happens except that its stored in
a list.  Storage is different in that when the DataStoreProvider is
registered, at that time DataStoreProvider.configure() will be called.
 Notice that even though DataStoreProvider.configure() has the same
signature as ComponentLifecycle.configure() it is different.  Honestly
with the new Spring code we can remove DataStoreProvider completely,
but that's a different discussion

After the context is initialized following the steps above, it will
then initialize all the child contexts.  Once all contexts have been
completely initialized, ComponentLifecycle.start() will be call on all
beans starting with the parent context, then children.

A general guide line of when to use ComponentLifecycle.start() vs
@PostConstruct is if your initialization logic starts a background
thread or requires all extensible types to be registered, then you
should use ComponentLifecycle.start().  All other initialization code
should be in @PostConstruct.  ComponentLifecycle.configure() is
largely useless in my mind and should be avoided.

I hope somewhere in all that is something that answered your question.
 If not you can email me directly with your spring config and I can
help, or we can setup a GTM.

Darren

On Thu, Oct 24, 2013 at 1:04 PM, SuichII, Christopher
 wrote:
> Darren,
>
> I’m switching my plugin over to use the new modularized Spring stuff you just 
> merged and there is something I’m still battling with. I have other beans 
> that were previously instantiated before my DataStoreProvider which get 
> injected into the provider, lifecycle, etc. So, those beans need to be 
> instantiated before the DataStoreProvider. How can I ensure those beans are 
> created and setup before the DataStoreProvider does?
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>

Re: Metadata URL location

[Darren Shepherd]

So additionally you need to do

ip addr add dev eth0 169.254.169.254/0

On Thu, Oct 24, 2013 at 8:29 AM, Kris G. Lindgren  wrote:
> You would also need to supernet 169.254.169.254 on the virtual router
> (assigning it as 169.254.169.254 netmask 0.0.0.0 on eth0) that way it will
> still arp to servers that are calling it that have real ip addresses.
>
> Additionally we had some other iptables rules in there that would change
> the the ip address of the incoming request to metadata based upon the mac
> address that was hitting it.  This was to prevent spoofing of another vm's
> IP and getting someone else's metadata (at least in our metadata
> implementation we keyed off of the VM IP calling into metadata).  This
> also allowed a user to set whatever ipaddress they wanted, but as long as
> the mac address was the same and they still had a zeroconfig route on the
> VM, they still got only their metadata.
> 
>
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
>
> This email message and any attachment(s) hereto are intended for use only
> by its intended recipient(s) and may contain confidential information. If
> you have received this email in error, please immediately notify the
> sender and permanently delete the original and any copy of this message
> and its attachments.
>
>
>
>
>
>
>
> On 10/24/13 9:12 AM, "Darren Shepherd"  wrote:
>
>>My guess, I don't really know, would be because its hard.  The VR uses
>>link local for the control network so 169.254/16 is bound to the wrong
>>nic.  To fix this you just need some ip routing magic in linux (credit
>>goes to Kris Lindgren who showed me how to do this).  Add the below to
>>a file, substitute eth0 for the guest network nic, run "ip -b "
>>The below effectively creates a routing table dedicated to the IP
>>169.254.169.254 that sets it default route to go out the guest network
>>nic.
>>
>>rule add from 169.254.169.254 table 70
>>rule add to 169.254.169.254 dev eth0 table 70
>>route flush table 70
>>route add default dev eth0 src 169.254.169.254 table 70
>>route flush cache
>>
>>Darren
>>
>>On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
>> wrote:
>>> Hi Guys,
>>>
>>> CloudStack metadata services are on the default gateway while on EC2,
>>> its at 169.254.169.254. Am curious to know why CloudStack does not
>>> use a link local address for meta data services.
>>>
>>> A search of the Wiki
>>>(https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK
>>>&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearc
>>>h=true&queryString=metadata) didn¹t seem to list any doc related to the
>>>design of this service.
>>>
>>> TIA.
>>>
>>> --
>>> @shankerbalan
>>>
>>> M: +91 98860 60539 | O: +91 (80) 67935867
>>> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
>>> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade
>>>Centre, Bangalore - 560 055
>>>
>>> CloudStack Bootcamp Training on 27/28 November, Bangalore
>>> http://www.shapeblue.com/cloudstack-training/
>>>
>>>
>>>
>>>
>>> This email and any attachments to it may be confidential and are
>>>intended solely for the use of the individual to whom it is addressed.
>>>Any views or opinions expressed are solely those of the author and do
>>>not necessarily represent those of Shape Blue Ltd or related companies.
>>>If you are not the intended recipient of this email, you must neither
>>>take any action based upon its contents, nor copy or show it to anyone.
>>>Please contact the sender if you believe you have received this email in
>>>error. Shape Blue Ltd is a company incorporated in England & Wales.
>>>ShapeBlue Services India LLP is a company incorporated in India and is
>>>operated under license from Shape Blue Ltd. Shape Blue Brasil
>>>Consultoria Ltda is a company incorporated in Brasil and is operated
>>>under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>

Re: Metadata URL location

[Darren Shepherd]

My guess, I don't really know, would be because its hard.  The VR uses
link local for the control network so 169.254/16 is bound to the wrong
nic.  To fix this you just need some ip routing magic in linux (credit
goes to Kris Lindgren who showed me how to do this).  Add the below to
a file, substitute eth0 for the guest network nic, run "ip -b "
The below effectively creates a routing table dedicated to the IP
169.254.169.254 that sets it default route to go out the guest network
nic.

rule add from 169.254.169.254 table 70
rule add to 169.254.169.254 dev eth0 table 70
route flush table 70
route add default dev eth0 src 169.254.169.254 table 70
route flush cache

Darren

On Thu, Oct 24, 2013 at 6:10 AM, Shanker Balan
 wrote:
> Hi Guys,
>
> CloudStack metadata services are on the default gateway while on EC2,
> its at 169.254.169.254. Am curious to know why CloudStack does not
> use a link local address for meta data services.
>
> A search of the Wiki 
> (https://cwiki.apache.org/confluence/dosearchsite.action?where=CLOUDSTACK&tooltip=Type+ALL%3A+in+your+query+to+search+all+of+Confluence&spaceSearch=true&queryString=metadata)
>  didn’t seem to list any doc related to the design of this service.
>
> TIA.
>
> --
> @shankerbalan
>
> M: +91 98860 60539 | O: +91 (80) 67935867
> shanker.ba...@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, 
> Bangalore - 560 055
>
> CloudStack Bootcamp Training on 27/28 November, Bangalore
> http://www.shapeblue.com/cloudstack-training/
>
>
>
>
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark.

Re: is marvin on master broken?

[Darren Shepherd]

Traceback (most recent call last):
  File "tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,
AttributeError: 'NoneType' object has no attribute 'debug'

On Thu, Oct 24, 2013 at 1:21 AM, Santhosh Edukulla
 wrote:
> Hello Darren,
>
> Some trace is still missing i believe. i could not see the last stack frame 
> in the below trace as what lead to this trace?
>
> I just pulled the latest from master branch and used marvin  to deploy few 
> cloudstack entities and it worked. Can you please provide the command you are 
> using to run marvin tests? or what command lead to the below trace?
>
> Regards,
> Santhosh
> 
> From: Darren Shepherd [darren.s.sheph...@gmail.com]
> Sent: Thursday, October 24, 2013 3:48 AM
> To: dev@cloudstack.apache.org
> Subject: is marvin on master broken?
>
> Whenever I use marvin on master I get
>
> Traceback (most recent call last):
>   File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
> deploy.deploy()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
> self.loadCfg()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
> mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
>   File "./tools/marvin/marvin/deployDataCenter.py", line 492, in 
> registerApiKey
> listuserRes = self.testClient.getApiClient().listUsers(listuser)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 505, in listUsers
> response = self.connection.marvinRequest(command,
> response_type=response, method=method)
>   File 
> "/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
> line 269, in marvinRequest
> self.logging.debug("sending %s request: %s %s" % (method, cmdname,
>
> So it looks like cloudConnection needs a logger passed to the
> constructor and its not being passed?  I've just been hacking up
> ./tools/marvin/marvin/cloudstackConnection.py to work around this.
>
> Darren

Re: how to use hashes on c.a.o?

[Darren Shepherd]

But how does one validate it?  I just wrote a dumb script to
concatenation, remove whitespace, lowercase and then pass to
"sha512sum -c."  I've never seen anyone provide SHAs in that format.
I wouldn't expect many people to know how to use them.  Why can't we
use the good old GNU coreutils style?

Darren

On Wed, Oct 23, 2013 at 7:14 PM, John Kinsella  wrote:
> This is the output of gpg -v --print-md SHA512, generated as part of the 
> release procedure [1] by tools/build/build_asf.sh
>
> 1: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Procedure
>
>
> On Oct 17, 2013, at 7:56 PM, Darren Shepherd  
> wrote:
>
>> The hashes that are on c.a.o for the releases have a format like
>>
>> http://www.apache.org/dist/cloudstack/releases/4.2.0/apache-cloudstack-4.2.0-src.tar.bz2.sha
>>
>> apache-cloudstack-4.2.0-src.tar.bz2: CC487DF3 7E7B6800 F9DC05A3 5B72DEFD
>> 684E0094 F1666F57 5D694916 CF74ED98
>> 9D7CDF35 4021D3C5 8BFD4BB9 39AB02CD
>> EA82D42C 78880EDB 04F2532A 61376537
>>
>> I've never seen this.  Is this some hip new format I'm not aware of,
>> and I'm the uncool kid still using GNU coreutils?
>>
>> Darren
>
>
>

is marvin on master broken?

[Darren Shepherd]

Whenever I use marvin on master I get

Traceback (most recent call last):
  File "./tools/marvin/marvin/deployDataCenter.py", line 610, in 
deploy.deploy()
  File "./tools/marvin/marvin/deployDataCenter.py", line 596, in deploy
self.loadCfg()
  File "./tools/marvin/marvin/deployDataCenter.py", line 557, in loadCfg
mgtDetails.apiKey, mgtDetails.securityKey = self.registerApiKey()
  File "./tools/marvin/marvin/deployDataCenter.py", line 492, in registerApiKey
listuserRes = self.testClient.getApiClient().listUsers(listuser)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 505, in listUsers
response = self.connection.marvinRequest(command,
response_type=response, method=method)
  File 
"/home/darren/src/cloudstack/tools/marvin/marvin/cloudstackConnection.py",
line 269, in marvinRequest
self.logging.debug("sending %s request: %s %s" % (method, cmdname,

So it looks like cloudConnection needs a logger passed to the
constructor and its not being passed?  I've just been hacking up
./tools/marvin/marvin/cloudstackConnection.py to work around this.

Darren

Re: [ASF4.2.1] default to 64-bit system VM template

[Darren Shepherd]

Slightly off topic but I've noticed that the scripts used to create the vm 
templates are slightly out of sync between 32bit and 64bit.  I just assumed 
nobody used 64bit and that's why it got out of sync.

Darren

> On Oct 23, 2013, at 7:21 AM, Marty Sweet  wrote:
> 
> What would be the main reasoning behind this change? Surely the
> minimalistic specifications that SystemVMs are assigned suit 32-bit better
> in terms of Memory?
> 
> "The main disadvantage of 64-bit architectures is that, relative to 32-bit
> architectures, the same data occupies more space in memory (due to longer
> pointers and possibly other types, and alignment padding). This increases
> the memory requirements of a given process and can have implications for
> efficient processor cache utilization."
> 
> 
> On Wed, Oct 23, 2013 at 1:48 PM, Abhinandan Prateek <
> abhinandan.prat...@citrix.com> wrote:
> 
>>> On 23/10/13 3:48 pm, "David Nalley"  wrote:
>>> 
>>> On Wed, Oct 23, 2013 at 1:32 AM, Abhinandan Prateek
>>>  wrote:
 
  We are planning to  make 64-bit system vm templates as default
 offering in 4.2.1.
 This is an initial email to have thoughts from the community on this.
 
 -abhi
>>> 
>>> -1
>>> Who is this we?
>>> I agree with Wido We adhere to semver, and accordingly 4.2.1 should be
>>> a bugfix release.
>> 
>> We were some of the fellow committers, even I was of the same thinking but
>> thought it will be good to have a general opinion.
>> 
>> -abhi
>> 
>> 

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

Updated f3e968b9830a667cb3b55477f1fe8259e56ceffb

Darren

On Wed, Oct 23, 2013 at 4:23 PM, Darren Shepherd
 wrote:
> Frankly, I don't really care, I'll just change it to how it was
> before.  I'll remove the get methods that return a collection.
>
> Darren
>
> On Wed, Oct 23, 2013 at 4:12 PM, SuichII, Christopher
>  wrote:
>> If there are arguments against it, then lets keep the discussion going. I’m 
>> fine with sorting as well - it was requested by someone else to optimize 
>> this. However, just to play devil’s advocate: where would you need a sorted 
>> list of strategies rather than just needing the best fit?
>>
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 23, 2013, at 6:39 PM, Darren Shepherd  
>> wrote:
>>
>>> So your asking to not use the sorting logic and instead do the style of
>>>
>>> Priority highestPriority = Priority.CANT_HANDLE;
>>> Priority priority = strategy.canHandle(...);
>>> if (priority.ordinal() > highestPriority.ordinal()) {
>>>highestPriority = priority;
>>>strategyToUse = strategy;
>>> }
>>>
>>> The problem I have with the above style is that its never possible to
>>> get a sorted collection of strategies.  That logic just allow you to
>>> get the best match.  If you look at
>>> org.apache.cloudstack.engine.subsystem.api.storage.StorageStrategyFactory
>>> there are methods to get the first item and a collection.  In terms of
>>> efficiently, it doesn't matter, were really optimizing a couple of CPU
>>> cycles.
>>>
>>> Darren
>>>
>>> On Wed, Oct 23, 2013 at 3:30 PM, SuichII, Christopher
>>>  wrote:
>>>> Take a look at revision 3 of my changes here:
>>>> https://reviews.apache.org/r/14522/diff/3/#index_header
>>>>
>>>> The changes I made were due to discussion in the reviews. It should be
>>>> simpler, cleaner and more efficient logic than using comparators.
>>>>
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 23, 2013, at 6:25 PM, Darren Shepherd 
>>>> wrote:
>>>>
>>>> Sorry, I don't follow your question.  I have time today.  What would
>>>> you like me to do?  As it stands right now, what is on master, I'm not
>>>> aware of any issues.
>>>>
>>>> Darren
>>>>
>>>> On Wed, Oct 23, 2013 at 3:22 PM, SuichII, Christopher
>>>>  wrote:
>>>>
>>>> Darren,
>>>>
>>>> Would you be able to look into copy the logic back into your refactoring
>>>> today or tomorrow? If not, I may be able to in the morning.
>>>>
>>>> -Chris
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 23, 2013, at 5:56 PM, SuichII, Christopher 
>>>> wrote:
>>>>
>>>> My understanding is that it is still a work in progress to get those test
>>>> back running. Is this correct, Edison?
>>>>
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 23, 2013, at 5:48 PM, Darren Shepherd 
>>>> wrote:
>>>>
>>>> I fixed all the compilation errors in engine/storage/integration-test.
>>>> I don't know how to run those test though, so I can't validate the
>>>> changes.
>>>>
>>>> Darren
>>>>
>>>> On Wed, Oct 23, 2013 at 1:53 PM, SuichII, Christopher
>>>>  wrote:
>>>>
>>>> Yep. I’m running on a clean master.
>>>>
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 23, 2013, at 4:30 PM, Darren Shepherd 
>>>>

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

Frankly, I don't really care, I'll just change it to how it was
before.  I'll remove the get methods that return a collection.

Darren

On Wed, Oct 23, 2013 at 4:12 PM, SuichII, Christopher
 wrote:
> If there are arguments against it, then lets keep the discussion going. I’m 
> fine with sorting as well - it was requested by someone else to optimize 
> this. However, just to play devil’s advocate: where would you need a sorted 
> list of strategies rather than just needing the best fit?
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 6:39 PM, Darren Shepherd  
> wrote:
>
>> So your asking to not use the sorting logic and instead do the style of
>>
>> Priority highestPriority = Priority.CANT_HANDLE;
>> Priority priority = strategy.canHandle(...);
>> if (priority.ordinal() > highestPriority.ordinal()) {
>>highestPriority = priority;
>>strategyToUse = strategy;
>> }
>>
>> The problem I have with the above style is that its never possible to
>> get a sorted collection of strategies.  That logic just allow you to
>> get the best match.  If you look at
>> org.apache.cloudstack.engine.subsystem.api.storage.StorageStrategyFactory
>> there are methods to get the first item and a collection.  In terms of
>> efficiently, it doesn't matter, were really optimizing a couple of CPU
>> cycles.
>>
>> Darren
>>
>> On Wed, Oct 23, 2013 at 3:30 PM, SuichII, Christopher
>>  wrote:
>>> Take a look at revision 3 of my changes here:
>>> https://reviews.apache.org/r/14522/diff/3/#index_header
>>>
>>> The changes I made were due to discussion in the reviews. It should be
>>> simpler, cleaner and more efficient logic than using comparators.
>>>
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 23, 2013, at 6:25 PM, Darren Shepherd 
>>> wrote:
>>>
>>> Sorry, I don't follow your question.  I have time today.  What would
>>> you like me to do?  As it stands right now, what is on master, I'm not
>>> aware of any issues.
>>>
>>> Darren
>>>
>>> On Wed, Oct 23, 2013 at 3:22 PM, SuichII, Christopher
>>>  wrote:
>>>
>>> Darren,
>>>
>>> Would you be able to look into copy the logic back into your refactoring
>>> today or tomorrow? If not, I may be able to in the morning.
>>>
>>> -Chris
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 23, 2013, at 5:56 PM, SuichII, Christopher 
>>> wrote:
>>>
>>> My understanding is that it is still a work in progress to get those test
>>> back running. Is this correct, Edison?
>>>
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 23, 2013, at 5:48 PM, Darren Shepherd 
>>> wrote:
>>>
>>> I fixed all the compilation errors in engine/storage/integration-test.
>>> I don't know how to run those test though, so I can't validate the
>>> changes.
>>>
>>> Darren
>>>
>>> On Wed, Oct 23, 2013 at 1:53 PM, SuichII, Christopher
>>>  wrote:
>>>
>>> Yep. I’m running on a clean master.
>>>
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 23, 2013, at 4:30 PM, Darren Shepherd 
>>> wrote:
>>>
>>> Okay let me look at that.  Are you 100% sure your looking at a clean version
>>> of master?
>>>
>>> Darren
>>>
>>> On Oct 23, 2013, at 1:17 PM, "SuichII, Christopher" 
>>> wrote:
>>>
>>> Er, sorry. That was poorly worded on my part. Some classes, like
>>> SnapshotTest.java and all the storage providers, did not get updated
>>> references to your refactoring. They still reference
>>> StrategyPriority.pickStrategy(), etc. Additionally, I chan

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

So your asking to not use the sorting logic and instead do the style of

Priority highestPriority = Priority.CANT_HANDLE;
Priority priority = strategy.canHandle(...);
if (priority.ordinal() > highestPriority.ordinal()) {
highestPriority = priority;
strategyToUse = strategy;
}

The problem I have with the above style is that its never possible to
get a sorted collection of strategies.  That logic just allow you to
get the best match.  If you look at
org.apache.cloudstack.engine.subsystem.api.storage.StorageStrategyFactory
there are methods to get the first item and a collection.  In terms of
efficiently, it doesn't matter, were really optimizing a couple of CPU
cycles.

Darren

On Wed, Oct 23, 2013 at 3:30 PM, SuichII, Christopher
 wrote:
> Take a look at revision 3 of my changes here:
> https://reviews.apache.org/r/14522/diff/3/#index_header
>
> The changes I made were due to discussion in the reviews. It should be
> simpler, cleaner and more efficient logic than using comparators.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 6:25 PM, Darren Shepherd 
> wrote:
>
> Sorry, I don't follow your question.  I have time today.  What would
> you like me to do?  As it stands right now, what is on master, I'm not
> aware of any issues.
>
> Darren
>
> On Wed, Oct 23, 2013 at 3:22 PM, SuichII, Christopher
>  wrote:
>
> Darren,
>
> Would you be able to look into copy the logic back into your refactoring
> today or tomorrow? If not, I may be able to in the morning.
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 5:56 PM, SuichII, Christopher 
> wrote:
>
> My understanding is that it is still a work in progress to get those test
> back running. Is this correct, Edison?
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 5:48 PM, Darren Shepherd 
> wrote:
>
> I fixed all the compilation errors in engine/storage/integration-test.
> I don't know how to run those test though, so I can't validate the
> changes.
>
> Darren
>
> On Wed, Oct 23, 2013 at 1:53 PM, SuichII, Christopher
>  wrote:
>
> Yep. I’m running on a clean master.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 4:30 PM, Darren Shepherd 
> wrote:
>
> Okay let me look at that.  Are you 100% sure your looking at a clean version
> of master?
>
> Darren
>
> On Oct 23, 2013, at 1:17 PM, "SuichII, Christopher" 
> wrote:
>
> Er, sorry. That was poorly worded on my part. Some classes, like
> SnapshotTest.java and all the storage providers, did not get updated
> references to your refactoring. They still reference
> StrategyPriority.pickStrategy(), etc. Additionally, I changed the
> pickStrategy() logic from using a comparator to looping over the list once
> keeping a reference to the best result. This logic was lost in the merge.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 4:13 PM, Darren Shepherd 
> wrote:
>
> The transaction API was changed in the merge.  I could have maybe
> missed updating a class.  Let me check.   When you said "It looks like
> the changes from us didn’t make it through your merge at all," can you
> point to something specific that got lost?
>
> Darren
>
> On Wed, Oct 23, 2013 at 1:05 PM, SuichII, Christopher
>  wrote:
>
> And it looks like some of your changes may have not merged correctly. I’m
> getting compile errors like:
>
> The method close() is undefined for the type Transaction
>
> This shouldn’t have come from our merge.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 3:52 PM, Darren Shepherd 
> wrote:
>
> Chris, Edison,
>
> You guys just committed 'Support Revert VM Disk from Snapshot.'  At
> the same time I was merging both my txn-refactor and
> spring-modularization branches.  They are really tricky merges and
> each time I have to rebase it takes awhile to figure out.  Anyhow,
> your change + my changes breaks master.  So I quickly rebased rb14823
> and committed to master.  rb14823 is the patch that makes the Storage
> Strategies work with my spring work plus clean up some things.
> Additionally I found out you can't inject List to
> the Snapshot object, so we really have to go with my change to
> centralize the ownership of the strategies to a single class.
>
> Can you please pull master and revalidate that I didn't break
> anything, if its not too much of a pain.
>
> Thanks,
> Darren
>
>
>
>
>
>

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

Sorry, I don't follow your question.  I have time today.  What would
you like me to do?  As it stands right now, what is on master, I'm not
aware of any issues.

Darren

On Wed, Oct 23, 2013 at 3:22 PM, SuichII, Christopher
 wrote:
> Darren,
>
> Would you be able to look into copy the logic back into your refactoring 
> today or tomorrow? If not, I may be able to in the morning.
>
> -Chris
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 5:56 PM, SuichII, Christopher  
> wrote:
>
>> My understanding is that it is still a work in progress to get those test 
>> back running. Is this correct, Edison?
>>
>> --
>> Chris Suich
>> chris.su...@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 23, 2013, at 5:48 PM, Darren Shepherd  
>> wrote:
>>
>>> I fixed all the compilation errors in engine/storage/integration-test.
>>> I don't know how to run those test though, so I can't validate the
>>> changes.
>>>
>>> Darren
>>>
>>> On Wed, Oct 23, 2013 at 1:53 PM, SuichII, Christopher
>>>  wrote:
>>>> Yep. I’m running on a clean master.
>>>>
>>>> --
>>>> Chris Suich
>>>> chris.su...@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 23, 2013, at 4:30 PM, Darren Shepherd  
>>>> wrote:
>>>>
>>>>> Okay let me look at that.  Are you 100% sure your looking at a clean 
>>>>> version of master?
>>>>>
>>>>> Darren
>>>>>
>>>>>> On Oct 23, 2013, at 1:17 PM, "SuichII, Christopher" 
>>>>>>  wrote:
>>>>>>
>>>>>> Er, sorry. That was poorly worded on my part. Some classes, like 
>>>>>> SnapshotTest.java and all the storage providers, did not get updated 
>>>>>> references to your refactoring. They still reference 
>>>>>> StrategyPriority.pickStrategy(), etc. Additionally, I changed the 
>>>>>> pickStrategy() logic from using a comparator to looping over the list 
>>>>>> once keeping a reference to the best result. This logic was lost in the 
>>>>>> merge.
>>>>>>
>>>>>> --
>>>>>> Chris Suich
>>>>>> chris.su...@netapp.com
>>>>>> NetApp Software Engineer
>>>>>> Data Center Platforms – Cloud Solutions
>>>>>> Citrix, Cisco & Red Hat
>>>>>>
>>>>>>> On Oct 23, 2013, at 4:13 PM, Darren Shepherd 
>>>>>>>  wrote:
>>>>>>>
>>>>>>> The transaction API was changed in the merge.  I could have maybe
>>>>>>> missed updating a class.  Let me check.   When you said "It looks like
>>>>>>> the changes from us didn’t make it through your merge at all," can you
>>>>>>> point to something specific that got lost?
>>>>>>>
>>>>>>> Darren
>>>>>>>
>>>>>>> On Wed, Oct 23, 2013 at 1:05 PM, SuichII, Christopher
>>>>>>>  wrote:
>>>>>>>> And it looks like some of your changes may have not merged correctly. 
>>>>>>>> I’m getting compile errors like:
>>>>>>>>
>>>>>>>> The method close() is undefined for the type Transaction
>>>>>>>>
>>>>>>>> This shouldn’t have come from our merge.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chris Suich
>>>>>>>> chris.su...@netapp.com
>>>>>>>> NetApp Software Engineer
>>>>>>>> Data Center Platforms – Cloud Solutions
>>>>>>>> Citrix, Cisco & Red Hat
>>>>>>>>
>>>>>>>>> On Oct 23, 2013, at 3:52 PM, Darren Shepherd 
>>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Chris, Edison,
>>>>>>>>>
>>>>>>>>> You guys just committed 'Support Revert VM Disk from Snapshot.'  At
>>>>>>>>> the same time I was merging both my txn-refactor and
>>>>>>>>> spring-modularization branches.  They are really tricky merges and
>>>>>>>>> each time I have to rebase it takes awhile to figure out.  Anyhow,
>>>>>>>>> your change + my changes breaks master.  So I quickly rebased rb14823
>>>>>>>>> and committed to master.  rb14823 is the patch that makes the Storage
>>>>>>>>> Strategies work with my spring work plus clean up some things.
>>>>>>>>> Additionally I found out you can't inject List to
>>>>>>>>> the Snapshot object, so we really have to go with my change to
>>>>>>>>> centralize the ownership of the strategies to a single class.
>>>>>>>>>
>>>>>>>>> Can you please pull master and revalidate that I didn't break
>>>>>>>>> anything, if its not too much of a pain.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Darren
>>>>>>
>>>>
>>
>

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

I fixed all the compilation errors in engine/storage/integration-test.
 I don't know how to run those test though, so I can't validate the
changes.

Darren

On Wed, Oct 23, 2013 at 1:53 PM, SuichII, Christopher
 wrote:
> Yep. I’m running on a clean master.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 4:30 PM, Darren Shepherd  
> wrote:
>
>> Okay let me look at that.  Are you 100% sure your looking at a clean version 
>> of master?
>>
>> Darren
>>
>>> On Oct 23, 2013, at 1:17 PM, "SuichII, Christopher" 
>>>  wrote:
>>>
>>> Er, sorry. That was poorly worded on my part. Some classes, like 
>>> SnapshotTest.java and all the storage providers, did not get updated 
>>> references to your refactoring. They still reference 
>>> StrategyPriority.pickStrategy(), etc. Additionally, I changed the 
>>> pickStrategy() logic from using a comparator to looping over the list once 
>>> keeping a reference to the best result. This logic was lost in the merge.
>>>
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>>> On Oct 23, 2013, at 4:13 PM, Darren Shepherd  
>>>> wrote:
>>>>
>>>> The transaction API was changed in the merge.  I could have maybe
>>>> missed updating a class.  Let me check.   When you said "It looks like
>>>> the changes from us didn’t make it through your merge at all," can you
>>>> point to something specific that got lost?
>>>>
>>>> Darren
>>>>
>>>> On Wed, Oct 23, 2013 at 1:05 PM, SuichII, Christopher
>>>>  wrote:
>>>>> And it looks like some of your changes may have not merged correctly. I’m 
>>>>> getting compile errors like:
>>>>>
>>>>> The method close() is undefined for the type Transaction
>>>>>
>>>>> This shouldn’t have come from our merge.
>>>>>
>>>>> --
>>>>> Chris Suich
>>>>> chris.su...@netapp.com
>>>>> NetApp Software Engineer
>>>>> Data Center Platforms – Cloud Solutions
>>>>> Citrix, Cisco & Red Hat
>>>>>
>>>>>> On Oct 23, 2013, at 3:52 PM, Darren Shepherd 
>>>>>>  wrote:
>>>>>>
>>>>>> Chris, Edison,
>>>>>>
>>>>>> You guys just committed 'Support Revert VM Disk from Snapshot.'  At
>>>>>> the same time I was merging both my txn-refactor and
>>>>>> spring-modularization branches.  They are really tricky merges and
>>>>>> each time I have to rebase it takes awhile to figure out.  Anyhow,
>>>>>> your change + my changes breaks master.  So I quickly rebased rb14823
>>>>>> and committed to master.  rb14823 is the patch that makes the Storage
>>>>>> Strategies work with my spring work plus clean up some things.
>>>>>> Additionally I found out you can't inject List to
>>>>>> the Snapshot object, so we really have to go with my change to
>>>>>> centralize the ownership of the strategies to a single class.
>>>>>>
>>>>>> Can you please pull master and revalidate that I didn't break
>>>>>> anything, if its not too much of a pain.
>>>>>>
>>>>>> Thanks,
>>>>>> Darren
>>>
>

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

Okay let me look at that.  Are you 100% sure your looking at a clean version of 
master?

Darren

> On Oct 23, 2013, at 1:17 PM, "SuichII, Christopher"  
> wrote:
> 
> Er, sorry. That was poorly worded on my part. Some classes, like 
> SnapshotTest.java and all the storage providers, did not get updated 
> references to your refactoring. They still reference 
> StrategyPriority.pickStrategy(), etc. Additionally, I changed the 
> pickStrategy() logic from using a comparator to looping over the list once 
> keeping a reference to the best result. This logic was lost in the merge.
> 
> -- 
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
> 
>> On Oct 23, 2013, at 4:13 PM, Darren Shepherd  
>> wrote:
>> 
>> The transaction API was changed in the merge.  I could have maybe
>> missed updating a class.  Let me check.   When you said "It looks like
>> the changes from us didn’t make it through your merge at all," can you
>> point to something specific that got lost?
>> 
>> Darren
>> 
>> On Wed, Oct 23, 2013 at 1:05 PM, SuichII, Christopher
>>  wrote:
>>> And it looks like some of your changes may have not merged correctly. I’m 
>>> getting compile errors like:
>>> 
>>> The method close() is undefined for the type Transaction
>>> 
>>> This shouldn’t have come from our merge.
>>> 
>>> --
>>> Chris Suich
>>> chris.su...@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>> 
>>>> On Oct 23, 2013, at 3:52 PM, Darren Shepherd  
>>>> wrote:
>>>> 
>>>> Chris, Edison,
>>>> 
>>>> You guys just committed 'Support Revert VM Disk from Snapshot.'  At
>>>> the same time I was merging both my txn-refactor and
>>>> spring-modularization branches.  They are really tricky merges and
>>>> each time I have to rebase it takes awhile to figure out.  Anyhow,
>>>> your change + my changes breaks master.  So I quickly rebased rb14823
>>>> and committed to master.  rb14823 is the patch that makes the Storage
>>>> Strategies work with my spring work plus clean up some things.
>>>> Additionally I found out you can't inject List to
>>>> the Snapshot object, so we really have to go with my change to
>>>> centralize the ownership of the strategies to a single class.
>>>> 
>>>> Can you please pull master and revalidate that I didn't break
>>>> anything, if its not too much of a pain.
>>>> 
>>>> Thanks,
>>>> Darren
> 

Re: StrategyPriority changes w/ Spring Changes

[Darren Shepherd]

The transaction API was changed in the merge.  I could have maybe
missed updating a class.  Let me check.   When you said "It looks like
the changes from us didn’t make it through your merge at all," can you
point to something specific that got lost?

Darren

On Wed, Oct 23, 2013 at 1:05 PM, SuichII, Christopher
 wrote:
> And it looks like some of your changes may have not merged correctly. I’m 
> getting compile errors like:
>
> The method close() is undefined for the type Transaction
>
> This shouldn’t have come from our merge.
>
> --
> Chris Suich
> chris.su...@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 23, 2013, at 3:52 PM, Darren Shepherd  
> wrote:
>
>> Chris, Edison,
>>
>> You guys just committed 'Support Revert VM Disk from Snapshot.'  At
>> the same time I was merging both my txn-refactor and
>> spring-modularization branches.  They are really tricky merges and
>> each time I have to rebase it takes awhile to figure out.  Anyhow,
>> your change + my changes breaks master.  So I quickly rebased rb14823
>> and committed to master.  rb14823 is the patch that makes the Storage
>> Strategies work with my spring work plus clean up some things.
>> Additionally I found out you can't inject List to
>> the Snapshot object, so we really have to go with my change to
>> centralize the ownership of the strategies to a single class.
>>
>> Can you please pull master and revalidate that I didn't break
>> anything, if its not too much of a pain.
>>
>> Thanks,
>> Darren
>