Review Request 20557: CLOUDSTACK-6472 GenerateUsageRecords generates NPEs for expunging instances
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20557/ --- Review request for cloudstack. Bugs: CLOUDSTACK-6472 https://issues.apache.org/jira/browse/CLOUDSTACK-6472 Repository: cloudstack-git Description --- This is a review request for CLOUDSTACK-6472 "GenerateUsageRecords generates NPEs for expunging instances" The patch is against the 4.3 branch Diffs - server/src/com/cloud/api/ApiResponseHelper.java e543d1c Diff: https://reviews.apache.org/r/20557/diff/ Testing --- Thanks, Pierre-Yves Ritschard
Re: Review Request 20557: CLOUDSTACK-6472 listUsageRecords generates NPEs for expunging instances
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20557/ --- (Updated April 23, 2014, 7:46 a.m.) Review request for cloudstack. Changes --- correct api command Summary (updated) - CLOUDSTACK-6472 listUsageRecords generates NPEs for expunging instances Bugs: CLOUDSTACK-6472 https://issues.apache.org/jira/browse/CLOUDSTACK-6472 Repository: cloudstack-git Description (updated) --- This is a review request for CLOUDSTACK-6472 "listUsageRecords generates NPEs for expunging instances" The patch is against the 4.3 branch Diffs - server/src/com/cloud/api/ApiResponseHelper.java e543d1c Diff: https://reviews.apache.org/r/20557/diff/ Testing --- Thanks, Pierre-Yves Ritschard
Re: Review Request 20557: CLOUDSTACK-6472 listUsageRecords generates NPEs for expunging instances
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20557/ --- (Updated April 23, 2014, 12:47 p.m.) Review request for cloudstack. Changes --- Updated with no whitespace mixups Bugs: CLOUDSTACK-6472 https://issues.apache.org/jira/browse/CLOUDSTACK-6472 Repository: cloudstack-git Description --- This is a review request for CLOUDSTACK-6472 "listUsageRecords generates NPEs for expunging instances" The patch is against the 4.3 branch Diffs (updated) - server/src/com/cloud/api/ApiResponseHelper.java e543d1c Diff: https://reviews.apache.org/r/20557/diff/ Testing --- Thanks, Pierre-Yves Ritschard
Re: Review Request 20557: CLOUDSTACK-6472 listUsageRecords generates NPEs for expunging instances
> On April 23, 2014, 8:10 a.m., Rajesh Battala wrote: > > server/src/com/cloud/api/ApiResponseHelper.java, line 3350 > > <https://reviews.apache.org/r/20557/diff/1/?file=564547#file564547line3350> > > > > There are tabs/whitespace characters introduced > > Please add what are the tests you have executed/verified. Hi Rajesh, Sorry about the whitespace. As for the reproduction steps, they are documented in https://issues.apache.org/jira/browse/CLOUDSTACK-6472, I'll reproduce here: 1. Wait for cloud-usage to generate records for a running vm. 2. Delete the VM and wait for it to be in 'Expunged' state, it will have a non-null 'removed' field in the vm_instance table 3. Issue the listUsageRecords API command, it will fail and log a stack trace to the management-server log The stack-trace that appears is as follows: 2014-04-22 00:18:07,417 ERROR [cloud.api.ApiServer] (catalina-exec-2:ctx-60c87510 ctx-22de212f ctx-a88a3dd7) unhandled exception executing api command: listUsageRecords java.lang.NullPointerException at com.cloud.api.ApiResponseHelper.createUsageResponse(ApiResponseHelper.java:3308) at org.apache.cloudstack.api.command.admin.usage.GetUsageRecordsCmd.execute(GetUsageRecordsCmd.java:119) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) To fix, apply the above patch, listUsageRecords will now correctly honor the request - Pierre-Yves --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20557/#review41128 ------- On April 23, 2014, 12:47 p.m., Pierre-Yves Ritschard wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20557/ > --- > > (Updated April 23, 2014, 12:47 p.m.) > > > Review request for cloudstack. > > > Bugs: CLOUDSTACK-6472 > https://issues.apache.org/jira/browse/CLOUDSTACK-6472 > > > Repository: cloudstack-git > > > Description > --- > > This is a review request for CLOUDSTACK-6472 "listUsageRecords generates NPEs > for expunging instances" > > The patch is against the 4.3 branch > > > Diffs > - > > server/src/com/cloud/api/ApiResponseHelper.java e543d1c > > Diff: https://reviews.apache.org/r/20557/diff/ > > > Testing > --- > > > Thanks, > > Pierre-Yves Ritschard > >
Re: Review Request 20557: CLOUDSTACK-6472 listUsageRecords generates NPEs for expunging instances
> On April 24, 2014, 12:08 p.m., Sebastien Goasguen wrote: > > Rajesh, can you make sure to apply this to master and 4.4-forward as well > > as 4.3 ? > > thx There is a small 2-line conflict when cherry-picking the commit on master but it is easily solved. - Pierre-Yves --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20557/#review41285 --- On April 23, 2014, 12:47 p.m., Pierre-Yves Ritschard wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20557/ > --- > > (Updated April 23, 2014, 12:47 p.m.) > > > Review request for cloudstack. > > > Bugs: CLOUDSTACK-6472 > https://issues.apache.org/jira/browse/CLOUDSTACK-6472 > > > Repository: cloudstack-git > > > Description > --- > > This is a review request for CLOUDSTACK-6472 "listUsageRecords generates NPEs > for expunging instances" > > The patch is against the 4.3 branch > > > Diffs > - > > server/src/com/cloud/api/ApiResponseHelper.java e543d1c > > Diff: https://reviews.apache.org/r/20557/diff/ > > > Testing > --- > > > Thanks, > > Pierre-Yves Ritschard > >
A plea for branching awsapi out of the main repo
Hi dev@, The following is an overview in kilobytes of each of component in the cloudstack code-base (as of master): 4 configure-info.in 4 LICENSE.header 4 NOTICE 4 README.md 4 version-info.in 8 build 8 INSTALL.md 8 maven-standard 12 agent-simulator 12 README.tools.md 20 developer 32 CHANGES 36 quickcloud 40 LICENSE 40 pom.xml 72 cloud-cli 76 debian 80 docs 104 awsapi-setup 108 packaging 288 python 400 agent 484 usage 660 vmware-base 1272 utils 1308 setup 1372 systemvm 1568 scripts 1864 client 1948 deps 2096 core 2700 framework 2968 services 4456 tools 6404 api 7360 test 7900 ui 8060 server 9036 engine 15976 plugins 45052 awsapi As you can see awsapi accounts for more than a third of the total code base of CS, mostly generated by the WSDL definition. Since there are now (lighter) alternative implementations of the EC2 to CS API bits, it might make sense to have a separate repo for this. This would make the repo a less unwieldy than it is currently. Thoughts on this ? Are there examples of other ASF projects where code is split across several repos ? - pyr
Re: [GitHub] cloudstack pull request: Remove AWS api bridge
It would be interesting to have usage stats on the AWS cloud bridge. It is very hard to get working right correctly (especially when compared to something like ec2stack) so I'd be very surprised if: - There were a large number of users - They had upgrade path issues I think the idea of chunking out awsapi in its own repo has some merit, even though it will most likely ending up being bitrot. As far as packaging is concerned, this PR still builds packages correctly for debian, someone should do a test build for RPM packages (i did the spec changes but didn't build). On Fri, Nov 21, 2014 at 3:50 PM, Ian Duffy wrote: > +1 on ec2stack working well (bias view). > > I've used it via vagrant-aws, boto and eucalyptus eutester without issue. > > It could use some documentation on deployment for production purposes, the > embedded webserver it exposes is OK but I'd feel safer with it bring behind > nginx/Apache. > On 21 Nov 2014 14:31, "Hugo Trippaers" wrote: > > > Let’s start by getting this on a feature branch. > > > > I would like to make sure that everything works before we remove the code > > and that includes deb and rpm packaging. We also need to think about the > > upgrade path. If a user is currently using awsapi, he needs an upgrade > path > > the start using the replacement. > > > > Cheers, > > > > Hugo > > > > > On 21 nov. 2014, at 14:39, Sebastien Goasguen > wrote: > > > > > > > > > On Nov 21, 2014, at 8:33 AM, Nux! wrote: > > > > > >> +1 what Daan said. > > >> > > >> Once ec2stack works well, then nuke awsapi. > > >> > > > > > > it works well. > > > > > > where can we see test about awsapi ? > > > > > >> my 2 pence > > >> > > >> -- > > >> Sent from the Delta quadrant using Borg technology! > > >> > > >> Nux! > > >> www.nux.ro > > >> > > >> - Original Message - > > >>> From: "Daan Hoogland" > > >>> To: "dev" > > >>> Sent: Friday, 21 November, 2014 13:16:25 > > >>> Subject: Re: [GitHub] cloudstack pull request: Remove AWS api bridge > > >> > > >>> As Seb mentioned on list there is an alternative. I don't think we > > >>> should remove this before the factored out version is working as well > > >>> (or the alternative he mentions is at least as complete) The idea of > > >>> isolating this bit is appealing though. > > >>> > > >>> Daan > > >>> > > >>> On Fri, Nov 21, 2014 at 12:23 PM, Nux! wrote: > > >>>> Hello, > > >>>> > > >>>> EC2 compatibility is an essential feature for potential ACS > adopters. > > >>>> What alternatives are there for the AWSAPI component? > > >>>> > > >>>> Lucian > > >>>> > > >>>> -- > > >>>> Sent from the Delta quadrant using Borg technology! > > >>>> > > >>>> Nux! > > >>>> www.nux.ro > > >>>> > > >>>> - Original Message - > > >>>>> From: "pyr" > > >>>>> To: dev@cloudstack.apache.org > > >>>>> Sent: Friday, 21 November, 2014 10:18:58 > > >>>>> Subject: [GitHub] cloudstack pull request: Remove AWS api bridge > > >>>> > > >>>>> GitHub user pyr opened a pull request: > > >>>>> > > >>>>> https://github.com/apache/cloudstack/pull/44 > > >>>>> > > >>>>> Remove AWS api bridge > > >>>>> > > >>>>> This has been a discussion point for a while. The (mostly > generated) > > >>>>> code for the AWS api bridge is by far the largest source component > > in > > >>>>> Cloudstack, while seldomly used. > > >>>>> > > >>>>> Now that alternate options exist to provide EC2 compatibility, it > > >>>>> makes sense to remove it for the few users who cannot directly > > >>>>> talk to the cloudstack API. > > >>>>> > > >>>>> You can merge this pull request into a Git repository by running: > > >>>>> > > >>>>> $ git pull https://github.com/pyr/cloudstack feature/
Re: [GitHub] cloudstack pull request: Use constant-time comparison functions wh...
I'll note here that this can be applied to 4.4 and 4.3 as well, modulo some simple changes. On Wed, Jan 14, 2015 at 11:32 AM, pyr wrote: > GitHub user pyr opened a pull request: > > https://github.com/apache/cloudstack/pull/65 > > Use constant-time comparison functions when checking signatures > > This limits the likeliness of timing attacks against the API. > See http://codahale.com/a-lesson-in-timing-attacks/ for the > full rationale. > > You can merge this pull request into a Git repository by running: > > $ git pull https://github.com/exoscale/cloudstack > feature/constant-time > > Alternatively you can review and apply these changes as the patch at: > > https://github.com/apache/cloudstack/pull/65.patch > > To close this pull request, make a commit to your master/trunk branch > with (at least) the following in the commit message: > > This closes #65 > > > commit 9b4e39e837af498599859c4a6687eb8bf9f8ad89 > Author: Pierre-Yves Ritschard > Date: 2015-01-14T10:27:35Z > > Use constant-time comparison functions when checking signatures > > This limits the likeliness of timing attacks against the API. > See http://codahale.com/a-lesson-in-timing-attacks/ for the > full rationale. > > Conflicts: > server/src/com/cloud/api/ApiServer.java > server/src/com/cloud/user/AccountManagerImpl.java > > > > > --- > If your project is set up for it, you can reply to this email and have your > reply appear on GitHub as well. If your project does not have this feature > enabled and wishes so, or if the feature is enabled but not working, please > contact infrastructure at infrastruct...@apache.org or file a JIRA ticket > with INFRA. > --- >
Re: Resource Management/Locking [was: Re: What would be your ideal solution?]
Hi Everyone, First off, I'm really excited that there is an undergoing discussion on these issues. I agree with john that CAP provides a good "framework" for looking at the individual properties of the distributed system that cloudstack is, as a whole. The separation between an orchestration layer and automation layer is also a valid abstraction of the main roles of the management server. As far as CAP properties are concerned, I don't think there is much question that the aim is for: * a CP orchestration layer (it will continue to rely on a CP system: an RDBMS) * an AP automation layer (it is tied to an AP system, a cluster of hypervisors) As far as operations are concerned I think the plugin approach in CS is great, it allows to distribute a very simple system to start with, where a single management server will most likely run. In largely distributed systems it is certainly not a crazy requirement to rely on zookeeper, in many shops using CS, ZK is already used anyhow, operation-wise, it is not more complex than, say, maintaining a highly available MySQL cluster. Before I go on, I'll just acknowledge here that I'm not addressing the issue of compatibility, all approaches discussed so far, except Darren's do not concern themselves with compatibility and upgrades which will be a major pain if the persistence layer / data store evolves in any significant way. I know this is a big concern for CS users and citrix, and will need to be taken into account, I don't have a clear picture of how this could be done. As far as persistence is concerned, there are different things that CS stores which have different requirements: * Organizational data needs strong consistency: users, accounts, domains, projects, configuration (for networks, templates, ...) * Transient resource data (vm running status) can only have eventual consistency * Usage data only requires eventual consistency (and does not need to clutter the main data store) I think one of the reasons for the head-scratching around resources right now is that the persistence layer is right now used both for storing the expected state of resources and their actual state, maybe their should be a transient persistence layer used for storing known states. So to sum up, as far as storage is concerned it might be easier to reason about CS in terms of three different persistence layer: * A main layer for organizational data, expected state and last known state * A layer for storing state as reported by resource owners (hypervisors) * A mechanism for distributing usage data With such a system, the mailbox approach is possible. I do think that the amount of work in CS would be huge and that we would risk ending up with a franken-erlang type system which java doesn't lend itself well too (surely scala could but this would imply a total rewrite). An intermediate step could be to look at resources the same way Apache Kafka does (or in a way Apache Cassandra). Managers could be seen as a homogeneous clusters responsible for an nth of the cluster (for a cluster of n managers). A good mechanism is needed for agreeing on cluster membership, but there are several proven and valid approaches for this (and its a problem that lends itself well to the plugin approach in CS). A typical incoming API request would thus hit any management node, which could either issue a redirect to the correct node, proxy it to the correct node or create a jobid and let the client query the jobid for its status. The upside of this approach is that it still makes it possible for CS to become the jenkins of cloud controllers (it would need an HSQLDB option for persistence though !) and rely on proven and well understood projects for larger deployments (like ZK, or when it stabilizes, an implementation of raft). A first step towards this would be to have some sort of agreement on the different layers of persistence needed throughout CS and try to move forward. I can get my hands dirty and try to evolve the Dao stuff that is everywhere in CS, but I'd like to know I'm not going towards a dead-end. On Mon, Nov 25, 2013 at 10:18 PM, Darren Shepherd < darren.s.sheph...@gmail.com> wrote: > Okay, I'll have to stew over this for a bit. My one general comment is > that it seems complicated. Such a system seems like it would take a good > amount of effort to construct properly and as such it's a risky endeavour. > > Darren > > > On Mon, Nov 25, 2013 at 12:10 PM, John Burwell wrote: > > > Darren, > > > > In a peer-to-peer model such as I describe, active-active is and is not a > > concept. The supervision tree is responsible for identifying failure, > and > > initiating process re-allocation for failed resources. For example, if a > > pod’s management process crashed, it would also crash all of the > processes > > managing the hosts in that pod. The zone would then attempt to restart > the > > pod’s management process (either local to the zone supervisor or on a > > remote instance whi
