Hi Marcus, > For mt own clarification, the storage refactoring code is in javelin, but > cloudstack is currently not using it, correct? At this point I was thinking > to myself that the storage refactor was not going to make 4.1, due to some > of the discussions and updates. Or if it did that we wouldn't have time to > work on making anything utilize it.
Yes. The storage refactoring code was written as a completely new project. It didn't touch the existing code. The reason is because it is very different than the current code set. The current code set mixes in orchestration and provisioning and forces provisioning to happen on the hypervisor and, as John pointed out in other posts, forces file system concepts. The new storage refactoring code does not do that. Therefore, we decided to write a completely new storage orchestration engine. In some sense, refactoring is really an incorrect term for it. The new storage engine pulled in the usage of Spring. That's why we decided to convert all of CloudStack to use Spring for di and aop in javelin. In terms of side effect, we can consider the current javelin branch as a Spring conversion branch because the new storage engine doesn't get used until the storage hook-up branch is merged in. > > So at least for the storage part, it sounds like the storage refactor code > will be there, but it's still a question on whether it will be hooked in > for 4.1 (I assume). By voting to merge this are we committing to having to > potentially push back code freeze if the things you mention don't get fixed > in time, or if this means that we have to wait for the storage hook-up > branch? We don't have to wait for the storage hook-up branch. I believe it will come in soon but that's dependent on javelin instead of javelin dependent on it. A little bit about the storage hook-up branch as Edison is probably too busy testing it than to talk about it on the list. The new storage engine forces provisioning into plugins called DataStoreProvider. This allows developers to write plugins to provision volumes, snapshots, and templates. The plugins can determine where the physical provisioning can take place, be it on the hypervisor or on a vm or somewhere else. What the storage hook-up branch does is wrap the existing CloudStack storage code in one big messy DataStoreProvider so that none of the current functionality, such as Ceph, S3, Swift, etc are lost. Then we can refactor that big DataStoreprovider in 4.2. > > Are you saying there is stuff already in master that will be broken without > Spring, or that other feature branches (besides the storage refactor) rely > on it? Nothing in master is broken without Spring. Several other feature branches have been branched off of javelin because they rather write their code/unit tests with Spring. > > > > > > Javelin has been checked with marvin integration testing to make sure > the > > > server behaves the same. skipTests has been turned off and the unit > > tests > > > are running clean. Non-oss is also converted over but there might be a > > small > > > issue with VmWare. Usage and awsapi also have small issues but we > > believe > > > we can fix those within the 72 hour waiting period. > > > Is the integration coverage good enough now that we trust it for a change > this large? Not to be pessimistic, and perhaps this has nothing to do with > the merge request itself, but two weeks ago when we were trying to add a > test to the test_volumes.py we found that it was only partially working to > begin with. Ryan fixed that one up. So I'm wondering if the integration > testing worked because the tests have been fixed up, or if you just mean > that the results are the same (tests that failed before still fail, > successes still succeed). I'm assuming this is questioning the Spring work given that storage engine is not activated yet. It is true that Spring touches almost every single file in CloudStack. In some respects, on principle only, you figure an aop system shouldn't have such deep impacts to a system. The Spring system and the original system are very similar in principle so once things are wired up, there's no change to the original business logic so the effect of the Spring system once the system is up and running is fairly minor. Expect some bugs which I think the two month of testing after code freeze should turn up easily. As for the tests, Rohit sent an email earlier showing why expunge failed. So yes if it failed in master, it still failed in javelin. For the expunge case, we will probably fix the business logic because it didn't make sense in the first place. But, again, no business logic changes due to Spring work. --Alex
