Hi, everyone. I've been preparing a proof of concept for the reorg, and
it's reached a stage where it's ready for comment.
https://github.com/ssorj/qpid-svn-reorg
Look here for an idea of how the Qpid subversion top level would appear
after these changes:
https://github.com/ssorj/qpid-svn-reorg/tree/trunk/qpid
I have remaining work to get the C++ tests in shape, but the overall form
of things is what I wish to propose. I would appreciate your comments.
This is a functioning alternate source organization, so you can download it
and test it directly.
So you can make comments inline, I've appended the markdown README below.
I will produce revised proposals after incorporating your input.
Thanks,
Justin
---
# Qpid Subversion reorganization
## Overview
This proof of concept represents an effort to achieve the source tree
layout proposed [here][1]. It allows the Qpid project to produce
independent releases of Qpid C++ and Python as well as other modules
that have heretofore been bundled into one large Qpid release.
[1]:
https://cwiki.apache.org/confluence/display/qpid/Source+tree+layout+proposal
### End-goal top-level codebases and source artifacts
New at the top level
qpid-cpp qpid/cpp/trunk qpid-cpp-VERSION.tar.gz
qpid-java-qmf qpid/java-qmf/trunk qpid-java-qmf-VERSION.tar.gz
qpid-python qpid/python/trunk qpid-python-VERSION.tar.gz
qpid-wcf qpid/wcf/trunk qpid-wcf-VERSION.tar.gz
saslwrapper qpid/saslwrapper/trunk saslwrapper-VERSION.tar.gz
Existing
qpid-java qpid/java/trunk qpid-java-VERSION.tar.gz
Existing and migrated to Git
qpid-dispatch qpid-dispatch.git qpid-dispatch-VERSION.tar.gz
qpid-jms qpid-jms.git apache-qpid-jms-VERSION.tar.gz
qpid-proton qpid-proton.git qpid-proton-VERSION.tar.gz
No source releases
qpid-maven qpid/maven/trunk [NA]
qpid-site qpid/site [NA]
qpid-specs qpid/specs [NA]
In the current proposal, qpid-java-qmf stands apart from qpid-java,
but it may make more sense for java-qmf to take a place inside of
qpid-java.
## Dependencies
qpid-cpp depends on qpid-proton for amqp 1.0 support
qpid-cpp depends on qpid-python for its tests
qpid-dispatch depends on qpid-proton
qpid-java depends on qpid-python for its tests
qpid-java-qmf depends on qpid-java
qpid-jms depends on qpid-proton
qpid-python optionally depends on saslwrapper
qpid-wcf depends on qpid-cpp
## Changes in the Subversion tree
Note that the proof of concept does not in fact move the new top-level
codebases into the standard Subversion structure with trunk, branches,
and tags. In order to minimize diffs, they instead remain in their
current locations, but updated to remove cross-tree source
dependencies.
### Relocations
Migrate qpid/trunk/qpid/tools python and ruby content to
qpid/cpp/management:
qpid/trunk/tools/setup.py ->
qpid/cpp/trunk/management/setup.py
qpid/trunk/tools/MANIFEST.in ->
qpid/cpp/trunk/management/MANIFEST.in
qpid/trunk/tools/*.txt ->
qpid/cpp/trunk/management/python/*.txt
qpid/trunk/tools/src/py/$libs ->
qpid/cpp/trunk/management/python/lib/$libs
qpid/trunk/extras/qmf/src/py/qmf ->
qpid/cpp/trunk/management/python/lib/qmf
qpid/trunk/tools/src/py/$tools ->
qpid/cpp/trunk/management/python/bin/$tools
qpid/trunk/tools/src/ruby/qpid_management ->
qpid/cpp/trunk/management/ruby/qpid_management
Migrate qpid/trunk/qpid/tools java content to qpid/java-qmf:
qpid/trunk/tools/src/java -> qpid/java-qmf/trunk
Migrate qpid/trunk/qpid/tests to qpid/python:
qpid/trunk/tests/src/py/qpid_tests ->
qpid/python/trunk/qpid_tests
Migrate Windows packaging scripts to qpid/cpp:
qpid/trunk/qpid/packaging/windows ->
qpid/cpp/trunk/packaging/windows
Migrate other self-contained components to the top level:
qpid/trunk/qpid/extras/sasl -> qpid/saslwrapper/trunk
qpid/trunk/qpid/wcf -> qpid/wcf/trunk
### Deletions
Obsolete:
qpid/trunk/qpid/QPID_VERSION.txt # No longer makes sense
qpid/trunk/qpid/LICENSE # No longer makes sense
qpid/trunk/qpid/NOTICE # No longer makes sense
qpid/trunk/qpid/extras/dispatch # Obsolete stub
qpid/trunk/qpid/doc/website # Obsolete stub
After migration:
qpid/trunk/qpid/doc/book # Moved under
qpid/cpp/trunk/docs
qpid/trunk/qpid/extras/qmf # Moved under
qpid/cpp/trunk/management/python
Apparently no longer in use. I'm judging "in use" by whether there
have been any changes--not an excellent metric. If you are using any
of these, please inform me so that I preserve them.
qpid/trunk/qpid/review # Not touched since 2009
qpid/trunk/qpid/sandbox # Not touched since 2010
qpid/trunk/qpid/bin # Not touched since 2012
qpid/trunk/qpid/buildtools # Not touched since 2010
qpid/trunk/qpid/doc/dev-readme # Not touched since 2010
qpid/trunk/qpid/etc # Not touched since 2008
## C++ subtree changes
Relocate some docs:
cpp/design_docs -> cpp/docs/design
cpp/docs/src/* -> cpp/docs/design
cpp/DESIGN -> cpp/docs/design/overview.txt
cpp/AMQP_1.0 -> cpp/docs/amqp-1.0.txt
cpp/SSL -> cpp/docs/ssl.txt
cpp/README-HA.txt -> cpp/docs/ha.txt
Collect winsdk stuff:
cpp/bld-winsdk.ps1 -> cpp/packaging/winsdk/bld-winsdk.ps1
cpp/README-winsdk.ps1 -> cpp/packaging/winsdk/README.txt
Clean up names and extensions:
cpp/QPID_VERSION.txt -> cpp/VERSION.txt # Matching what Dispatch
has
cpp/NOTICE -> cpp/NOTICE.txt
cpp/LICENSE -> cpp/LICENSE.txt # Using .txt extension
for these
cpp/INSTALL -> cpp/INSTALL.txt # is widespread practice
now
cpp/INSTALL-WINDOWS -> cpp/INSTALL-WINDOWS.txt
## Possibly impacted parties
- Steve Huston - Windows packaging code (qpid/packaging/windows)
- Java broker folks - Relocation of Python qpid_tests may affect Java
broker testing
## How to setup the Qpid Python dependency
cd qpid/python/trunk
./setup.py install --user
export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages
export PATH=$HOME/.local/bin:$PATH
## C++ tests
Changes:
- Run scripts ("run_" scripts) organized by feature area
- Run scripts are uniformly named
- Run scripts produce temporary work dirs of the form run_script\_name_XXXX
- They are removed on test success
- All run scripts are runnable directly from src/tests build dir
- Python test env and test broker tools - should be cross platform
- Permits removal of many scripts that have both bash and
powershell variants
- Broker creation is more consistent
- Broker termination and checking too
- Modernized bash scripts
Test results after initial relocation:
The following tests FAILED:
1 - api_check_qpidtypes (Failed)
2 - api_check_qpidmessaging (Failed)
20 - interop_tests (Failed)
21 - ha_tests (Failed)
22 - qpidd_qmfv2_tests (Failed)
23 - interlink_tests (Failed)
24 - idle_timeout_tests (Failed)
Errors while running CTest
Makefile:149: recipe for target 'test' failed
make: *** [test] Error 8
It was actually worse than this suggests. Lots of the tests silently
pass (!) if the python testing tools are not available.
Current test results on Fedora 23:
Total Test time (real) = 2053.14 sec
The following tests FAILED:
1 - api_check_qpidtypes (Failed) [C++11 abi changes]
2 - api_check_qpidmessaging (Failed) [C++11 abi changes]
4 - acl_tests (Failed) [valgrind error]
12 - python_tests (Failed) ["amqp:precondition-failed: No
target specified!"]
13 - queue_redirect_tests (Failed) [valgrind error]
26 - linearstore_python_tests (Failed) [valgrind error]
Errors while running CTest
## Notes
- Gah! Circular dependency. Qpid Python (via qpid.testlib) depends
on qpidtoollibs. This should be removed: Qpid C++ depends on Qpid
Python and not the reverse in any way.
- The new test regime more uniformly runs brokers with valgrind, and
this exposes new valgrind errors.
- I removed the following test install utils from src/tests:
qpid-build-rinstall, install_env.sh, etc. If these are still
important, I'd like to bring them back in a subdirectory or a
different project altogether. They are not central to the C++
tests.
## Todo
- Rewrite the index in qpid/README.txt - Update all the top-level
info
- Lots of tedious, infuriating Windows testing
- Current results at
https://ci.appveyor.com/project/ssorj/qpid-svn-reorg/
