Right, I should have mentioned that I thought swift already had these
capabilities. Sorry! :-)
So to me it seems like server-side processing of artifacts can wait. I
don't see any reason the workers can't decide at their level how to
format their logs and therefore who to process them through. Maybe down
the road we do end up with some middle-man type processor who just
pushes to swift itself.
Cheers,
Josh
--
Rackspace Australia
On 9/13/13 2:26 AM, James E. Blair wrote:
Thierry Carrez <thie...@openstack.org> writes:
Joshua Hesketh wrote:
Great overview and plan James, thanks for that :-).
So it seems to me that we're duplicating the job of swift a little bit
by writing a program to accept an object over http and store it on disk.
If our end-game is logs stored in swift then why not make jenkins (and
other workers) push the logs straight to swift?
I suspect the additional benefit is the token-based security which means
the slaves can have limited access to storage (they basically get a
token that can only be used for one very limited - and one-time - task).
Indeed -- we can't trust the systems producing artifacts (logs,
tarballs, docs, etc) at all, so we want to tightly scope what they are
able to publish/store. For example, a job should only be able to write
logs to a very specific path determined ahead of time by Zuul or a
tarball job should be able to publish a tarball only for the project in
question.
That said I agree that reinventing HTTP POST storage sounds overkill and
storing directly to swift would be a lot simpler. Does Swift include
complex access rules that we could leverage to implement the same kind
of security that James described in his original post ? If not, would it
make sense to actually add what we need to Swift itself ? (after all,
it's, you know, an OpenStack project)
It turns out that there may very well be a mechanism in swift to do what
we want -- but it's not documented in the swift API documentation!
Compare:
http://docs.openstack.org/api/openstack-object-storage/1.0/content/
To:
http://docs.rackspace.com/files/api/v1/cf-devguide/content/
I filed https://bugs.launchpad.net/openstack-manuals/+bug/1224562 for this.
At any rate, the feature that I apparently re-invented is called
"FormPost", and it's described here:
http://docs.rackspace.com/files/api/v1/cf-devguide/content/FormPost-d1a555.html
And here (see, it really is part of swift):
https://git.openstack.org/cgit/openstack/swift/tree/swift/common/middleware/formpost.py
It appears to support everything I specified in my token idea: HMAC
validation, expiration time, container and sub-path restrictions. As
well as max file size and max number of files.
We would lose the ability to do server-side processing of artifacts
before storage, though of course we still have the opportunity to
process them on the workers before sending them to swift.
We also need to deal with how to generate indexes -- without the
artifact-receiving component to add entries to a database, we may either
need something else to do that, or have something else create directory
markers in swift so that the artifact-server can create directory
indexes on the fly.
-Jim
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
