Re: The PostgreSQL charm, AWS and robustness
I have a new project that I'm wanting to deploy and it is a django project backed with a postgresql database. Pretty simple really. If you plan to use the python-django charm be sure to use this branch: https://code.launchpad.net/~patrick-hetu/charms/precise/python-django/charmhelpers The charm was rewritten to use charmhelpers + Ansible and it have not landed yet. -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
--constraints root-disk=16384M fails in EC2
Trying to deploy a charm with some extra root disk space. When using the root-disk constraint defined above I get the following error: '(error: no instance types in us-east-1 matching constraints cpu-power=100 root-disk=16384M)' I’m deploying a bundle with the following constraints: constraints: mem=4G arch=amd64”, but need more disk-space then the default provided. Any suggestions ? Stein Myrseth Bjørkesvingen 6J 3408 Tranby mob: +47 909 62 763 mailto:stein.myrs...@gmail.com smime.p7s Description: S/MIME cryptographic signature -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: --constraints root-disk=16384M fails in EC2
Hi Stein This does appear to be a bug in Juju's constraints handling for EC2. I'd have to do an experiment to confirm, but certainly reading the code appears to show a problem. Given how EC2 works, in that Juju asks for the specified root disk size when starting an instance, I don't have a workaround that I can think of to share with you. The fix for this would be relatively simple to implement and so can be done in time for the next stable release (1.20) which is due in a few weeks. Alternatively, we hope to have a new development release out next week (1.19.3). I'll try to get any fix done in time for that also. I've raised bug 1324729 for this issue. On Fri 30 May 2014 09:29:15 EST, GMail wrote: Trying to deploy a charm with some extra root disk space. When using the root-disk constraint defined above I get the following error: '(error: no instance types in us-east-1 matching constraints cpu-power=100 root-disk=16384M)' I’m deploying a bundle with the following constraints: constraints: mem=4G arch=amd64”, but need more disk-space then the default provided. Any suggestions ? Stein Myrseth Bjørkesvingen 6J 3408 Tranby mob: +47 909 62 763 mailto:stein.myrs...@gmail.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: --constraints root-disk=16384M fails in EC2
The link to the bug is here: https://bugs.launchpad.net/juju-core/+bug/1324729 On Thu, May 29, 2014 at 7:26 PM, Ian Booth ian.bo...@canonical.com wrote: Hi Stein This does appear to be a bug in Juju's constraints handling for EC2. I'd have to do an experiment to confirm, but certainly reading the code appears to show a problem. Given how EC2 works, in that Juju asks for the specified root disk size when starting an instance, I don't have a workaround that I can think of to share with you. The fix for this would be relatively simple to implement and so can be done in time for the next stable release (1.20) which is due in a few weeks. Alternatively, we hope to have a new development release out next week (1.19.3). I'll try to get any fix done in time for that also. I've raised bug 1324729 for this issue. On Fri 30 May 2014 09:29:15 EST, GMail wrote: Trying to deploy a charm with some extra root disk space. When using the root-disk constraint defined above I get the following error: '(error: no instance types in us-east-1 matching constraints cpu-power=100 root-disk=16384M)' I’m deploying a bundle with the following constraints: constraints: mem=4G arch=amd64”, but need more disk-space then the default provided. Any suggestions ? Stein Myrseth Bjørkesvingen 6J 3408 Tranby mob: +47 909 62 763 mailto:stein.myrs...@gmail.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: --constraints root-disk=16384M fails in EC2
If a root disk constraint is specified, Juju will translate that into a block device mapping request when the instance is started. Hence we do start an instance with the required root disk size but the subsequent constraints matching fails. That's my understanding anyway. On 30/05/14 11:42, Kapil Thangavelu wrote: fwiw. all the ubuntu cloud images root disks in ec2 have 8gb of disk size by default, juju doesn't reallocate the root volume size when creating an instance (if it did cloudinit will auto resize the root fs if its created with a larger root vol). On Thu, May 29, 2014 at 8:26 PM, Ian Booth ian.bo...@canonical.com wrote: Hi Stein This does appear to be a bug in Juju's constraints handling for EC2. I'd have to do an experiment to confirm, but certainly reading the code appears to show a problem. Given how EC2 works, in that Juju asks for the specified root disk size when starting an instance, I don't have a workaround that I can think of to share with you. The fix for this would be relatively simple to implement and so can be done in time for the next stable release (1.20) which is due in a few weeks. Alternatively, we hope to have a new development release out next week (1.19.3). I'll try to get any fix done in time for that also. I've raised bug 1324729 for this issue. On Fri 30 May 2014 09:29:15 EST, GMail wrote: Trying to deploy a charm with some extra root disk space. When using the root-disk constraint defined above I get the following error: '(error: no instance types in us-east-1 matching constraints cpu-power=100 root-disk=16384M)' I’m deploying a bundle with the following constraints: constraints: mem=4G arch=amd64”, but need more disk-space then the default provided. Any suggestions ? Stein Myrseth Bjørkesvingen 6J 3408 Tranby mob: +47 909 62 763 mailto:stein.myrs...@gmail.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: --constraints root-disk=16384M fails in EC2
i hadn't realized it was doing block dev maps, but yeah that's readily apparent in going through the source. cool. On Thu, May 29, 2014 at 9:47 PM, Ian Booth ian.bo...@canonical.com wrote: If a root disk constraint is specified, Juju will translate that into a block device mapping request when the instance is started. Hence we do start an instance with the required root disk size but the subsequent constraints matching fails. That's my understanding anyway. On 30/05/14 11:42, Kapil Thangavelu wrote: fwiw. all the ubuntu cloud images root disks in ec2 have 8gb of disk size by default, juju doesn't reallocate the root volume size when creating an instance (if it did cloudinit will auto resize the root fs if its created with a larger root vol). On Thu, May 29, 2014 at 8:26 PM, Ian Booth ian.bo...@canonical.com wrote: Hi Stein This does appear to be a bug in Juju's constraints handling for EC2. I'd have to do an experiment to confirm, but certainly reading the code appears to show a problem. Given how EC2 works, in that Juju asks for the specified root disk size when starting an instance, I don't have a workaround that I can think of to share with you. The fix for this would be relatively simple to implement and so can be done in time for the next stable release (1.20) which is due in a few weeks. Alternatively, we hope to have a new development release out next week (1.19.3). I'll try to get any fix done in time for that also. I've raised bug 1324729 for this issue. On Fri 30 May 2014 09:29:15 EST, GMail wrote: Trying to deploy a charm with some extra root disk space. When using the root-disk constraint defined above I get the following error: '(error: no instance types in us-east-1 matching constraints cpu-power=100 root-disk=16384M)' I’m deploying a bundle with the following constraints: constraints: mem=4G arch=amd64”, but need more disk-space then the default provided. Any suggestions ? Stein Myrseth Bjørkesvingen 6J 3408 Tranby mob: +47 909 62 763 mailto:stein.myrs...@gmail.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
On 29 May 2014 04:03, Tim Penhey tim.pen...@canonical.com wrote: On 29/05/14 01:11, roger peppe wrote: Writing code to explicitly declare function return types is also a perpetual drag on refactoring, but it's also incredibly useful. The set of possible returned errors *should* be part of the contract of a function, otherwise how could you know that a given error might be returned? This theory has also been applied to both Java and C++ in the past with checked error types, and in both situations it ended up being unworkable and discarded by both communities. There's a huge difference between adding errors formally as part of the type system and checking them as part of the runtime. The latter is indeed unworkable because you end up with a proliferation of method signatures that cannot work together. The former is more like a domain-specific form of Eiffel's postconditions. To put it another way, we don't try to encode the fact that math.Sqrt returns the square root of a number in its type signature. But that fact is definitely part of the contract of the function, and we write tests for it. Errors are worth treating specially here because they're they pass up through multiple layers, so it's very easy to break abstraction boundaries by relying on specific error types. I believe it's very important to *think* about the possible errors that can be returned from a given function, and not just throw up our hands and say you guys 5 layers down, why don't you just communicate *arbitrary stuff* to the guy 3 layers above. That already happens, and it has resulted in real bugs. The errgo stuff is trying to be a way of communicating that thinking in code and it defaults to a mode that closes off those arbitrary communication channels by default, reducing the overall complexity of the system. I am by no means saying that the errgo interface is perfect, or even particularly great - the API design feels cluttered and unnecessarily complex even though the code is quite simple. But I stand by that aspect of the design. cheers, rog. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
On 28 May 2014 19:40, Nate Finch nate.fi...@canonical.com wrote: I think the main problem with errgo as it exists, is that it is very awkward to allow error type information to pass through: return errgo.Mask(err, errgo.Any) This line is very misleading. It's saying - mask the error type, except wait, actually don't mask the type. It's never a good idea to use a function to do the opposite of what it's name says it does. That's partly why I originally called the function Wrap, but was persuaded to change it to Mask by popular request. And, I've said this before and I'm sure I'll say it again - it's actually very rare to need to return *any* error type information to pass through. When I changed all of juju to work this way, there were about 5 occurrences out of hundreds, and all were for good and understandable reasons. Instead, it would be much better to have a method like this: errgo.MaintainErrorTypeAndAddTraceback(err) (with a better name, obviously, but that's not the point) We could just call it Wrap. For the record, I still find the API of errgo to be baffling without a long ramp up period. Having both an underlying error and a cause is hard to wrap my head around. I finally managed to load it all up out of long term storage, so I get it again... but it's not a terribly easy package to understand, in my opinion. I think this is a valid criticism. I agree that having two entirely orthogonal errors is awkward (and Tim's struggles with producing an intuitive error message demonstrate that). I have been toying with the idea of always maintaining a single error stack, but noting the cause by keeping track of its depth in the stack. So to change the cause, you push an error on the stack that has a cause-depth of zero. If you don't need to change the cause, you push an error that has a cause-depth of one more than the cause-depth of the current top-of-stack. To extract or check the cause, you traverse cause-depth levels down the stack and return the error found there. (To make this more efficient, you could store the cause at each level) That means that the cause can be annotated by simply emphasising (with an asterisk or whatever) the error in the stack that's the current cause. The API could remain largely the same, with the exception that WithCausef would go. We'd probably unexport the Underlying_ and Cause_ fields from the Err struct type too. I think I quite like this idea - what do you think? -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: critical regression blocks 1.19.3 release
On 28/05/14 21:54, Curtis Hovey-Canonical wrote: I don't think we can do a release this week, or if we do, the release will be from a version that works from last week. The current blocker is lp:juju-core r2803 or lp:juju-core r2804 broke local precise deployments. Juju cannot bootstrap. https://bugs.launchpad.net/juju-core/+bug/1324255 CI has stumbled from one regression to the next this week. Each fix is followed by another feature that breaks CI. Since we intend to switch to github in a matter of hours, CI has little time to bless a recent revision for release. As I have been doing analysis of the problems, I have not had time to update CI to build tarballs from git. I will now work on that since I don't believe there is anything I can do to convince CI that juju trunk is releasable. It's highly likely that this is caused by the change to have local provider use replica sets. I'll setup a precise VM and see if I can confirm. If so we should back out that change. *sigh* (and sorry if it is indeed the case). Michael -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
HTTP Response for PUT vs POST
We currently have a test that we get 401 Unauthorized when you try to do a PUT instead of a POST for stuff like trying to push up Tools, etc. However, HTTP Spec seems to have a 405 Method Not Allowed, which is meant to handle this case of PUT isn't allowed, but POST would be. I'm looking into changing our Mux layer, to one that separately handles PUT from POST and will give us 405 codes for free. I just wanted to check that it is probably a good idea to conform more to the spec, and be returning 405, I'm guessing 401 here was just because that was what we had on hand. The current way actually checks for POST before checking the Authorization, so an invalid Auth'd PUT would return a 405 rather than 401, but that doesn't seem like bad information leakage. Is that ok? John =:- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: HTTP Response for PUT vs POST
It also returns a text/plain response rather than an application/json one. So we aren't actually encoding the error into a JSON response. I can workaround the Post vs Put stuff by just registering all handlers and leaving the code in place that just checks for only POST, but it seemed a bit of a shame to not leverage a helper that wants to help us out. John =:- On Thu, May 29, 2014 at 3:14 PM, John Meinel j...@arbash-meinel.com wrote: We currently have a test that we get 401 Unauthorized when you try to do a PUT instead of a POST for stuff like trying to push up Tools, etc. However, HTTP Spec seems to have a 405 Method Not Allowed, which is meant to handle this case of PUT isn't allowed, but POST would be. I'm looking into changing our Mux layer, to one that separately handles PUT from POST and will give us 405 codes for free. I just wanted to check that it is probably a good idea to conform more to the spec, and be returning 405, I'm guessing 401 here was just because that was what we had on hand. The current way actually checks for POST before checking the Authorization, so an invalid Auth'd PUT would return a 405 rather than 401, but that doesn't seem like bad information leakage. Is that ok? John =:- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Schema migration process
Team Onyx is accumulating tasks that can't be completed until the schema migrations infrastructure is in place so I've started digging in to the details to figure out what exactly needs to be done. I'm aiming to come up with an approach and some the work items for the initial incarnation of schema migrations. *Overall Approach* Building on John's thoughts, and adding Tim's and mine, here's what I've got so far:: - Introduce a database-version key into the EnvironConfig document which tracks the Juju version that the database schema matches. More on this later. - Introduce a MasterStateServer upgrade target which marks upgrade steps which are only to run on the master state server. Also more below. During an upgrade: - State machine agents get upgraded first - other agents don't see the new agent version until the state machine agents are running the new version (I have double-checked that this is already in place) - When state machines are restarted following an upgrade, don't allow other agents to Login until the upgrade is finished. Allow Client connections however (for monitoring during upgrades as discussed by John). - Non-master JobManageEnviron machine agents run their upgrade steps as usual and then watch for EnvironConfig changes. They don't consider the upgrade to be complete (and therefore let their other workers start) until database-version matches agent-version. This prevents the new version of the state server agents from running before the schema migrations for the new software version have run. - The master machine agent waits for the other JobManageEnviron agents to have upgraded to the new version and then runs the upgrade steps including those for the MasterStateServer upgrade target (these steps will be the schema upgrades). The wait ensures that the database schema isn't upgraded before a state server is ready for the changes (this could happen if a non-master state server is slow or the master is especially fast). - Once the master machine agent has completed its upgrade steps it updates database-version in EnvironConfig to the new Juju version. This signals to the other state machine agents that they can complete their upgrades, allowing their workers to start. - At this point, all state servers have completed their upgrades and allow agent logins. - Now that agents can connect to the upgraded state servers, they see the new agent version and upgrade themselves (this functionality already exists) *Observations/Questions/Issues* - There are a lot of moving parts here. What could be made simpler? - What do we do if the master mongo database or host fails during the upgrade? Is it a goal for one of the other state servers take over and run the schema upgrades itself and let the upgrade finish? If so, is this a must-have up-front requirement or a nice-to-have? - Upgrade steps currently have access to State but I think this probably won't be sufficient to perform many types of schema migrations (i.e. accessing defunct fields, removing fields, adding indexes etc). Do we want to extend State to provide a number of schema migration helpers or do we expose mongo connections directly to the upgrade steps? - There is a possibility that a non-master state server won't upgrade, blocking the master from completing the upgrade. Should there be a timeout before the master gives up on state servers upgrading themselves and performs its own upgrade steps anyway? - Given the order of documents a juju system stores, it's likely that the schema migration steps will be quite quick, even for a large installation. I'm new to all this so please chime in with your suggestions and corrections. Team Onyx (well, me at least) is likely to start on this next week. - Menno p.s. I'm out until Tuesday so it's unlikely I'll see (or at least respond to) any replies until then. On 16 May 2014 17:29, John Meinel j...@arbash-meinel.com wrote: So I'm pretty sure the ability to do schema upgrades was scoped in the HA work that Nate is working on. I believe the idea is roughly: 1) State machines get restarted first (there is code today to not let other agents notice that there is an upgrade ready until the state machine itself has been upgraded) 2) When the state machine is restarted, don't allow other agents to Login until the API Worker has connected to the API Server. This should mean that we are in a relatively quiet mode. 2a) For reasons I'll describe later, still allow Client connections. 3) When the APIWorker comes up and sees that there is an Upgrade that just happened and Upgrade actions are pending, wait to start the upgrade steps until we observe that all other machines with JobManageEnviron have been upgraded to the new version. Continue blocking Login requests. 4) Once all JobManageEnviron machine agent docs report the correct Agent Version, then the Singleton Upgrader (so the one running on the machine with the master Mongo DB), applies the DB Schema Upgrade
Re: critical regression blocks 1.19.3 release
On 28/05/14 21:54, Curtis Hovey-Canonical wrote: I don't think we can do a release this week, or if we do, the release will be from a version that works from last week. The current blocker is lp:juju-core r2803 or lp:juju-core r2804 broke local precise deployments. Juju cannot bootstrap. https://bugs.launchpad.net/juju-core/+bug/1324255 CI has stumbled from one regression to the next this week. Each fix is followed by another feature that breaks CI. Since we intend to switch to github in a matter of hours, CI has little time to bless a recent revision for release. As I have been doing analysis of the problems, I have not had time to update CI to build tarballs from git. I will now work on that since I don't believe there is anything I can do to convince CI that juju trunk is releasable. Hmmm... both Andrew Wilkins and I have tried trunk with precise and had it bootstrap fine. I'll dig in and see if I can work out what is causing the CI failures. Michael -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Sharing environments - a proposal
that makes it easy for Eric to connect to the environment. Now ideally where we'd like to get to is the following: juju connect eric@api-endpoint [env uuid] ... PROBLEM: right now all connections to the api server are secured with TLS and the client-cert. I'm pretty sure none of them use client certs. What they use is a certificate that is signed by a environment-specific CA. So we use the CACert to validate that the API server's certificate is valid, rather than just trusty any TLS connection. However, we *could* just trust the remote site to identify itself if we wanted to. ... We do have the current issue of knowing which end points will be SSL protected and which are TLS with a client-cert, but for now, we know that we need a client cert for the connection. In order to handle this behaviour now, I suggest we do the following: As mentioned, we just have regular TLS with server side certs, we just track the CA Cert so that we know if we can actually trust the cert. John =:- $ juju connect e...@random-aws.com fb5a2570-e6f2-11e3-ac10-0800200c9a66 --client-cert ~/Downloads/cert.txt password: local environment name [foo-production]: This at least moves us in the right direction. Thoughts? Tim UUID are still pretty ugly to pass around. Versus having named environments at API servers. I like having UUIDs be unambiguous under the covers, but I wonder if it is actually nice UI to have people use it for connections. [1] An alternative command name could be 'login'. We should also have an equivalent 'logout' or 'disconnect' that removes the .jenv file (with sufficient warnings about the environment still running). We've talked about juju forget-environment as a way to get rid of a .jenv without actually tearing down the environment. John =:- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Sharing environments - a proposal
[This discussion originated on juju-dev@lists.ubuntu.com - I'm replying to juju-...@lists.canonical.com because it touches on JAAS issues] On 29 May 2014 06:46, Tim Penhey tim.pen...@canonical.com wrote: Hi folks, The onyx team is dealing with sharing environments between multiple people. Currently how this is handled is that often there is a shared file system somewhere that contains the environments directory, and people use juju from there. We are in the middle of dealing with simple user management tasks, so we can at least have multiple different user identities connecting to an environment. Initially all users that can connect to an environment will have admin privileges. We want to ease the way that we share environments. I totally agree that this is an important issue, and it's great that you have been thinking constructively towards possible solutions. This is an excellent place to start considering this area, thanks. The initial work that has been signed off includes the ability for an existing admin to create a new user in state server and to create a .jenv file that is sufficient for that user to connect to the environment. This is half the problem solved. What we want to do is make it easy for a beginner user to start using someone else's environment. Let's say that I have an existing environment, and I want to allow Eric access to that environment. I'll run the command: $ juju user add eric Eric the Viking --output=foo-eric.jenv This will create a .jenv file that contains what is needed to allow Eric to connect to my environment. I now email that file to Eric (or pass it to him on a USB key). Now what? Let's also assume that Eric isn't very familiar with juju, and has just installed it. The proposal: We add a command [1] juju connect that makes it easy for Eric to connect to the environment. Now ideally where we'd like to get to is the following: juju connect eric@api-endpoint [env uuid] Where Eric wouldn't need the .jenv file at all, but this would connect to the API, prompt for a password, and then look for the environment identified by the uuid specified. For current environments where we have only one environment supported by the state server which has the api server that is being connected to, the uuid would be optional, but if it is specified, the command should fail if the uuids don't match. PROBLEM: right now all connections to the api server are secured with TLS and the client-cert. As John says, this isn't actually true - connections are secured with a server cert and a password. Unfortunately I believe it is impossible to lose either one of these without rendering juju fundamentally insecure against man-in-the-middle attacks. If we take the approach you suggest, that's what we'd end up with. Anyone that can subvert the network between the juju connect command and the API server could pretend to be the desired environment, forwarding and changing requests as maliciously as it liked. There's no way that the client can know that it's talking to the middle-man, and no way for the server to know that it's not being addressed by the expected client. There is also the problem that the endpoint can change - with HA the actual API addresses can and will change (and there are many possible addresses too - we try to connect to all of them; that's not very convenient for a small piece of information to copy and paste) The only reasonable solution that I can see is to use some kind of environment broker - a third party web site with a well known DNS name (and a trusted certificate issued to speak for that name) that stores .jenv files for people. Then the user can make a secure connection to that site to fetch a .jenv file that will allow them to make secure connections to their own juju environment. I believe that JAAS (or some component of JAAS) could and probably should potentially fulfil that role, even for environments that it does not manage itself. There is definitely a discussion to be had about whether the service should have plain-text access to the .jenv files (and therefore the user's provider keys) of every juju environment stored in it. That is possible to avoid (along the lines of lastpass.com) which would avoid creating a single point of security failure for every juju environment, but it's a decision that needs to be thought about. I feel it would be nice to avoid building a system that gave the NSA easy and automatic root access to any juju environment. As a first step, it would be quite feasible to make it possible for juju to be able to access an external service to fetch and store .jenv files. FWIW I actually built a prototype that illustrated this some time ago. With such a service, we could imagine an environment variable or configuration file setting that points juju towards the service, allowing any juju operation to work seamlessly in conjunction with remotely created environments.
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
On 2014-05-29 09:50, roger peppe wrote: On 29 May 2014 04:03, Tim Penhey tim.pen...@canonical.com wrote: Errors are worth treating specially here because they're they pass up through multiple layers, so it's very easy to break abstraction boundaries by relying on specific error types. I believe it's very important to *think* about the possible errors that can be returned from a given function, and not just throw up our hands and say you guys 5 layers down, why don't you just communicate *arbitrary stuff* to the guy 3 layers above. It may help to consider this as two problems. One: caller needs to know about a specific failure — e.g. because it's a failure to the callee but not to the caller. Definitely part of the contract. You either: (a) define a super-specific error (exception class, error code, etc.), or (b) document that standard error X means failure Y in this case, and the caller picks up the error as close as possible to its origin. With these errors you have to make sure that the information isn't diluted as it propagates, but usually you don't have to take it too far up the call chain. Two: a caller can deal better with some errors, given more detailed information. You can help by attaching more information to the error (tags, taxonomy, properties) but only on a best-effort basis. You accept that you don't know exactly which errors can come out of the code further down. For example, if you're writing code which speaks to another program over the network, you may want to know: is this connection still usable? Do I know what happened to the operation I requested? Were we able to connect in the first place? Am I doing something that shouldn't ever work, such as mis-spell a command? With these errors you act on the best information you have, but you can always just fail. Jeroen -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
What Roger is talking about, is when an error is implementation-specific and may change if the implementation changes. For example, if you have a black box service that saves data, it might write to the network or it might write to a filesystem, but where it saves is an implementation detail. Let's say in the current implementation, Save() returns a FileNotFoundError if you haven't called Initialize() first, then people may write code that handles a FileNotFoundError by calling Initialize() now if you change the implementation to write to the network, so now if they haven't called Initialize(), you return a NetworkNotFoundError in the same situation their code will break. If you had instead written your own error, or just masked the type of the error to something generic, then they wouldn't have been able to write their code that makes bad assumptions. Note that this is the most simple case of the problem. In reality, it can be that some sub-sub-sub package was returning that specific error, and your code had no idea, it just passed it on up unchanged. This is where it gets insidious, because *your* code didn't change. Some dependency of a dependency changed that you might not even realize you were calling. And now people using your code are broken. If you masked all returned errors except the ones you explicitly wanted to return, then this couldn't happen. On Thu, May 29, 2014 at 3:00 PM, Jeroen Vermeulen jeroen.vermeu...@canonical.com wrote: On 2014-05-29 09:50, roger peppe wrote: On 29 May 2014 04:03, Tim Penhey tim.pen...@canonical.com wrote: Errors are worth treating specially here because they're they pass up through multiple layers, so it's very easy to break abstraction boundaries by relying on specific error types. I believe it's very important to *think* about the possible errors that can be returned from a given function, and not just throw up our hands and say you guys 5 layers down, why don't you just communicate *arbitrary stuff* to the guy 3 layers above. It may help to consider this as two problems. One: caller needs to know about a specific failure — e.g. because it's a failure to the callee but not to the caller. Definitely part of the contract. You either: (a) define a super-specific error (exception class, error code, etc.), or (b) document that standard error X means failure Y in this case, and the caller picks up the error as close as possible to its origin. With these errors you have to make sure that the information isn't diluted as it propagates, but usually you don't have to take it too far up the call chain. Two: a caller can deal better with some errors, given more detailed information. You can help by attaching more information to the error (tags, taxonomy, properties) but only on a best-effort basis. You accept that you don't know exactly which errors can come out of the code further down. For example, if you're writing code which speaks to another program over the network, you may want to know: is this connection still usable? Do I know what happened to the operation I requested? Were we able to connect in the first place? Am I doing something that shouldn't ever work, such as mis-spell a command? With these errors you act on the best information you have, but you can always just fail. Jeroen -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/ mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
Note that this kind of error masking doesn't need to be done in the same package as the base error package. I made a little standalone package http://godoc.org/github.com/natefinch/filterr that does it, we could do something similar if we wanted to keep the error package simpler (which is always a good thing). On Thu, May 29, 2014 at 3:16 PM, Nate Finch nate.fi...@canonical.com wrote: What Roger is talking about, is when an error is implementation-specific and may change if the implementation changes. For example, if you have a black box service that saves data, it might write to the network or it might write to a filesystem, but where it saves is an implementation detail. Let's say in the current implementation, Save() returns a FileNotFoundError if you haven't called Initialize() first, then people may write code that handles a FileNotFoundError by calling Initialize() now if you change the implementation to write to the network, so now if they haven't called Initialize(), you return a NetworkNotFoundError in the same situation their code will break. If you had instead written your own error, or just masked the type of the error to something generic, then they wouldn't have been able to write their code that makes bad assumptions. Note that this is the most simple case of the problem. In reality, it can be that some sub-sub-sub package was returning that specific error, and your code had no idea, it just passed it on up unchanged. This is where it gets insidious, because *your* code didn't change. Some dependency of a dependency changed that you might not even realize you were calling. And now people using your code are broken. If you masked all returned errors except the ones you explicitly wanted to return, then this couldn't happen. On Thu, May 29, 2014 at 3:00 PM, Jeroen Vermeulen jeroen.vermeu...@canonical.com wrote: On 2014-05-29 09:50, roger peppe wrote: On 29 May 2014 04:03, Tim Penhey tim.pen...@canonical.com wrote: Errors are worth treating specially here because they're they pass up through multiple layers, so it's very easy to break abstraction boundaries by relying on specific error types. I believe it's very important to *think* about the possible errors that can be returned from a given function, and not just throw up our hands and say you guys 5 layers down, why don't you just communicate *arbitrary stuff* to the guy 3 layers above. It may help to consider this as two problems. One: caller needs to know about a specific failure — e.g. because it's a failure to the callee but not to the caller. Definitely part of the contract. You either: (a) define a super-specific error (exception class, error code, etc.), or (b) document that standard error X means failure Y in this case, and the caller picks up the error as close as possible to its origin. With these errors you have to make sure that the information isn't diluted as it propagates, but usually you don't have to take it too far up the call chain. Two: a caller can deal better with some errors, given more detailed information. You can help by attaching more information to the error (tags, taxonomy, properties) but only on a best-effort basis. You accept that you don't know exactly which errors can come out of the code further down. For example, if you're writing code which speaks to another program over the network, you may want to know: is this connection still usable? Do I know what happened to the operation I requested? Were we able to connect in the first place? Am I doing something that shouldn't ever work, such as mis-spell a command? With these errors you act on the best information you have, but you can always just fail. Jeroen -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/ mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposed new dependency: github.com/juju/errors (and github.com/juju/errgo)
Masking is often necessary, and information hiding is valuable, and depending on spooky action at a distance is generally foolish. But the granularity with which we want to track the propagation of a given error is *much* finer than the granularity of judicious information hiding: the error is likely to pass through a lot more stack frames than it is package boundaries, and the former transitions are the ones we need to track to properly diagnose errors [0], while the latter are the ones we need to mask to properly maintain global sanity [1]. To mask by default, at every annotation point, leads quickly to code whose action is characterised by bizarre and mystifying errors [2]. To *fail* to mask at sensible boundaries also leads to bugs, and we've seen one or two of those as well; the trouble is that masking is *always easy*, while pure annotation is (relatively) hard. To use a library that makes a point of keeping preservation comparatively verbose and complex is to fundamentally misunderstand the problem. Cheers William [0] it's not a perfect one-to-one match, but it's a reasonable heuristic. [1] it's not a perfect one-to-one match, but it's a reasonable heuristic. [2] this because developers are forced to choose between obscuring their logic with verbose workarounds for the masking... or to just avoid annotation entirely, and hence come to habitually avoid masking even when it's necessary. My lived experience indicates that they will overwhelmingly choose the latter, and that the consequences are unpretty. On Thu, May 29, 2014 at 9:16 PM, Nate Finch nate.fi...@canonical.comwrote: What Roger is talking about, is when an error is implementation-specific and may change if the implementation changes. For example, if you have a black box service that saves data, it might write to the network or it might write to a filesystem, but where it saves is an implementation detail. Let's say in the current implementation, Save() returns a FileNotFoundError if you haven't called Initialize() first, then people may write code that handles a FileNotFoundError by calling Initialize() now if you change the implementation to write to the network, so now if they haven't called Initialize(), you return a NetworkNotFoundError in the same situation their code will break. If you had instead written your own error, or just masked the type of the error to something generic, then they wouldn't have been able to write their code that makes bad assumptions. Note that this is the most simple case of the problem. In reality, it can be that some sub-sub-sub package was returning that specific error, and your code had no idea, it just passed it on up unchanged. This is where it gets insidious, because *your* code didn't change. Some dependency of a dependency changed that you might not even realize you were calling. And now people using your code are broken. If you masked all returned errors except the ones you explicitly wanted to return, then this couldn't happen. On Thu, May 29, 2014 at 3:00 PM, Jeroen Vermeulen jeroen.vermeu...@canonical.com wrote: On 2014-05-29 09:50, roger peppe wrote: On 29 May 2014 04:03, Tim Penhey tim.pen...@canonical.com wrote: Errors are worth treating specially here because they're they pass up through multiple layers, so it's very easy to break abstraction boundaries by relying on specific error types. I believe it's very important to *think* about the possible errors that can be returned from a given function, and not just throw up our hands and say you guys 5 layers down, why don't you just communicate *arbitrary stuff* to the guy 3 layers above. It may help to consider this as two problems. One: caller needs to know about a specific failure — e.g. because it's a failure to the callee but not to the caller. Definitely part of the contract. You either: (a) define a super-specific error (exception class, error code, etc.), or (b) document that standard error X means failure Y in this case, and the caller picks up the error as close as possible to its origin. With these errors you have to make sure that the information isn't diluted as it propagates, but usually you don't have to take it too far up the call chain. Two: a caller can deal better with some errors, given more detailed information. You can help by attaching more information to the error (tags, taxonomy, properties) but only on a best-effort basis. You accept that you don't know exactly which errors can come out of the code further down. For example, if you're writing code which speaks to another program over the network, you may want to know: is this connection still usable? Do I know what happened to the operation I requested? Were we able to connect in the first place? Am I doing something that shouldn't ever work, such as mis-spell a command? With these errors you act on the best information you have, but you can always just fail.
