Nan
That looks like a great way of working around this requirement...
One quick q - how does '*provider.initialized?*' get set?
Other than that, looks great... Will start working up some code now :)
Cheers
Gav
On 19 February 2013 01:47, Nan Liu <nan....@gmail.com> wrote:
> On Mon, Feb 18, 2013 at 3:21 AM, fatmcgav <fatmc...@gmail.com> wrote:
>
>>
>> On 18 February 2013 10:50, Andy Parker <a...@puppetlabs.com> wrote:
>>
>>> I just took a look and see that you got no responses on puppet-users.
>>> That is unfortunate :(
>>>
>>> On Mon, Feb 18, 2013 at 12:13 AM, Gavin Williams <fatmc...@gmail.com>wrote:
>>>
>>>> Morning All
>>>>
>>>> I posted this on Puppet-users a few days ago, but I thought i'd post it
>>>> on here aswell to get a Dev's view-point...
>>>>
>>>> Firstly, apologies for the length of this post, however I thought it
>>>> probably most useful to fully outline the challenge and the desired
>>>> result...
>>>>
>>>> Ok, so we're in the process of Puppetizing our Oracle/NetApp platform
>>>> for Live/DR running.
>>>>
>>>> The current manual process, upon setting up a new database, a set of
>>>> volumes are created to contain the various Oracle DB elements, and these
>>>> are then SnapMirror'd to the DR site.
>>>> This SnapMirror process requires a period of time to copy the base data
>>>> over... This time period is directly relational to the amount of data
>>>> required... I.e. a copy of 20Gb may take an hour, 200Gb may take 10
>>>> hours...
>>>> During this period, the SnapMirror resource is in an 'initializing'
>>>> state. Once the data copy is complete, then the resource will change to an
>>>> 'initialized' state.
>>>> The next step in the process is then to break the relationship so that
>>>> the DR end can be used in a R/W mode...
>>>>
>>>> Now, in order to Puppetize this, I need to be able to replicate the
>>>> above behaviour...
>>>> I've got Puppet to create and initialize the relationship, and that
>>>> works as expected. However Puppet doesn't currently care about the
>>>> relationship state. Now that's easy enough to add in as a new property
>>>> against the type/provider.
>>>>
>>>
>>> Based on how you are describing this, I'm not sure that expressing it as
>>> a parameter is best. It sounds like you are describing a situation where
>>> there are a few states that you care about, but transitioning between those
>>> states requires sitting in other "non-interesting" states for a while.
>>> Describing the "non-interesting" states pushes the management of those
>>> state transitions outside of puppet and possibly makes them harder to work
>>> with.
>>>
>>
>> Ok, that makes sense... Unless I do lots of masking and mapping of the
>> intermediate status' into something that Puppet knows, but again, that adds
>> complication etc...
>>
>>
>>>
>>>
>>>> However what I'm struggling to understand is how, or if it's even
>>>> possible, to automate the switch from 'Initialized' state to a 'Broken'
>>>> state upon completion of the initialization stage???
>>>>
>>>>
>>> Yeah. Normally puppet deals with achieving the desired state in a single
>>> run of puppet. So one possible solution is to have puppet block! I really
>>> don't think that in this situation that would be a good idea, since it
>>> would leave everything else on the machine unmanaged for an unknown length
>>> of time.
>>>
>>
>> Yeh, we could be looking at transfer times of 24-48 hours on some of our
>> larger datasets, so wouldn't want Puppet blocking for that long a period...
>>
>
> So just to explore this a bit. An ensurable resource by default have a
> present absent state, and the transition between them is pretty
> straightforward.
>
> present -> absent (def destroy)
> absent -> present (def create)
>
> I'm assuming present -> absent is short enough you can wait for the
> process to complete, so create is the only problematic state.
>
> For now the closest thing appears to be a transition state that fails
> (intending to block dependent resource):
>
> absent -> initializing -> present
>
> The custom ensurable block:
> ensurable do
> newvalue(:present) do
> unless provider.initialized?
> provider.create
> else
> provider.progress
> end
> end
>
> newvalue(:absent) do
> provider.destroy
> end
>
> newvalue(:initializing) do
> provider.progress
> end
> end
>
> So when a resource is in an initializing state just report back the
> progress status and fail:
> def progress
> percent = File.stat(resource[:name]).size / 100.0
> fail("Creation in progress #{percent}% complete.")
> end
>
> Here's the output example:
>
> # initial create
> $ puppet apply tests/transition.pp
> err: /Stage[main]//Transition[/tmp/demo_a]/ensure: change from absent to
> present failed: Creation in progress 0.0% complete.
> notice: /Stage[main]//Notify[complete]: Dependency Transition[/tmp/demo_a]
> has failures: true
> warning: /Stage[main]//Notify[complete]: Skipping because of failed
> dependencies
> notice: Finished catalog run in 0.08 seconds
>
> # in progress
> $ puppet apply tests/transition.pp
> err: /Stage[main]//Transition[/tmp/demo_a]/ensure: change from absent to
> present failed: Creation in progress 12% complete.
> notice: /Stage[main]//Notify[complete]: Dependency Transition[/tmp/demo_a]
> has failures: true
> warning: /Stage[main]//Notify[complete]: Skipping because of failed
> dependencies
> notice: Finished catalog run in 0.08 seconds
>
> # finished:
> $ puppet apply tests/transition.pp
> notice: complete
> notice: /Stage[main]//Notify[complete]/message: defined 'message' as
> 'complete'
> notice: Finished catalog run in 0.08 seconds
>
> Now these databases definitions are currently driven from a YAML
>>>> backend which maintains information such as database name, volume
>>>> information, primary netapp controller, replication netapp controller,
>>>> etc... Currently, this YAML file is just a file on the puppet master...
>>>> However there are ambitions to move this into a more dynamic backend, such
>>>> as CouchDB or similar... So that opens the possibility to automatically
>>>> update the YAML resource state.. However Puppet still needs to be able to
>>>> support updating that backend based on the information it gets from the
>>>> actual resource...
>>>>
>>>> So to flow it out:
>>>>
>>>> 1. Create a new database in backend ->
>>>> 2. Puppet creates volumes on primary ->
>>>> 3. Data is added to volumes ->
>>>> 4. Backend updated to indicate replication is required ->
>>>> 5. Puppet creates volumes on Secondary and adds Snapmirror
>>>> relationship ->
>>>> 6. Snapmirror initializes in background ->
>>>> 7. Puppet periodically runs against network device and checks
>>>> resource state ->
>>>> 8. Backend resource state is updated following each run? ->
>>>> 9. Snapmirror initialization completes ->
>>>> 10. Puppet runs, detects new resource state and then triggers break?
>>>> 11. Backend resource state updated to 'broken'?
>>>>
>>>> Now 1 to 7 above are fine, but 8 to 11 are where I get a bit unsure...
>>>>
>>> I think you have most of the picture here. Puppet manages some of the
>>> transitions between states in order to get to that final "broken" state.
>>> Using defined resource types or parameterized classes won't get you there
>>> since the information about whether the next step of the management of the
>>> resource can be taken is on the node. As you said earlier, it is once the
>>> snapmirror process reaches the "initialized" state that puppet should
>>> finish its job.
>>>
>>> Since the data needs to come from the node, then there are a couple of
>>> choices:
>>> * a custom fact: doesn't seem good since you would be encoding in
>>> facter the presence of particular resources
>>> * an ENC the probes the Snapmirror system: seems doable, but once
>>> again encodes the presence of particular resources outside the manifests
>>> * a custom type: probably the best solution, the replication itself is
>>> a kind of resource that you want to manage, and what needs to be done is
>>> heavily dependent on the current state and desired state of the resource
>>>
>>> So I would suggest creating a custom type and provider for a "replicated
>>> data" resource, or even try splitting it up into several different
>>> resources. Doing this will let you make the final transition without having
>>> to change the catalog.
>>>
>>> I'll admit, though, that puppet doesn't really have a concept of an "in
>>> progress" convergence of a resource, so I'm not sure how the report will
>>> work out for these kinds of resources. I suspect that it would show a
>>> change every time that puppet runs and the replication is still in progress.
>>>
>>
> The problem is failing is a bit misleading. Certainly an interesting use
> case if we can mark the resource as pending and subsequent resources simply
> noop, but as it stands we can't do anything like this:
>
> $ puppet apply tests/transition.pp
> warning: Could not retrieve fact fqdn
> notice: /Stage[main]//Transition[/tmp/demo_a]/ensure: current_value
> absent, should be present Progress: 0.0 % (pending)
> notice: /Stage[main]//Notify[complete]/message: current_value absent,
> should be complete (noop)
> notice: Finished catalog run in 0.08 seconds
>
> Andy, is this worth filing a feature request?
>
> Thanks,
>
> Nan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+unsubscr...@googlegroups.com.
> To post to this group, send email to puppet-dev@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
