Hi Akila,
Any thoughts? Do we need to discuss interactively, or did my explanation make
sense?
(Sorry to nag, but without this, the grouping/dependency feature being
discussed elsewhere is dead in the water :-)).
Thanks, Shaheed
-----Original Message-----
From: Shaheedur Haque (shahhaqu)
Sent: 06 June 2014 15:34
To: Akila Ravihansa Perera
Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim
(vlim); Reka Thirunavukkarasu (r...@wso2.com); Martin Eppel (meppel); Matt
Turner (matttur)
Subject: RE: FW: Cartridge Agent Extension Points
Hi Akila,
Yes, I have a default port mapping set up:
"portMapping": [
{
"port": "22",
"protocol": "http",
"proxyPort": "8280"
}
],
As per my "trace", I believe what is happening that the long time spent in the
start_servers.sh extension means that we get an onArtifactUpdateEvent() before
we even get to the port testing logic, giving rise to the behaviour I traced.
In other words, the artifact_update raced with start_servers/port_knocking and
got to instance_activated first.
On the extensions, yes, I agree they should all be synchronous as I now believe
they are (for my original email, I had read the code incorrectly and concluded
they were async because I did not see the waitFor() call I was expecting). What
changes are you planning: ensuring that multiple threads/event handlers don't
call extensions at the same time? If so, I agree that is good/necessary.
If it helps, I'd be happy to share a webex or similar to work through this.
Thanks, Shaheed
-----Original Message-----
From: Akila Ravihansa Perera [mailto:raviha...@wso2.com]
Sent: 05 June 2014 17:24
To: Shaheedur Haque (shahhaqu)
Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim
(vlim); Reka Thirunavukkarasu (r...@wso2.com); Martin Eppel (meppel); Matt
Turner (matttur)
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
If you have set port mappings in your cartridge json definition, the agent will
wait (blocking) until all ports are active before sending the instance
activated event and executing instance_activated.sh script.
Are you saying that the sequence you've said is taking place even if you have
set the port mappings in cartridge json? Please confirm.
Following is a sample port mapping.
"portMapping": [
{
"protocol": "https",
"port": "9443",
"proxyPort": "443"
}
],
Please note that we are in the process of improving cartridge agent extension
points. We need a better locking mechanism for synchronous behavior. IMO, every
extension should be synchronous. This is to avoid any conflicts that might
occur when two extensions are executed in parallel.
Thanks.
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu) <shahh...@cisco.com>
wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> From: Nirmal Fernando [mailto:nirmal070...@gmail.com]
> Sent: 05 June 2014 15:10
> To: Shaheedur Haque (shahhaqu)
> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim);
> Reka Thirunavukkarasu (r...@wso2.com); Martin Eppel (meppel); Matt
> Turner (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event
> sends to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance
> Activated J. But problem 1 is real in that the Artifact Update event
> can publish InstanceActivated even though port knocking has not said
> “yes”. This is visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in the
> main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the
> Artifact Updated path, and so is not the overall fix, it merely makes
> sure this race is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> From: Isuru Haththotuwa [mailto:isu...@wso2.com]
> Sent: 05 June 2014 14:26
> To: Nirmal Fernando
> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka
> Thirunavukkarasu (r...@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando
> <nirmal070...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event
> only once. If there is no repo URL, this will happen in the run()
> method of CartridgeAgent class. If there is a repo URL, only once
> after SM sends Artifact Update Event. There is a check in the
> onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute
> the extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Ag
> ent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to
> the left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could
> even argue that just because an artifact update happens, the main
> state of the agent should not be affected. (I know there is another
> discussion about having artifact updates come much earlier in the
> lifecycle, and that further illustrates this point). So, I tried a simple
> change to address item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member
> activation (i.e.
> item 2). If you agree that item 2 should be fixed, then a slightly
> simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would
> you prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC updates.
> We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before*
> + the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.g
> etInstance().getListenAddress(),
>
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated
> event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lak...@wso2.com]
> Sent: 05 June 2014 09:22
> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (r...@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get
> this fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely affect
> the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping to
> deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other
> ways forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> From: Shaheedur Haque (shahhaqu)
> Sent: 31 May 2014 20:36
> To: d...@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel
> (meppel)
> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> Subject: Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless. The
> whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension
> names) should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have
> new synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java
> thread here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
>> Yes, We should have python based agent in 4.1.0 release.
>
>>
>
>>
>
>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
>>
>
>> <raviha...@wso2.com>wrote:
>
>> > Hi all,
>
>> >
>
>> > AFAIK, the Stratos community has decided to put effort in writing a
>
>> > new cartridge-agent using Python. In that case, Java based agent
>> > will
>
>> > become deprecated and all DevOps level extensions will have to be
>> > done
>
>> > using Python. But until then we can use this shell script based model.
>
>> >
>
>> > I'll update the thread once I've finished writing documentation for
>
>> > agent extension points.
>
>> >
>
>> > Thanks.
>
>> >
>
>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
>> >
>
>> > <raviha...@wso2.com> wrote:
>
>> > > Hi Shaheed,
>
>> > >
>
>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
>> > >
>
>> > > <shahh...@cisco.com> wrote:
>
>> > >> Is there any documentation on how this works form the scripter's
>> > >> point
>
>> >
>
>> > of view? Specifically:
>
>> > > I'm working on writing documentation content for cartridge-agent
>
>> > > extension points. Also planning to publish some articles on
>> > > cartridge
>
>> > > deployment use cases (for eg - clustering MongoDB) in which these
>
>> > > extensions are being used.
>
>> > >
>
>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>
>> >
>
>> > /extensions directory), is the new code backwards compatible with
>> > scripts
>
>> > using the /extensions mechanism?
>
>> >
>
>> > > There will be no changes done for previous non-topology events.
>> > > The
>
>> > > new code will use the same /extensions directory mechanism. Only
>> > > the
>
>> > > shell scripts' file names for each extension point will be passed
>> > > as
>
>> > > parameters to the cartridge-agent.
>
>> > >
>
>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder),
>> > >> can
>
>> >
>
>> > you confirm if environment variables set before the JVM is invoked
>> > will
>
>> > still be visible to the shell scripts?
>
>> >
>
>> > > I've tested and confirmed that environment variables set before
>> > > the
>
>> > > JVM is invoked are visible to the shell scripts. On the side
>> > > note, the
>
>> > > JavaProcessBuilder will only append our custom parameters to the
>> > > shell
>
>> > > script process environment block. This is same as writing to
>
>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
>> > > note
>
>> > > that these env variables set by the cartridge-agent are *not*
>> > > shared
>
>> > > globally across other processes.
>
>> > >
>
>> > >> Thanks, Shaheed
>
>> > >>
>
>> > >>
>
>> > >> -----Original Message-----
>
>> > >> From: Akila Ravihansa Perera [mailto:raviha...@wso2.com]
>
>> > >> Sent: 07 May 2014 05:24
>
>> > >> To: d...@stratos.incubator.apache.org
>
>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
>> > >> Subject: Re: Cartridge Agent Extension Points
>
>> > >>
>
>> > >> Pl note that patch is against the master branch (as of yesterday).
>
>> > >>
>
>> > >> Following are the list of event listeners that it will subscribe
>> > >> to 1.
>
>> >
>
>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
>> > SubscriptionDomainsAddedEventListener
>
>> >
>
>> > >> 6. SubscriptionDomainsRemovedEventListener
>
>> > >> 7. CompleteTenantEvent
>
>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
>> >
>
>> > InstanceCleanupMemberEventListener (only locally) 10.
>
>> > InstanceCleanupClusterEventListener (only locally) 11. Instance
>> > Started
>
>> > Event (only locally) 12. Start Servers event (only locally) 13.
>> > Instance
>
>> > Activated Event (only locally)
>
>> >
>
>> > >> These event handlers are called through an interface
>
>> >
>
>> > (ExtensionHandler). Therefore someone else can easily implement
>> > this
>
>> > interface and customize extensions from code level. The main
>> > purpose is
>
>> > however to extend the cartridge agent through shell scripts without
>
>> > changing the code.
>
>> >
>
>> > >> PS - Is there an on-going development effort to develop the
>> > >> cartridge
>
>> >
>
>> > agent in Python or Google GO? If so I would like to get involved :)
>
>> >
>
>> > >> Thanks.
>
>> > >>
>
>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
>> >
>
>> > raviha...@wso2.com> wrote:
>
>> > >>> Hi,
>
>> > >>>
>
>> > >>> I've developed cartridge agent extension points feature for
>> > >>> Stratos
>
>> > >>> cartridge agent. This will work by listening to event messages
>
>> > >>> published by CC, ADC and agents running on instances.
>
>> > >>>
>
>> > >>> Once an event is received it will execute a shell script
>> > >>> (system
>
>> > >>> command) with parameters passed to it as environment variables
>> > >>> through
>
>> > >>> Java ProcessBuilder class. These environment variables are
>> > >>> visible
>
>> > >>> only to that process so there is no risk of dirty read/write.
>
>> > >>>
>
>> > >>> These script names can be set from stratos.sh and these are not
>
>> > >>> hard-coded. Puppet templates can be used to set script names
>> > >>> for each
>
>> > >>> event type. This can be useful when dealing with different
>> > >>> platforms
>
>> > >>> (For eg - Linux, Windows, OS X etc)
>
>> > >>>
>
>> > >>> I had to make some changes to the core messaging component to
>> > >>> get
>
>> > >>> LBClusterID from Member Activated event. This is because it is
>
>> > >>> necessary for LBs to identify which members belong to their LB
>> > >>> cluster
>
>> > >>> when they receive a member activated event. If someone can tell
>> > >>> a
>
>> > >>> workaround, please do share.
>
>> > >>>
>
>> > >>> I have attached the patch in [1] as a JIRA issue.
>
>> > >>>
>
>> > >>> Thanks.
>
>> > >>>
>
>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>> > >>>
>
>> > >>>
>
>> > >>> --
>
>> > >>> Akila Ravihansa Perera
>
>> > >>> Software Engineer
>
>> > >>> WSO2 Inc.
>
>> > >>> http://wso2.com
>
>> > >>>
>
>> > >>> Phone: +94 77 64 154 38
>
>> > >>> Blog: http://ravihansa3000.blogspot.com
>
>> > >>
>
>> > >> --
>
>> > >> Akila Ravihansa Perera
>
>> > >> Software Engineer
>
>> > >> WSO2 Inc.
>
>> > >> http://wso2.com
>
>> > >>
>
>> > >> Phone: +94 77 64 154 38
>
>> > >> Blog: http://ravihansa3000.blogspot.com
>
>> > >
>
>> > > --
>
>> > > Akila Ravihansa Perera
>
>> > > Software Engineer
>
>> > > WSO2 Inc.
>
>> > > http://wso2.com
>
>> > >
>
>> > > Phone: +94 77 64 154 38
>
>> > > Blog: http://ravihansa3000.blogspot.com
>
>> >
>
>> > --
>
>> > Akila Ravihansa Perera
>
>> > Software Engineer
>
>> > WSO2 Inc.
>
>> > http://wso2.com
>
>> >
>
>> > Phone: +94 77 64 154 38
>
>> > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
