Hi Shaheedur,
If you have set port mappings in your cartridge json definition, the
agent will wait (blocking) until all ports are active before sending
the instance activated event and executing instance_activated.sh
script.
Are you saying that the sequence you've said is taking place even if
you have set the port mappings in cartridge json? Please confirm.
Following is a sample port mapping.
"portMapping": [
{
"protocol": "https",
"port": "9443",
"proxyPort": "443"
}
],
Please note that we are in the process of improving cartridge agent
extension points. We need a better locking mechanism for synchronous
behavior. IMO, every extension should be synchronous. This is to avoid
any conflicts that might occur when two extensions are executed in
parallel.
Thanks.
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu)
<shahh...@cisco.com> wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> From: Nirmal Fernando [mailto:nirmal070...@gmail.com]
> Sent: 05 June 2014 15:10
> To: Shaheedur Haque (shahhaqu)
> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka
> Thirunavukkarasu (r...@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event sends
> to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance Activated
> J. But problem 1 is real in that the Artifact Update event can publish
> InstanceActivated even though port knocking has not said “yes”. This is
> visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in the
> main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the Artifact
> Updated path, and so is not the overall fix, it merely makes sure this race
> is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> From: Isuru Haththotuwa [mailto:isu...@wso2.com]
> Sent: 05 June 2014 14:26
> To: Nirmal Fernando
> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu
> (r...@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila
> Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <nirmal070...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event only
> once. If there is no repo URL, this will happen in the run() method of
> CartridgeAgent class. If there is a repo URL, only once after SM sends
> Artifact Update Event. There is a check in the onArtifactUpdatedEvent()
> method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that further
> illustrates this point). So, I tried a simple change to address item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port activation
> (i.e. fixing item 1) but does not address multiple member activation (i.e.
> item 2). If you agree that item 2 should be fixed, then a slightly simpler
> fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC updates.
> We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lak...@wso2.com]
> Sent: 05 June 2014 09:22
> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (r...@wso2.com); Martin Eppel
> (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru Haththotuwa
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give more
> in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
> <shahh...@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely affect
> the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping to
> deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> From: Shaheedur Haque (shahhaqu)
> Sent: 31 May 2014 20:36
> To: d...@stratos.incubator.apache.org; Luca Martini (lmartini); Martin Eppel
> (meppel)
> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam; Reka
> Thirunavukkarasu
>
>
> Subject: Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless. The
> whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread here
> (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
>> Yes, We should have python based agent in 4.1.0 release.
>
>>
>
>>
>
>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
>>
>
>> <raviha...@wso2.com>wrote:
>
>> > Hi all,
>
>> >
>
>> > AFAIK, the Stratos community has decided to put effort in writing a
>
>> > new cartridge-agent using Python. In that case, Java based agent will
>
>> > become deprecated and all DevOps level extensions will have to be done
>
>> > using Python. But until then we can use this shell script based model.
>
>> >
>
>> > I'll update the thread once I've finished writing documentation for
>
>> > agent extension points.
>
>> >
>
>> > Thanks.
>
>> >
>
>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
>> >
>
>> > <raviha...@wso2.com> wrote:
>
>> > > Hi Shaheed,
>
>> > >
>
>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
>> > >
>
>> > > <shahh...@cisco.com> wrote:
>
>> > >> Is there any documentation on how this works form the scripter's
>> > >> point
>
>> >
>
>> > of view? Specifically:
>
>> > > I'm working on writing documentation content for cartridge-agent
>
>> > > extension points. Also planning to publish some articles on cartridge
>
>> > > deployment use cases (for eg - clustering MongoDB) in which these
>
>> > > extensions are being used.
>
>> > >
>
>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>
>> >
>
>> > /extensions directory), is the new code backwards compatible with
>> > scripts
>
>> > using the /extensions mechanism?
>
>> >
>
>> > > There will be no changes done for previous non-topology events. The
>
>> > > new code will use the same /extensions directory mechanism. Only the
>
>> > > shell scripts' file names for each extension point will be passed as
>
>> > > parameters to the cartridge-agent.
>
>> > >
>
>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
>> >
>
>> > you confirm if environment variables set before the JVM is invoked will
>
>> > still be visible to the shell scripts?
>
>> >
>
>> > > I've tested and confirmed that environment variables set before the
>
>> > > JVM is invoked are visible to the shell scripts. On the side note, the
>
>> > > JavaProcessBuilder will only append our custom parameters to the shell
>
>> > > script process environment block. This is same as writing to
>
>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
>
>> > > that these env variables set by the cartridge-agent are *not* shared
>
>> > > globally across other processes.
>
>> > >
>
>> > >> Thanks, Shaheed
>
>> > >>
>
>> > >>
>
>> > >> -----Original Message-----
>
>> > >> From: Akila Ravihansa Perera [mailto:raviha...@wso2.com]
>
>> > >> Sent: 07 May 2014 05:24
>
>> > >> To: d...@stratos.incubator.apache.org
>
>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
>> > >> Subject: Re: Cartridge Agent Extension Points
>
>> > >>
>
>> > >> Pl note that patch is against the master branch (as of yesterday).
>
>> > >>
>
>> > >> Following are the list of event listeners that it will subscribe to
>> > >> 1.
>
>> >
>
>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
>> > SubscriptionDomainsAddedEventListener
>
>> >
>
>> > >> 6. SubscriptionDomainsRemovedEventListener
>
>> > >> 7. CompleteTenantEvent
>
>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
>> >
>
>> > InstanceCleanupMemberEventListener (only locally) 10.
>
>> > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
>> > Event (only locally) 12. Start Servers event (only locally) 13. Instance
>
>> > Activated Event (only locally)
>
>> >
>
>> > >> These event handlers are called through an interface
>
>> >
>
>> > (ExtensionHandler). Therefore someone else can easily implement this
>
>> > interface and customize extensions from code level. The main purpose is
>
>> > however to extend the cartridge agent through shell scripts without
>
>> > changing the code.
>
>> >
>
>> > >> PS - Is there an on-going development effort to develop the cartridge
>
>> >
>
>> > agent in Python or Google GO? If so I would like to get involved :)
>
>> >
>
>> > >> Thanks.
>
>> > >>
>
>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
>> >
>
>> > raviha...@wso2.com> wrote:
>
>> > >>> Hi,
>
>> > >>>
>
>> > >>> I've developed cartridge agent extension points feature for Stratos
>
>> > >>> cartridge agent. This will work by listening to event messages
>
>> > >>> published by CC, ADC and agents running on instances.
>
>> > >>>
>
>> > >>> Once an event is received it will execute a shell script (system
>
>> > >>> command) with parameters passed to it as environment variables
>> > >>> through
>
>> > >>> Java ProcessBuilder class. These environment variables are visible
>
>> > >>> only to that process so there is no risk of dirty read/write.
>
>> > >>>
>
>> > >>> These script names can be set from stratos.sh and these are not
>
>> > >>> hard-coded. Puppet templates can be used to set script names for
>> > >>> each
>
>> > >>> event type. This can be useful when dealing with different platforms
>
>> > >>> (For eg - Linux, Windows, OS X etc)
>
>> > >>>
>
>> > >>> I had to make some changes to the core messaging component to get
>
>> > >>> LBClusterID from Member Activated event. This is because it is
>
>> > >>> necessary for LBs to identify which members belong to their LB
>> > >>> cluster
>
>> > >>> when they receive a member activated event. If someone can tell a
>
>> > >>> workaround, please do share.
>
>> > >>>
>
>> > >>> I have attached the patch in [1] as a JIRA issue.
>
>> > >>>
>
>> > >>> Thanks.
>
>> > >>>
>
>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>> > >>>
>
>> > >>>
>
>> > >>> --
>
>> > >>> Akila Ravihansa Perera
>
>> > >>> Software Engineer
>
>> > >>> WSO2 Inc.
>
>> > >>> http://wso2.com
>
>> > >>>
>
>> > >>> Phone: +94 77 64 154 38
>
>> > >>> Blog: http://ravihansa3000.blogspot.com
>
>> > >>
>
>> > >> --
>
>> > >> Akila Ravihansa Perera
>
>> > >> Software Engineer
>
>> > >> WSO2 Inc.
>
>> > >> http://wso2.com
>
>> > >>
>
>> > >> Phone: +94 77 64 154 38
>
>> > >> Blog: http://ravihansa3000.blogspot.com
>
>> > >
>
>> > > --
>
>> > > Akila Ravihansa Perera
>
>> > > Software Engineer
>
>> > > WSO2 Inc.
>
>> > > http://wso2.com
>
>> > >
>
>> > > Phone: +94 77 64 154 38
>
>> > > Blog: http://ravihansa3000.blogspot.com
>
>> >
>
>> > --
>
>> > Akila Ravihansa Perera
>
>> > Software Engineer
>
>> > WSO2 Inc.
>
>> > http://wso2.com
>
>> >
>
>> > Phone: +94 77 64 154 38
>
>> > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
