Re: [Architecture] WSO2 MB Proposed RDBMS Design
Prabath, I agree that triggers make comprehending code a bit more difficult. But triggers are a mechanism to achieve active database features, something that is very difficult to get without triggers. Thus, I would not say that they should not be used. I don't understand the statement that triggers are executed outside of SQL transactions: in case your trigger only makes use of SQL statements (in contrast to generating effects outside of the DBMS) the trigger actions are included in the embracing transaction - at least in DB2, which I am familiar with. Furthermore, the use of stored procedures can be an excellent means to avoid network latency by shuffling data between a client and the DBMS. Thus, it is a means to improve response time, throughput etc. Also, an SQL CALL is part of the embracing transaction. Again, I would not banish the use of stored procedures at all... Best regards, Frank 2014-09-12 10:25 GMT+02:00 Prabath Abeysekera praba...@wso2.com: In addition, triggers, usually, are like stored procedures in its context and at times, might store some good portion of your business logic within the database layer which makes them less visible to someone who's evaluating a particular high-level business use-case just by looking at some code level implementation. So, they usually go unnoticed, I would say, a lot more than it is with a stored procedure. So, IMO, it would be good if we can avoid them whenever possible. Cheers, Prabath On Fri, Sep 12, 2014 at 1:21 PM, Asitha Nanayakkara asi...@wso2.com wrote: According to an offline chat I had with PrabathA SQL triggers should be avoided since SQL triggers execute outside SQL transactions. On Thu, Jul 24, 2014 at 6:30 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi, According to mail arch@ Removing Global Queue from MB, maybe we need to update this design. Better we discuss upfront if so. It will bring changes to messageStore interface. Thanks On Thu, Jul 24, 2014 at 3:04 PM, Asitha Nanayakkara asi...@wso2.com wrote: With the addition of message expiration feature RDBMS design for Metadata in MB needs to be changed. Updated design is as follows. * Design considerations* - Only a subset of messages comes with message expiration and a separate thread handles deletion of expired messages. - Periodically expired message deletion thread queries for a chunk of expired messages. The subset of messages to go through would be less when a separated Expiration table is used. However when deleting a message from Expiration table should trigger a delete of all the messages with the respective message_id from Metadata table. In addition, to update reference count of messages an SQL trigger will be created. Is there any performance hit using triggers? On Thu, Jul 24, 2014 at 12:06 PM, Asitha Nanayakkara asi...@wso2.com wrote: With current design in memory message store will be used only in single node mode. On Wed, Jul 23, 2014 at 11:36 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Also, normally publisher mention whether to persist or not messages in message itself (delivery mode). So based on that MB will process messages in memory and/or persist to a persistence store. So if it's process in memory How do we communicate through the MB cluster? http://activemq.apache.org/what-is-the-difference-between-persistent-and-non-persistent-delivery.html *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 12:36 PM, Dhanuka Ranasinghe dhan...@wso2.com wrote: IMO, If we gonna keep huge messages as chunks in memory and insert into DB as bulk it will heavily affect on MB heap memory. My suggestion is we need to handle this case by case. For example, if it's small messages it will be efficient to keep in memory while huge messages it will be efficient to insert into DB early as possible and let others to use heap memory. For this we will have to make this functionality more configurable but again we will have to think about how gonna support fail over (probably have to change db schema). *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara asi...@wso2.com wrote: We are planning to insert message chunks as batch insert queries. On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Are we going to insert whole message or as chunks On 18 Jul 2014 18:06, Asitha Nanayakkara asi...@wso2.com wrote: Hi, Following is the RDBMS design for WSO2 MB 3.0.0 Messages model Message metadata model Following are the concerns came across in the discussion *- Why we use reference counting for message meta data?* Reference counting is needed to delete topic messages from the
Re: [Architecture] WSO2 MB Proposed RDBMS Design
This is true in general: even a simple statement like SELECT behaves a bit differently in different RDBMS implementations. I.e. minor differences in SQL statements is not restricted to triggers, however we use RDBMS... Best regards, Frank 2014-09-12 10:24 GMT+02:00 Sumedha Rubasinghe sume...@wso2.com: If you use triggers, you cannot have a database engine independent design. On Fri, Sep 12, 2014 at 1:21 PM, Asitha Nanayakkara asi...@wso2.com wrote: According to an offline chat I had with PrabathA SQL triggers should be avoided since SQL triggers execute outside SQL transactions. On Thu, Jul 24, 2014 at 6:30 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi, According to mail arch@ Removing Global Queue from MB, maybe we need to update this design. Better we discuss upfront if so. It will bring changes to messageStore interface. Thanks On Thu, Jul 24, 2014 at 3:04 PM, Asitha Nanayakkara asi...@wso2.com wrote: With the addition of message expiration feature RDBMS design for Metadata in MB needs to be changed. Updated design is as follows. * Design considerations* - Only a subset of messages comes with message expiration and a separate thread handles deletion of expired messages. - Periodically expired message deletion thread queries for a chunk of expired messages. The subset of messages to go through would be less when a separated Expiration table is used. However when deleting a message from Expiration table should trigger a delete of all the messages with the respective message_id from Metadata table. In addition, to update reference count of messages an SQL trigger will be created. Is there any performance hit using triggers? On Thu, Jul 24, 2014 at 12:06 PM, Asitha Nanayakkara asi...@wso2.com wrote: With current design in memory message store will be used only in single node mode. On Wed, Jul 23, 2014 at 11:36 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Also, normally publisher mention whether to persist or not messages in message itself (delivery mode). So based on that MB will process messages in memory and/or persist to a persistence store. So if it's process in memory How do we communicate through the MB cluster? http://activemq.apache.org/what-is-the-difference-between-persistent-and-non-persistent-delivery.html *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 12:36 PM, Dhanuka Ranasinghe dhan...@wso2.com wrote: IMO, If we gonna keep huge messages as chunks in memory and insert into DB as bulk it will heavily affect on MB heap memory. My suggestion is we need to handle this case by case. For example, if it's small messages it will be efficient to keep in memory while huge messages it will be efficient to insert into DB early as possible and let others to use heap memory. For this we will have to make this functionality more configurable but again we will have to think about how gonna support fail over (probably have to change db schema). *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara asi...@wso2.com wrote: We are planning to insert message chunks as batch insert queries. On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Are we going to insert whole message or as chunks On 18 Jul 2014 18:06, Asitha Nanayakkara asi...@wso2.com wrote: Hi, Following is the RDBMS design for WSO2 MB 3.0.0 Messages model Message metadata model Following are the concerns came across in the discussion *- Why we use reference counting for message meta data?* Reference counting is needed to delete topic messages from the database reliably in a cluster deployment *- How to manage a large tables like Messages table?* for Messages table use database partitioning For Metadata queries there will be no SQL joins, hence even if the table would grow large that won't be an issue. Inserts and delete operation can be done as batch operations. *- Following option to save metadata was rejected due to following reasons* In the above design MB instance will create Node_Queue tables and Topic_Node_Queue tables when each node connects to a cluster. This design was rejected due to following reasons. It's DB admins tasks to create and delete tables. MB should not modify schema when joining to the cluster. There will be instances where MB users might not have privileges to create tables. *- Supporting several SQL implementations.* Since we are using simple SQL operations those will not become an issue. Thanks -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 ___ Architecture mailing
Re: [Architecture] WSO2 MB Proposed RDBMS Design
Frank, To make testing and development simple, we try to avoid database specific code as much as possible with in Carbon platform. Currently most of the products support close to 9 RDBMS engines. 99% of the SQLs we have in our DAO classes are ANSI compatible. However there are database specific DDLs under product-name/dbscripts folder. But these only contain DDLs that creates tables, indexes, triggers for identity in a db specific way. We have not used triggers for any other purpose. Although stored procedures provide heavy advantages, we have avoided using them due to the complexity of managing them for different databases. On Sep 14, 2014 4:19 PM, Frank Leymann fr...@wso2.com wrote: This is true in general: even a simple statement like SELECT behaves a bit differently in different RDBMS implementations. I.e. minor differences in SQL statements is not restricted to triggers, however we use RDBMS... Best regards, Frank 2014-09-12 10:24 GMT+02:00 Sumedha Rubasinghe sume...@wso2.com: If you use triggers, you cannot have a database engine independent design. On Fri, Sep 12, 2014 at 1:21 PM, Asitha Nanayakkara asi...@wso2.com wrote: According to an offline chat I had with PrabathA SQL triggers should be avoided since SQL triggers execute outside SQL transactions. On Thu, Jul 24, 2014 at 6:30 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi, According to mail arch@ Removing Global Queue from MB, maybe we need to update this design. Better we discuss upfront if so. It will bring changes to messageStore interface. Thanks On Thu, Jul 24, 2014 at 3:04 PM, Asitha Nanayakkara asi...@wso2.com wrote: With the addition of message expiration feature RDBMS design for Metadata in MB needs to be changed. Updated design is as follows. * Design considerations* - Only a subset of messages comes with message expiration and a separate thread handles deletion of expired messages. - Periodically expired message deletion thread queries for a chunk of expired messages. The subset of messages to go through would be less when a separated Expiration table is used. However when deleting a message from Expiration table should trigger a delete of all the messages with the respective message_id from Metadata table. In addition, to update reference count of messages an SQL trigger will be created. Is there any performance hit using triggers? On Thu, Jul 24, 2014 at 12:06 PM, Asitha Nanayakkara asi...@wso2.com wrote: With current design in memory message store will be used only in single node mode. On Wed, Jul 23, 2014 at 11:36 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Also, normally publisher mention whether to persist or not messages in message itself (delivery mode). So based on that MB will process messages in memory and/or persist to a persistence store. So if it's process in memory How do we communicate through the MB cluster? http://activemq.apache.org/what-is-the-difference-between-persistent-and-non-persistent-delivery.html *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 12:36 PM, Dhanuka Ranasinghe dhan...@wso2.com wrote: IMO, If we gonna keep huge messages as chunks in memory and insert into DB as bulk it will heavily affect on MB heap memory. My suggestion is we need to handle this case by case. For example, if it's small messages it will be efficient to keep in memory while huge messages it will be efficient to insert into DB early as possible and let others to use heap memory. For this we will have to make this functionality more configurable but again we will have to think about how gonna support fail over (probably have to change db schema). *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara asi...@wso2.com wrote: We are planning to insert message chunks as batch insert queries. On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Are we going to insert whole message or as chunks On 18 Jul 2014 18:06, Asitha Nanayakkara asi...@wso2.com wrote: Hi, Following is the RDBMS design for WSO2 MB 3.0.0 Messages model Message metadata model Following are the concerns came across in the discussion *- Why we use reference counting for message meta data?* Reference counting is needed to delete topic messages from the database reliably in a cluster deployment *- How to manage a large tables like Messages table?* for Messages table use database partitioning For Metadata queries there will be no SQL joins, hence even if the table would grow large that won't be an issue. Inserts and delete operation can be done as batch operations. *- Following option to save metadata was rejected due to
Re: [Architecture] WSO2 MB Proposed RDBMS Design
Sumedha, sure, I understand: I also developed products that had to support multiple RDBMS. But we ended up developing RDBMS specific code for performance reasons. This of course depends on customer requirements, i.e. as long as there are no customer complains, you are doing the right things :-) Best regards, Frank 2014-09-14 14:12 GMT+02:00 Sumedha Rubasinghe sume...@wso2.com: Frank, To make testing and development simple, we try to avoid database specific code as much as possible with in Carbon platform. Currently most of the products support close to 9 RDBMS engines. 99% of the SQLs we have in our DAO classes are ANSI compatible. However there are database specific DDLs under product-name/dbscripts folder. But these only contain DDLs that creates tables, indexes, triggers for identity in a db specific way. We have not used triggers for any other purpose. Although stored procedures provide heavy advantages, we have avoided using them due to the complexity of managing them for different databases. On Sep 14, 2014 4:19 PM, Frank Leymann fr...@wso2.com wrote: This is true in general: even a simple statement like SELECT behaves a bit differently in different RDBMS implementations. I.e. minor differences in SQL statements is not restricted to triggers, however we use RDBMS... Best regards, Frank 2014-09-12 10:24 GMT+02:00 Sumedha Rubasinghe sume...@wso2.com: If you use triggers, you cannot have a database engine independent design. On Fri, Sep 12, 2014 at 1:21 PM, Asitha Nanayakkara asi...@wso2.com wrote: According to an offline chat I had with PrabathA SQL triggers should be avoided since SQL triggers execute outside SQL transactions. On Thu, Jul 24, 2014 at 6:30 PM, Hasitha Hiranya hasit...@wso2.com wrote: Hi, According to mail arch@ Removing Global Queue from MB, maybe we need to update this design. Better we discuss upfront if so. It will bring changes to messageStore interface. Thanks On Thu, Jul 24, 2014 at 3:04 PM, Asitha Nanayakkara asi...@wso2.com wrote: With the addition of message expiration feature RDBMS design for Metadata in MB needs to be changed. Updated design is as follows. * Design considerations* - Only a subset of messages comes with message expiration and a separate thread handles deletion of expired messages. - Periodically expired message deletion thread queries for a chunk of expired messages. The subset of messages to go through would be less when a separated Expiration table is used. However when deleting a message from Expiration table should trigger a delete of all the messages with the respective message_id from Metadata table. In addition, to update reference count of messages an SQL trigger will be created. Is there any performance hit using triggers? On Thu, Jul 24, 2014 at 12:06 PM, Asitha Nanayakkara asi...@wso2.com wrote: With current design in memory message store will be used only in single node mode. On Wed, Jul 23, 2014 at 11:36 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Also, normally publisher mention whether to persist or not messages in message itself (delivery mode). So based on that MB will process messages in memory and/or persist to a persistence store. So if it's process in memory How do we communicate through the MB cluster? http://activemq.apache.org/what-is-the-difference-between-persistent-and-non-persistent-delivery.html *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 12:36 PM, Dhanuka Ranasinghe dhan...@wso2.com wrote: IMO, If we gonna keep huge messages as chunks in memory and insert into DB as bulk it will heavily affect on MB heap memory. My suggestion is we need to handle this case by case. For example, if it's small messages it will be efficient to keep in memory while huge messages it will be efficient to insert into DB early as possible and let others to use heap memory. For this we will have to make this functionality more configurable but again we will have to think about how gonna support fail over (probably have to change db schema). *Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara asi...@wso2.com wrote: We are planning to insert message chunks as batch insert queries. On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe dhan...@wso2.com wrote: Are we going to insert whole message or as chunks On 18 Jul 2014 18:06, Asitha Nanayakkara asi...@wso2.com wrote: Hi, Following is the RDBMS design for WSO2 MB 3.0.0 Messages model Message metadata model Following are the concerns came across in the discussion *- Why we use reference counting for message meta data?* Reference counting is needed to delete topic messages
Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory
Hi Anuruddha, Is the build server mentioned above for the test cases to run is a separate one or the one we already have inside AF? *Kasun de Silva* Software Engineer | *WSO2 Inc.*; http://wso2.com lean.enterprise.middleware email : kas...@wso2.com mobile : +94 77 794 4260 On Sun, Sep 14, 2014 at 9:07 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi Madhawa, Right now we are adding this support based on a configuration flag in the apptype.xml. But yes, we have to bring this as an improvement to add user option in the AF UI. On Fri, Sep 12, 2014 at 9:04 AM, Madhawa Bandara madh...@wso2.com wrote: Hi Anurudda, So if the test support is added per app type, tests are added to an application by default or a user is able to select it as an option? Sent from Windows phone -- From: Anuruddha Premalal anurud...@wso2.com Sent: 9/10/2014 15:36 To: architecture@wso2.org Subject: Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory Hi Danushka, Basically we need to have a repo for tests to trigger builds separately (where the build server node have browser launching capabilities). Do you have alternative suggestions for this? Yes users will be able to download logs related to the test via UI; Hope to add this new UI features in other services tab. WDYT? On Wed, Sep 10, 2014 at 11:54 AM, Danushka Fernando danush...@wso2.com wrote: Hi Anuruddha On Wed, Sep 10, 2014 at 10:16 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi All, We are planning to bring selenium test support to Appfactory web apps. Following is the initial architecture. * Selenium tests will added to a web app based on a configuration flag on the apptype xml. - Test suit creation will happen in the app creation phase. - Idea is to create a separate git repository for the test suit. Why we need a seperate git repo. If we do so IMO we need to version that repo also in same way we version the master repo. * Users will able to run tests if there are any changes in the selenium default test suit. - These test will run on a build server installed on a machine with web browser launch support. (install ubuntu desktop package) - We are avoiding headless mode in selenium because of some javascript execution issues which can cause due this mode. So will the user get the report of test output. IMO it should be downloadable from AF ui. WDYT? * Tests specific user actions and test results will categorize under the other services tab in Appfactory. Your feedback is welcome. Regards, -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ESB Connector for CodePlex
Hi All, We had an issue with codeplex hook subscription where it returns 500 internal server error when we subscribe for a Code Change event and enable it. So we have informed this matter to codeplex and waiting for a response. Anyway we will proceed with developments of hook subscription create and updating operations. Regards Tharik On Mon, Aug 25, 2014 at 5:14 PM, Vanjikumaran Sivajothy va...@wso2.com wrote: Hi, Seems good and send us the detail project plan. Best Regards, Vanji On Mon, Aug 25, 2014 at 2:24 PM, Tharik Kanaka tha...@wso2.com wrote: Hi All, Myself and pumudu (pum...@wso2.com) did the initial review of CodePlex connector and decided to implement it for WSO2 ESB. *CodePlex Introduction* CodePlex is an open source project hosting website provided by Microsoft. It consists of several features such as wiki pages, source controlling (using mercurial, Team Foundation Server, Subversion or Git), discussion forums, issue tracking, project tagging, RSS support, statistics and releases. There are many .Net projects (Windows Forms, WPF, ASP.Net, Share point) in codeplex which are collaboratively developed and maintained by community. *CodePlex HTTP API* CodePlex HTTP provides a REST interface where sending and receiving of data is carried out with JSON. Resources and operations of the API is stated below. - Users - Get a user - Get the authenticated user - Projects - Get a user's projects - Get the authenticated user's projects - Hooks - Get the list of supported service hooks - Get the list of subscriptions for a service hook - Get a subscription - Create a subscription - Update a subscription - Delete a subscription *API Version* 2012-09-01 (x-ms-version: 2012-09-01) *Authentication * Supports OAuth2 protocol (which allows applications to act on behalf of users without requiring passwords. These authorisations can also be revoked by users at anytime) *Documentation URL* https://www.codeplex.com/site/developers Thanks, Tharik -- *Tharik Kanaka* | Associate Software Engineer WSO2, Inc |#20, Palm Grove, Colombo 03, Sri Lanka Mobile: +94 77 1616060 | Work: +94 112145345 Email: tha...@wso2.com | Web: www.wso2.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Sivajothy Vanjikumaran *Senior Software Engineer* *Integration Technologies Team* *WSO2 Inc. http://wso2.com http://wso2.com/* *Mobile:(+94)777219209* [image: Facebook] https://www.facebook.com/vanjikumaran [image: Twitter] https://twitter.com/vanjikumaran [image: LinkedIn] http://www.linkedin.com/pub/vanjikumaran-sivajothy/25/b31/293 [image: Blogger] http://vanjikumaran.blogspot.com/ [image: SlideShare] http://www.slideshare.net/vanjikumaran This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, re-transmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Tharik Kanaka* | Associate Software Engineer WSO2, Inc |#20, Palm Grove, Colombo 03, Sri Lanka Email: tha...@wso2.com | Web: www.wso2.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory
Hi Kasun, We want to run selenium test in headless mode, this requires the build server instance to be installed in an environment with browser launching capabilities. And it doesn't mean we cannot use the same server which we used to build apps. We'll use this existing one as for the initial step. WDYT? Regards, Anuruddha. On Sun, Sep 14, 2014 at 10:12 PM, Kasun De Silva kas...@wso2.com wrote: Hi Anuruddha, Is the build server mentioned above for the test cases to run is a separate one or the one we already have inside AF? *Kasun de Silva* Software Engineer | *WSO2 Inc.*; http://wso2.com lean.enterprise.middleware email : kas...@wso2.com mobile : +94 77 794 4260 On Sun, Sep 14, 2014 at 9:07 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi Madhawa, Right now we are adding this support based on a configuration flag in the apptype.xml. But yes, we have to bring this as an improvement to add user option in the AF UI. On Fri, Sep 12, 2014 at 9:04 AM, Madhawa Bandara madh...@wso2.com wrote: Hi Anurudda, So if the test support is added per app type, tests are added to an application by default or a user is able to select it as an option? Sent from Windows phone -- From: Anuruddha Premalal anurud...@wso2.com Sent: 9/10/2014 15:36 To: architecture@wso2.org Subject: Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory Hi Danushka, Basically we need to have a repo for tests to trigger builds separately (where the build server node have browser launching capabilities). Do you have alternative suggestions for this? Yes users will be able to download logs related to the test via UI; Hope to add this new UI features in other services tab. WDYT? On Wed, Sep 10, 2014 at 11:54 AM, Danushka Fernando danush...@wso2.com wrote: Hi Anuruddha On Wed, Sep 10, 2014 at 10:16 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi All, We are planning to bring selenium test support to Appfactory web apps. Following is the initial architecture. * Selenium tests will added to a web app based on a configuration flag on the apptype xml. - Test suit creation will happen in the app creation phase. - Idea is to create a separate git repository for the test suit. Why we need a seperate git repo. If we do so IMO we need to version that repo also in same way we version the master repo. * Users will able to run tests if there are any changes in the selenium default test suit. - These test will run on a build server installed on a machine with web browser launch support. (install ubuntu desktop package) - We are avoiding headless mode in selenium because of some javascript execution issues which can cause due this mode. So will the user get the report of test output. IMO it should be downloadable from AF ui. WDYT? * Tests specific user actions and test results will categorize under the other services tab in Appfactory. Your feedback is welcome. Regards, -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory
Yes IMO that's a better plan for the initial setup, since I have tried using a separate build server for some other feature of AF and it needs lot of effort. So better to go with the same build server which we use to build apps for now. Regards, Kasun *Kasun de Silva* Software Engineer | *WSO2 Inc.*; http://wso2.com lean.enterprise.middleware email : kas...@wso2.com mobile : +94 77 794 4260 On Mon, Sep 15, 2014 at 9:26 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi Kasun, We want to run selenium test in headless mode, this requires the build server instance to be installed in an environment with browser launching capabilities. And it doesn't mean we cannot use the same server which we used to build apps. We'll use this existing one as for the initial step. WDYT? Regards, Anuruddha. On Sun, Sep 14, 2014 at 10:12 PM, Kasun De Silva kas...@wso2.com wrote: Hi Anuruddha, Is the build server mentioned above for the test cases to run is a separate one or the one we already have inside AF? *Kasun de Silva* Software Engineer | *WSO2 Inc.*; http://wso2.com lean.enterprise.middleware email : kas...@wso2.com mobile : +94 77 794 4260 On Sun, Sep 14, 2014 at 9:07 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi Madhawa, Right now we are adding this support based on a configuration flag in the apptype.xml. But yes, we have to bring this as an improvement to add user option in the AF UI. On Fri, Sep 12, 2014 at 9:04 AM, Madhawa Bandara madh...@wso2.com wrote: Hi Anurudda, So if the test support is added per app type, tests are added to an application by default or a user is able to select it as an option? Sent from Windows phone -- From: Anuruddha Premalal anurud...@wso2.com Sent: 9/10/2014 15:36 To: architecture@wso2.org Subject: Re: [Architecture] [APPFAC][FEATURE] Selenium test integration toAppfactory Hi Danushka, Basically we need to have a repo for tests to trigger builds separately (where the build server node have browser launching capabilities). Do you have alternative suggestions for this? Yes users will be able to download logs related to the test via UI; Hope to add this new UI features in other services tab. WDYT? On Wed, Sep 10, 2014 at 11:54 AM, Danushka Fernando danush...@wso2.com wrote: Hi Anuruddha On Wed, Sep 10, 2014 at 10:16 AM, Anuruddha Premalal anurud...@wso2.com wrote: Hi All, We are planning to bring selenium test support to Appfactory web apps. Following is the initial architecture. * Selenium tests will added to a web app based on a configuration flag on the apptype xml. - Test suit creation will happen in the app creation phase. - Idea is to create a separate git repository for the test suit. Why we need a seperate git repo. If we do so IMO we need to version that repo also in same way we version the master repo. * Users will able to run tests if there are any changes in the selenium default test suit. - These test will run on a build server installed on a machine with web browser launch support. (install ubuntu desktop package) - We are avoiding headless mode in selenium because of some javascript execution issues which can cause due this mode. So will the user get the report of test output. IMO it should be downloadable from AF ui. WDYT? * Tests specific user actions and test results will categorize under the other services tab in Appfactory. Your feedback is welcome. Regards, -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Anuruddha Premalal* Software Eng. | WSO2 Inc. Mobile : +94710461070 Web site : www.regilandvalley.com ___ Architecture mailing list Architecture@wso2.org
Re: [Architecture] A mechanism to communicate server shutdown to the bundles, just before actual server shutdown
Hi Amila, Sorry about the confusion. Generally, in OSGi bundles, we register services provided by a particular bundle in the bundle activation, That is true. But in this scenario, I am registering the CarbonServerShutdownPrepareService in the org.wso2.carbon.core.init.CarbonServerManager.shutdownGracefully() method, which is called when the server about to shutdown. As actual server shutting down process is not yet started, all the bundles are active and the server is still in the full operational mode. So, we can register a new service without any harm. In OSGi framework, service registration, activation of dependent bundles, notifying listers and trackers happen in synchronized manner. So the shutdownGracefully() cannot proceed until the above service registration tasks completes. So, the bundles, which was waiting for the CarbonServerShutdownPrepareService can start doing its expected tasks (house keeping tasks that has to be carried out just before the server shutdown) Thanks Jayanga. *Jayanga Dissanayake* Senior Software Engineer WSO2 Inc. - http://wso2.com/ lean . enterprise . middleware email: jaya...@wso2.com mobile: +94772207259 On Sun, Sep 14, 2014 at 1:36 AM, Amila Maha Arachchi ami...@wso2.com wrote: On Thu, Sep 11, 2014 at 10:47 PM, Jayanga Dissanayake jaya...@wso2.com wrote: Hi, There was a requirement to detect the server shutdown from some bundles before the OSGi framework begin to shutdown. Because there were situations where some bundle have there own transports. If server shutdown happens while there were buffered messages, those should be processes before that particular bundle get deactivated. As the bundle stopping sequence is not guaranteed in OSGi environments, There may be some required bundles/services already being stopped when that particular bundle wants to do the finalizing tasks. So, as a solution I tried the following, Register an OSGi service CarbonServerShutdownPrepareService in the shutdownGracefully() before the actual server shutdown is called. So, just before the server shutdown CarbonServerShutdownPrepareService get activated. Then add some dummy OSGi Declarative Service components, that is waiting for ”CarbonServerShutdownPrepareService”, Then start the server and shutdown the server, Observations, When the server is shutting down, 1. first it registers the service “CarbonServerShutdownPrepareService” 2. then all the bundles waiting for that service get activated, (if there are listers or trackers, those get notified) 3. then actual server shutdown happens AFAIK, OSGi services are registered when the components startup. Here, you have said that CarbonServerShutdownPrepareService is registered when the server is shut down. This is not clear to me. Did you mean the following approach: 1. Register an OSGi service at the server startup. 2. Add declarative service dependencies to the bundles which needs to be notified about the shutdown. These bundles should implement a listener or a tracker. 3. When the server shutdown is called, it notifies the dependents. Is this sequential? It seems that I don't understand something here. If it is the case, please explain it to me. :) Thanks. As the osgi framework's service registration and un-registration happens in a single thread, It is guaranteed that registration of “CarbonServerShutdownPrepareService” and notification of that registration to the other bundles happens in a sequential manner in a single thread. So, any bundle can perform there house keeping tasks that has to carry out just before the server shutdown. Suggestions, improvements and alternatives are welcome Thanks, *Jayanga Dissanayake* Senior Software Engineer WSO2 Inc. - http://wso2.com/ lean . enterprise . middleware email: jaya...@wso2.com mobile: +94772207259 -- *Amila Maharachchi* Senior Technical Lead WSO2, Inc.; http://wso2.com Blog: http://maharachchi.blogspot.com Mobile: +94719371446 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] A mechanism to communicate server shutdown to the bundles, just before actual server shutdown
Hi Jayanga, Why can't register the CarbonServerShutdownPrepareService in normal OSGi service registration manner (without registering it in server shutdown)? Is that because you need to have synchronized manner or any other particular reason to do so? Thanks, Best Regards, Lakshitha Harshan Software Engineer Mobile: *+94724423048* Email: hars...@wso2.com Blog : http://harshanliyanage.blogspot.com/ *WSO2, Inc. :** wso2.com http://wso2.com/* lean.enterprise.middleware. On Mon, Sep 15, 2014 at 10:13 AM, Jayanga Dissanayake jaya...@wso2.com wrote: Hi Amila, Sorry about the confusion. Generally, in OSGi bundles, we register services provided by a particular bundle in the bundle activation, That is true. But in this scenario, I am registering the CarbonServerShutdownPrepareService in the org.wso2.carbon.core.init.CarbonServerManager.shutdownGracefully() method, which is called when the server about to shutdown. As actual server shutting down process is not yet started, all the bundles are active and the server is still in the full operational mode. So, we can register a new service without any harm. In OSGi framework, service registration, activation of dependent bundles, notifying listers and trackers happen in synchronized manner. So the shutdownGracefully() cannot proceed until the above service registration tasks completes. So, the bundles, which was waiting for the CarbonServerShutdownPrepareService can start doing its expected tasks (house keeping tasks that has to be carried out just before the server shutdown) Thanks Jayanga. *Jayanga Dissanayake* Senior Software Engineer WSO2 Inc. - http://wso2.com/ lean . enterprise . middleware email: jaya...@wso2.com mobile: +94772207259 On Sun, Sep 14, 2014 at 1:36 AM, Amila Maha Arachchi ami...@wso2.com wrote: On Thu, Sep 11, 2014 at 10:47 PM, Jayanga Dissanayake jaya...@wso2.com wrote: Hi, There was a requirement to detect the server shutdown from some bundles before the OSGi framework begin to shutdown. Because there were situations where some bundle have there own transports. If server shutdown happens while there were buffered messages, those should be processes before that particular bundle get deactivated. As the bundle stopping sequence is not guaranteed in OSGi environments, There may be some required bundles/services already being stopped when that particular bundle wants to do the finalizing tasks. So, as a solution I tried the following, Register an OSGi service CarbonServerShutdownPrepareService in the shutdownGracefully() before the actual server shutdown is called. So, just before the server shutdown CarbonServerShutdownPrepareService get activated. Then add some dummy OSGi Declarative Service components, that is waiting for ”CarbonServerShutdownPrepareService”, Then start the server and shutdown the server, Observations, When the server is shutting down, 1. first it registers the service “CarbonServerShutdownPrepareService” 2. then all the bundles waiting for that service get activated, (if there are listers or trackers, those get notified) 3. then actual server shutdown happens AFAIK, OSGi services are registered when the components startup. Here, you have said that CarbonServerShutdownPrepareService is registered when the server is shut down. This is not clear to me. Did you mean the following approach: 1. Register an OSGi service at the server startup. 2. Add declarative service dependencies to the bundles which needs to be notified about the shutdown. These bundles should implement a listener or a tracker. 3. When the server shutdown is called, it notifies the dependents. Is this sequential? It seems that I don't understand something here. If it is the case, please explain it to me. :) Thanks. As the osgi framework's service registration and un-registration happens in a single thread, It is guaranteed that registration of “CarbonServerShutdownPrepareService” and notification of that registration to the other bundles happens in a sequential manner in a single thread. So, any bundle can perform there house keeping tasks that has to carry out just before the server shutdown. Suggestions, improvements and alternatives are welcome Thanks, *Jayanga Dissanayake* Senior Software Engineer WSO2 Inc. - http://wso2.com/ lean . enterprise . middleware email: jaya...@wso2.com mobile: +94772207259 -- *Amila Maharachchi* Senior Technical Lead WSO2, Inc.; http://wso2.com Blog: http://maharachchi.blogspot.com Mobile: +94719371446 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Some thoughts of BAM Capacity Planing
On Mon, Aug 25, 2014 at 1:44 PM, Srinath Perera srin...@wso2.com wrote: Hi All, BAM capacity planning it bit involved than other products. I was doing one such case with Mifan and following are steps we followed. Please comment. 1. Need to know roughly how much data you need to handle. One way to get this is knowing the event rate. Lets say 100 TPS. That means if each record is 2K, then it is about 18GB per day, about 550GB per month, and about 6.5TB per year. 2. First Questions is how long we keep data. Generally, this should be like 1 month. You need enough disk to hold the data. If you have 3 replicas you need 3X that, and Cassandra also add an overhead ( http://www.datastax.com/documentation/cassandra/1.2/cassandra/architecture/architecturePlanningUserData_t.html ) 3. Then think how much data you will be processing with Hive. If you process once a day, each processing will pull like 18GB. Need to make sure there is enough computing power. Hadoop is mostly decided by number of cores, and Cassandra we follow data stacks recommendations. I think core can do about 20MB second (only a guess). See http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/ . --Srinath -- Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/ -- Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
