Re: [Architecture] Deployment Automation - Clustering with large number of nodes

2015-01-20 Thread Dharshana Warusavitharana
Hi All,

Moving on with Deployment Automation, we are evaluating a tool for
deploying cross platform setup in order to integrate with 3rd party
products with deployment framework.
Main concerns are


   1. Testing actoss multiple environments (OS, JVM, DBs etc).
   2. Limitness of resources to maintain seperate environments of different
   servers, JVMs, OS and diffeent versions of differnt serves.
   3. Eazy to manage configurations.
   4. Ability to work with open stack, EC2 and local networks.

Currently we are planning to eveluate Vagrant [1] based solution to achieve
this

As first phase we are planning to implement a solution based on Virtual box
and OpenStack.

[1]. https://www.vagrantup.com/

Thank You,
Dharshana.

On Mon, Jan 5, 2015 at 3:45 PM, Suhan Dharmasuriya suh...@wso2.com wrote:

 Thanks Manula.
 Will look for the possibility and move the central puppet config.pp file
 contents out of the module.

 Thanks,
 Suhan

 On Mon, Jan 5, 2015 at 3:09 PM, Manula Waidyanatha man...@wso2.com
 wrote:

 Hi Suhan,
 You have to use regex if you want to have different configurations for
 each server (hostname). If you can move all the node specific data out of
 the module (to node.pp or site.pp)

 On Mon, Jan 5, 2015 at 10:45 AM, Suhan Dharmasuriya suh...@wso2.com
 wrote:

 Hi,

 Test Automation team is currently in to RD of deployment automation.
 Currently we are focusing on deployment of WSO2 Application Server
 clusters of worker manager pattern.

 As of now, we have configured deployment automation in a way that the
 cluster node names in a deployment configuration file [1]  and the product
 configuration information (appserver's axis2.xml, carbon.xml, etc...)
 in a config.pp [2] file which will be placed as
 /etc/puppet/modules/appserver/params.pp after running the main python
 script.

 Therefore when new instances are spawned in OpenStack according to the
 node names in config file [1] puppet agents pre-installed in them will
 retrieve their respective xml configurations from
 /etc/puppet/modules/appserver/params.pp file and perform puppet catalog
 runs.

 Refer Github repo [3] python_scripts for more clarifications.

 Currently node names are matched directly against the product
 configuration puppet file $servers names [2].

 Our next step is to give the number of worker/manager subdomain node
 combination and spawn
 the cluster in to existence on OpenStack environment.
 We are planning of fronting an elb for each worker/manager subdomain.

 Then, our deployment.cfg file will be modified as follows by indicating
 the number of sub domain nodes.
 [nodes]
 appserver-mgr = 10
 appserver-wkr = 20

 config.pp file will be as follows.
 $servers = { appserver-mgr = { axis2 = {subDomain = 'mgt', members
 = ['elb1-ip', 'elb2-ip']}, carbon = {subDomain = 'mgt',},
 serverOptions = '-Dsetup', server_home = $carbon_home, }, appserver-
 wkr = { axis2 = {subDomain = 'worker', members = ['elb1-ip', '
 elb2-ip']}, carbon = {subDomain = 'worker',}, serverOptions = '
 -DworkerNode=true', server_home = $carbon_home, }, }


 Instances will be spawned in OpenStack as follows.
 appserver-mgr-1
 appserver-mgr-2
 ...
 appserver-mgr-10
 appserver-wkr-1
 appserver-wkr-2
 ...
 appserver-wkr-20

 How can I match the new server names against the new product
 configuration file?
 Can I use regular expressions? Is it a recommended approach?

 Is there any better alternative for pushing product configurations in
 terms of standards, practices and performance for clusters with large
 number of nodes?

 [1]
 https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/deployment.cfg
 [2]
 https://github.com/suhand/Deployment-Automation/blob/master/deployment-automation-total-puppet/python_scripts/config.pp
 [3]
 https://github.com/suhand/Deployment-Automation/tree/master/deployment-automation-total-puppet

 Thanks,
 Suhan
 --
 Suhan Dharmasuriya
 Software Engineer - Test Automation

 *WSO2, Inc. *

 lean . enterprise . middleware
 Tel: +94 112 145345
 Mob: +94 779 869138
 Blog: http://suhan-opensource.blogspot.com/




 --
 Regards,
 Manula Waidyanatha
 Systems Engineer, WSO2, Inc.
 Phone: +94 774 195853
 Blog: blog.manula.org




 --
 Suhan Dharmasuriya
 Software Engineer - Test Automation

 *WSO2, Inc. *

 lean . enterprise . middleware
 Tel: +94 112 145345
 Mob: +94 779 869138
 Blog: http://suhan-opensource.blogspot.com/




-- 

Dharshana Warusavitharana
Senior Software Engineer , Test Automation
WSO2 Inc. http://wso2.com
email : dharsha...@wso2.com dharsha...@wso2.com
Tel  : +94 11 214 5345
Fax :+94 11 2145300
cell : +94770342233
blog : http://dharshanaw.blogspot.com

lean . enterprise . middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

2015-01-20 Thread Gimantha Bandara
Hi Harshan,

Thanks for the feedback. The list of IDs of the records to be retrieved can
be too long. So setting a list of ids as a query param is not convenient.
Even for the Search, we have to pass a query, which can be too long. Thats
why the post method is used for Search.

Thanks,

On Tue, Jan 20, 2015 at 12:52 PM, Sinthuja Ragendran sinth...@wso2.com
wrote:

 Hi Gimantha,

 I think following the conventions will be more intuitive to the users, and
 we should have a proper reason to not follow. And the post request is
 generally needs to be sent to create the object and the back end is going
 to decide where to create the tables resource, therefore it should be POST
 to '/analytics/tables' and message body should be having the table name. If
 you want to use /analytics/{tableName}, then you should use PUT rather
 POST [1]. But IMO POST is the most suitable operation for this context.

 And through out the below given URL patterns, in order to fetch the
 records from a table, you have used '/analytics/records/{tableName}' url
 pattern where 'records' is in the middle and it is not the correct data
 hierarchy and again I feel it's not the convention. Basically tables
 contains records and not records contains tables. Therefore if we use
 POST to '/analytics/tables' for create table, then you can use simply user
 'analytics/{tableName}' to GET/POST for the tables, IMHO the records is
 just an additional segment which is not needed to be here.

 [1] http://restcookbook.com/HTTP%20Methods/put-vs-post

 Thanks,
 Sinthuja.

 On Tue, Jan 20, 2015 at 1:29 AM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Sinduja,

 Thank you for the feedback.

 On Mon, Jan 19, 2015 at 12:04 PM, Sinthuja Ragendran sinth...@wso2.com
 wrote:

 Hi gimantha,

 Please see the comments inline.

 On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi,
 Currently, I am working on $subject. Basically the methods in
 AnalyticsDataService will be exposed through REST APIs. Please refer to
 Architecture mail thread *[Architecture] BAM 3.0 Data Layer
 Implementation / RDBMS / Distributed Indexing / Search* for more
 Details. Following are the supported REST APIs.

 1. Create a table
 *POST /analytics/{tableName}*
  ; tableName - The name of the table to be created.


 IMHO the above should be POST to '/analytics/*tables*' and the request
 content should have the table name as given below.
 {
  tableName : Test
 }

 Since the POST takes only the table name, it is straightforward to use it
 as a path parameter.


 2. Delete a table
 *DELETE /analytics/{tableName} *
 ; tableName - The name of the table to be deleted.


 3. Check if a table exists
 *GET /analytics/{tableName} *
 ; tableName - The name of the table being checked.

 4. List All the tables
 *GET /analytics/tables*
 ;Response will be an JSON array of table names. e.g. [ table1 ,
 table2 , table3 ]

 5. Get the records from a table.
 *GET /analytics/records/{tableName}/{from}/{to}/{start}/{count} *
 ; tableName - The name of the table from which the records are
 retrieved.
 ; from - The starting time to get records from.
 ; to - The ending time to get records to.
 ; start - The paginated index from value
 ; count - The paginated records count to be read
 ; response - takes the format of the request content of No.7


 Do we need to have 'records' in the URL?  I think it's better to have  
 */analytics/{tableName}/{from}/{to}/{start}/{count}
 *


 records is there to avoid conflicts with other contexts. As an example,
 If we remove records, the URL will take the format
 /analytics/{tableName}, which is already a defined REST API.


 6. Get the records from a table (By IDs)
 *POST /analytics/records/{tableName}*
 ; tableName - The name of the table from which the records are
 retrieved.
 ; Content  - A List of IDs of the records to be retrieved in the
 following format.
 [ id1 , id2 , id3 ]
 ; response - takes the format of the request content of No.7


 Similarly can we have this as * /analytics/{tableName}?*


 7. Create records ( can be created in different tables or in the same )
 *POST /analytics/records*
 ; Content - A list of records in json format like in below.
 [
 {
 id: id1,
 tenantId: -1234,
 tableName: tableName1,
 timestamp: -mm-dd hh:mm:ss,
 values:
 {
 columnName1: value1,
 columnName2: value2
 }
 },
{
 id: id2,
 tenantId: -1234,
 tableName: tableName2,
 timestamp: -mm-dd hh:mm:ss,
 values:
 {
 columnName1: value1,
 columnName2: value2
 }
 },
 ]

 8. Delete records
 *DELETE /analytics/records/{tableName}/{timeFrom}/{timeTo}*
 ; tableName - Name of the table from which the records are deleted.
 ; timeFrom - The starting time to delete records from.
 ; timeTo - The end time to delete records to.


 Again do we need to have 'records' in the middle?  IMHO
 

Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

2015-01-20 Thread Harshan Liyanage
Hi Gimantha,

Could you please explain the use-case for the API  #6? IMO there should not
be any operation to fetch a list of records by ids. Instead there could be
an API which sends a list of records as the response for a query.

For the API #14 you can still use GET method with query strings. I think
you have included start  count parameters to control the pagination. For
that you can use the HTTP range-header [1] or link headers as mentioned in
[2].

[1]. http://otac0n.com/blog/2012/11/21/range-header-i-choose-you.html
[2]. https://developer.github.com/v3/#pagination

Regards,

Lakshitha Harshan
Software Engineer
Mobile: *+94724423048*
Email: hars...@wso2.com
Blog : http://harshanliyanage.blogspot.com/
*WSO2, Inc. :** wso2.com http://wso2.com/*
lean.enterprise.middleware.

On Tue, Jan 20, 2015 at 6:34 PM, Gimantha Bandara giman...@wso2.com wrote:

 Hi Harshan,

 Thanks for the feedback. The list of IDs of the records to be retrieved
 can be too long. So setting a list of ids as a query param is not
 convenient. Even for the Search, we have to pass a query, which can be too
 long. Thats why the post method is used for Search.

 Thanks,

 On Tue, Jan 20, 2015 at 12:52 PM, Sinthuja Ragendran sinth...@wso2.com
 wrote:

 Hi Gimantha,

 I think following the conventions will be more intuitive to the users,
 and we should have a proper reason to not follow. And the post request is
 generally needs to be sent to create the object and the back end is going
 to decide where to create the tables resource, therefore it should be POST
 to '/analytics/tables' and message body should be having the table name. If
 you want to use /analytics/{tableName}, then you should use PUT rather
 POST [1]. But IMO POST is the most suitable operation for this context.

 And through out the below given URL patterns, in order to fetch the
 records from a table, you have used '/analytics/records/{tableName}' url
 pattern where 'records' is in the middle and it is not the correct data
 hierarchy and again I feel it's not the convention. Basically tables
 contains records and not records contains tables. Therefore if we use
 POST to '/analytics/tables' for create table, then you can use simply
 user 'analytics/{tableName}' to GET/POST for the tables, IMHO the records
 is just an additional segment which is not needed to be here.

 [1] http://restcookbook.com/HTTP%20Methods/put-vs-post

 Thanks,
 Sinthuja.

 On Tue, Jan 20, 2015 at 1:29 AM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Sinduja,

 Thank you for the feedback.

 On Mon, Jan 19, 2015 at 12:04 PM, Sinthuja Ragendran sinth...@wso2.com
 wrote:

 Hi gimantha,

 Please see the comments inline.

 On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi,
 Currently, I am working on $subject. Basically the methods in
 AnalyticsDataService will be exposed through REST APIs. Please refer to
 Architecture mail thread *[Architecture] BAM 3.0 Data Layer
 Implementation / RDBMS / Distributed Indexing / Search* for more
 Details. Following are the supported REST APIs.

 1. Create a table
 *POST /analytics/{tableName}*
  ; tableName - The name of the table to be created.


 IMHO the above should be POST to '/analytics/*tables*' and the request
 content should have the table name as given below.
 {
  tableName : Test
 }

 Since the POST takes only the table name, it is straightforward to use
 it as a path parameter.


 2. Delete a table
 *DELETE /analytics/{tableName} *
 ; tableName - The name of the table to be deleted.


 3. Check if a table exists
 *GET /analytics/{tableName} *
 ; tableName - The name of the table being checked.

 4. List All the tables
 *GET /analytics/tables*
 ;Response will be an JSON array of table names. e.g. [ table1 ,
 table2 , table3 ]

 5. Get the records from a table.
 *GET /analytics/records/{tableName}/{from}/{to}/{start}/{count} *
 ; tableName - The name of the table from which the records are
 retrieved.
 ; from - The starting time to get records from.
 ; to - The ending time to get records to.
 ; start - The paginated index from value
 ; count - The paginated records count to be read
 ; response - takes the format of the request content of No.7


 Do we need to have 'records' in the URL?  I think it's better to have  
 */analytics/{tableName}/{from}/{to}/{start}/{count}
 *


 records is there to avoid conflicts with other contexts. As an
 example, If we remove records, the URL will take the format
 /analytics/{tableName}, which is already a defined REST API.


 6. Get the records from a table (By IDs)
 *POST /analytics/records/{tableName}*
 ; tableName - The name of the table from which the records are
 retrieved.
 ; Content  - A List of IDs of the records to be retrieved in the
 following format.
 [ id1 , id2 , id3 ]
 ; response - takes the format of the request content of No.7


 Similarly can we have this as * /analytics/{tableName}?*


 7. Create records ( can be created in different tables or in the same )
 *POST 

Re: [Architecture] Maintaining carbon-store GitHub repo

2015-01-20 Thread Ayesha Dissanayaka
Hi all,

I have completed basic repo separation discussed above along with commit
history of the code base.(https://github.com/ayshsandu/carbon-store). I
still need to include some necessary resource directories to this repo for
it to be fully functional.

There are 3 folders listed under product-es/disribution/src [1]

   - *product-es/disribution/src/dbscripts/storage* [2]
- contains ES storage related dbscripts and exported from
  org.wso2.es.feature and org.wso2.store.feature.
  - In our repo separation we should move this to *carbon-store* repo.
  - I suggest this can be located in below directory structure in
  *carbon-store* repo, and update export paths in org.wso2.es.feature
  and org.wso2.store.feature.
 - *carbon-store/features/resources/dbscripts*
 - Otherwise where should this be located?


   - *product-es/distribution/src/repository *[3]
  - This folder is also exported from org.wso2.es.feature and
  org.wso2.store.feature.
  - There are several folders in this location and as I see most of
  them are ES prodouct specific.
  - I suggest that we keep product specific stuff in
  *product-es/disribution/src/repository* as it is and move other
  necessary files to *carbon-store/features/resources/repository *location
  and update export paths in org.wso2.es.feature and
  org.wso2.store.feature.
  - But I am having hard-time on exactly identifying files in
*product-es/disribution/src/repository
  *location what are ES-product related and what should be exported
  from features. Please, advice on this separation.


   - *product-es/distribution/src/assembly*
  - This contains ES product specific files and will be kept in
  product-es as it is.

Please suggest your concerns on above points.

[1] https://github.com/wso2/product-es/tree/master/modules/distribution/src
[2]
https://github.com/wso2/product-es/tree/master/modules/distribution/src/dbscripts/storage
[3]
https://github.com/wso2/product-es/tree/master/modules/distribution/src/repository

Thanks!
- Ayesha

On Mon, Jan 19, 2015 at 8:10 PM, Sagara Gunathunga sag...@wso2.com wrote:

 +1

 Thanks !

 On Mon, Jan 19, 2015 at 7:33 PM, Ayesha Dissanayaka aye...@wso2.com
 wrote:

 Hi,

 Please refer the repo structure mentioned above as  below.


 ​​
 Sorry about missing it fist-place.

 Thanks!
 -Ayesha

 On Mon, Jan 19, 2015 at 7:24 PM, Ayesha Dissanayaka aye...@wso2.com
 wrote:

 Hi all,

 I have started creating carbon-store GitHub repo as suggested in the
 mail thread Creating carbon-store GitHub repo.

 Along with *components* and *features* we will be moving *apps* and 
 *jaggery-modules
 *into this repo. Finally the main repo structure would be like below.


 ​

- *apps**[1] *will contain jaggeryapps we ship with ES pack and
reused by other products.
- *jaggery-modules* [2] are reusable and required by *apps*

 After completing this repo separation, other products which are using ES
 stuffs will be able to integrate all the ES related components via this 
 *carbon-store
 *repo and will not be depend on *product-es*.

 What are the other inputs on this?
 How have APIM has exposed jaggery related components in this case?

 [1] https://github.com/wso2/product-es/tree/master/modules/apps
 [2]
 https://github.com/wso2/product-es/tree/master/modules/jaggery-modules/

 Thanks!
 - Ayesha
 --
 *Ayesha Dissanayaka*
 Software Engineer,
 WSO2, Inc : http://wso2.com
 http://www.google.com/url?q=http%3A%2F%2Fwso2.comsa=Dsntz=1usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg
 20, Palmgrove Avenue, Colombo 3
 E-Mail: aye...@wso2.com ayshsa...@gmail.com




 --
 *Ayesha Dissanayaka*
 Software Engineer,
 WSO2, Inc : http://wso2.com
 http://www.google.com/url?q=http%3A%2F%2Fwso2.comsa=Dsntz=1usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg
 20, Palmgrove Avenue, Colombo 3
 E-Mail: aye...@wso2.com ayshsa...@gmail.com




 --
 Sagara Gunathunga

 Senior Technical Lead; WSO2, Inc.;  http://wso2.com
 V.P Apache Web Services;http://ws.apache.org/
 Linkedin; http://www.linkedin.com/in/ssagara
 Blog ;  http://ssagara.blogspot.com




-- 
*Ayesha Dissanayaka*
Software Engineer,
WSO2, Inc : http://wso2.com
http://www.google.com/url?q=http%3A%2F%2Fwso2.comsa=Dsntz=1usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg
20, Palmgrove Avenue, Colombo 3
E-Mail: aye...@wso2.com ayshsa...@gmail.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

2015-01-20 Thread Gimantha Bandara
Hi,
Goood point!. Initially the search was implemented such a way that it
returns a list of ids of records that match the query. Now the search
method is changed, so it returns the records. So +1 for removing the API
#6.

@Sinduja, @Harshan
   Thanks for the reference links and for the feedback.

On Tue, Jan 20, 2015 at 6:52 PM, Harshan Liyanage hars...@wso2.com wrote:

 Hi Gimantha,

 Could you please explain the use-case for the API  #6? IMO there should
 not be any operation to fetch a list of records by ids. Instead there could
 be an API which sends a list of records as the response for a query.

 For the API #14 you can still use GET method with query strings. I think
 you have included start  count parameters to control the pagination. For
 that you can use the HTTP range-header [1] or link headers as mentioned in
 [2].

 [1]. http://otac0n.com/blog/2012/11/21/range-header-i-choose-you.html
 [2]. https://developer.github.com/v3/#pagination

 Regards,

 Lakshitha Harshan
 Software Engineer
 Mobile: *+94724423048*
 Email: hars...@wso2.com
 Blog : http://harshanliyanage.blogspot.com/
 *WSO2, Inc. :** wso2.com http://wso2.com/*
 lean.enterprise.middleware.

 On Tue, Jan 20, 2015 at 6:34 PM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Harshan,

 Thanks for the feedback. The list of IDs of the records to be retrieved
 can be too long. So setting a list of ids as a query param is not
 convenient. Even for the Search, we have to pass a query, which can be too
 long. Thats why the post method is used for Search.

 Thanks,

 On Tue, Jan 20, 2015 at 12:52 PM, Sinthuja Ragendran sinth...@wso2.com
 wrote:

 Hi Gimantha,

 I think following the conventions will be more intuitive to the users,
 and we should have a proper reason to not follow. And the post request is
 generally needs to be sent to create the object and the back end is going
 to decide where to create the tables resource, therefore it should be POST
 to '/analytics/tables' and message body should be having the table name. If
 you want to use /analytics/{tableName}, then you should use PUT rather
 POST [1]. But IMO POST is the most suitable operation for this context.

 And through out the below given URL patterns, in order to fetch the
 records from a table, you have used '/analytics/records/{tableName}' url
 pattern where 'records' is in the middle and it is not the correct data
 hierarchy and again I feel it's not the convention. Basically tables
 contains records and not records contains tables. Therefore if we use
 POST to '/analytics/tables' for create table, then you can use simply
 user 'analytics/{tableName}' to GET/POST for the tables, IMHO the records
 is just an additional segment which is not needed to be here.

 [1] http://restcookbook.com/HTTP%20Methods/put-vs-post

 Thanks,
 Sinthuja.

 On Tue, Jan 20, 2015 at 1:29 AM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Sinduja,

 Thank you for the feedback.

 On Mon, Jan 19, 2015 at 12:04 PM, Sinthuja Ragendran sinth...@wso2.com
  wrote:

 Hi gimantha,

 Please see the comments inline.

 On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi,
 Currently, I am working on $subject. Basically the methods in
 AnalyticsDataService will be exposed through REST APIs. Please refer to
 Architecture mail thread *[Architecture] BAM 3.0 Data Layer
 Implementation / RDBMS / Distributed Indexing / Search* for more
 Details. Following are the supported REST APIs.

 1. Create a table
 *POST /analytics/{tableName}*
  ; tableName - The name of the table to be created.


 IMHO the above should be POST to '/analytics/*tables*' and the
 request content should have the table name as given below.
 {
  tableName : Test
 }

 Since the POST takes only the table name, it is straightforward to use
 it as a path parameter.


 2. Delete a table
 *DELETE /analytics/{tableName} *
 ; tableName - The name of the table to be deleted.


 3. Check if a table exists
 *GET /analytics/{tableName} *
 ; tableName - The name of the table being checked.

 4. List All the tables
 *GET /analytics/tables*
 ;Response will be an JSON array of table names. e.g. [ table1 ,
 table2 , table3 ]

 5. Get the records from a table.
 *GET /analytics/records/{tableName}/{from}/{to}/{start}/{count} *
 ; tableName - The name of the table from which the records are
 retrieved.
 ; from - The starting time to get records from.
 ; to - The ending time to get records to.
 ; start - The paginated index from value
 ; count - The paginated records count to be read
 ; response - takes the format of the request content of No.7


 Do we need to have 'records' in the URL?  I think it's better to have  
 */analytics/{tableName}/{from}/{to}/{start}/{count}
 *


 records is there to avoid conflicts with other contexts. As an
 example, If we remove records, the URL will take the format
 /analytics/{tableName}, which is already a defined REST API.


 6. Get the records from a table (By IDs)
 *POST /analytics/records/{tableName}*
 ; 

Re: [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing / Search

2015-01-20 Thread Anjana Fernando
Hi Gimantha,

You haven't really replied to Sinthuja's points there. We have to decide if
we are going ahead with Sinthuja's suggestions, or not. Lets first educate
ourselves on the best practices of RESTful API design a bit, Googling helps
[1]. For example, I'm also not sure if should be
/analytics/{tableName}/records or /analytics/records/{tableName}, each has
benefits, like, first approach can have ambiguity issues, specially when
there other operations with optional fields. And also, I guess we will need
to have /analytics/tables/{tableName} for creating tables etc.. or else,
for example when we do a POST for like /analytics/T1, it may not be that
informative what we are actually doing there.

[1]
https://www.google.com/search?q=REST+service+designgws_rd=ssl#q=REST+api+design

Cheers,
Anjana.

On Wed, Jan 21, 2015 at 3:35 AM, Gimantha Bandara giman...@wso2.com wrote:

 Hi,
 Goood point!. Initially the search was implemented such a way that it
 returns a list of ids of records that match the query. Now the search
 method is changed, so it returns the records. So +1 for removing the API
 #6.

 @Sinduja, @Harshan
Thanks for the reference links and for the feedback.

 On Tue, Jan 20, 2015 at 6:52 PM, Harshan Liyanage hars...@wso2.com
 wrote:

 Hi Gimantha,

 Could you please explain the use-case for the API  #6? IMO there should
 not be any operation to fetch a list of records by ids. Instead there could
 be an API which sends a list of records as the response for a query.

 For the API #14 you can still use GET method with query strings. I think
 you have included start  count parameters to control the pagination. For
 that you can use the HTTP range-header [1] or link headers as mentioned in
 [2].

 [1]. http://otac0n.com/blog/2012/11/21/range-header-i-choose-you.html
 [2]. https://developer.github.com/v3/#pagination

 Regards,

 Lakshitha Harshan
 Software Engineer
 Mobile: *+94724423048*
 Email: hars...@wso2.com
 Blog : http://harshanliyanage.blogspot.com/
 *WSO2, Inc. :** wso2.com http://wso2.com/*
 lean.enterprise.middleware.

 On Tue, Jan 20, 2015 at 6:34 PM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Harshan,

 Thanks for the feedback. The list of IDs of the records to be retrieved
 can be too long. So setting a list of ids as a query param is not
 convenient. Even for the Search, we have to pass a query, which can be too
 long. Thats why the post method is used for Search.

 Thanks,

 On Tue, Jan 20, 2015 at 12:52 PM, Sinthuja Ragendran sinth...@wso2.com
 wrote:

 Hi Gimantha,

 I think following the conventions will be more intuitive to the users,
 and we should have a proper reason to not follow. And the post request is
 generally needs to be sent to create the object and the back end is going
 to decide where to create the tables resource, therefore it should be POST
 to '/analytics/tables' and message body should be having the table name. If
 you want to use /analytics/{tableName}, then you should use PUT rather
 POST [1]. But IMO POST is the most suitable operation for this context.

 And through out the below given URL patterns, in order to fetch the
 records from a table, you have used '/analytics/records/{tableName}' url
 pattern where 'records' is in the middle and it is not the correct data
 hierarchy and again I feel it's not the convention. Basically tables
 contains records and not records contains tables. Therefore if we use
 POST to '/analytics/tables' for create table, then you can use simply
 user 'analytics/{tableName}' to GET/POST for the tables, IMHO the records
 is just an additional segment which is not needed to be here.

 [1] http://restcookbook.com/HTTP%20Methods/put-vs-post

 Thanks,
 Sinthuja.

 On Tue, Jan 20, 2015 at 1:29 AM, Gimantha Bandara giman...@wso2.com
 wrote:

 Hi Sinduja,

 Thank you for the feedback.

 On Mon, Jan 19, 2015 at 12:04 PM, Sinthuja Ragendran 
 sinth...@wso2.com wrote:

 Hi gimantha,

 Please see the comments inline.

 On Sun, Jan 18, 2015 at 11:24 PM, Gimantha Bandara giman...@wso2.com
  wrote:

 Hi,
 Currently, I am working on $subject. Basically the methods in
 AnalyticsDataService will be exposed through REST APIs. Please refer to
 Architecture mail thread *[Architecture] BAM 3.0 Data Layer
 Implementation / RDBMS / Distributed Indexing / Search* for more
 Details. Following are the supported REST APIs.

 1. Create a table
 *POST /analytics/{tableName}*
  ; tableName - The name of the table to be created.


 IMHO the above should be POST to '/analytics/*tables*' and the
 request content should have the table name as given below.
 {
  tableName : Test
 }

 Since the POST takes only the table name, it is straightforward to use
 it as a path parameter.


 2. Delete a table
 *DELETE /analytics/{tableName} *
 ; tableName - The name of the table to be deleted.


 3. Check if a table exists
 *GET /analytics/{tableName} *
 ; tableName - The name of the table being checked.

 4. List All the tables
 *GET /analytics/tables*
 ;Response will be