[Status Update] Apache Cassandra backend for Sling
Hi Ian, I am starting this thread to keep track on things related to the GSoC project related milestone status updates and related discussions. So the first task over view will be as follows as per GSoC proposal provided. 1. Implementing a CassandraResourceProvider to READ from Cassandra. Implementation Details [1] [1] : Implementation Details: 1.A) Write a CassanrdaResourceProviderUtil which is basically a cassendra client which will facilitate all cassandra related operations required by other modules (CassandraResourceProvider and CassandraResourceResolver). 1.B) Implementation of CassandraResourceProvider 1.C) Implementation of CassandraResourceResolver 1.D) Implementation of CassandraResource And I will start writing the CassanrdaResourceProviderUtil class which will do basic add and get using hector API. Please provide any feedback that will be useful to accomplish this task. So for this how does path mapping should be done. Because for example, the path of the cassendra node will not be same as the jcr node path. i.e provider will ask a node path /system/myapps/test/foo and where should we return it from Cassandra. Aren't we have to first consider the WRITE aspect to Cassandra ? -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
Hi Ian, What is the data mapping should be between Cassandra and Sling resource. I mean is a Sling Resource maps to a Cassandra Column ? Or Column Family ? Because to get this Cassandra and Sling story correct we need to finalize this. For an example what we eventually returns is a Sling resource. Everything that needs to fill in to create Sling resource should be stored in Cassandra. In a Sling resource, - Path - direct sling resource path - ResourceType - nt:cassandra - ResourceSuperType - ? - ResourceMetadata - we can create this on the fly with the data from the corresponding column. At insertion, those need to be stored. Following are the ones which I thought might be useful by default to be set for any node. Please add if we need anything more. - ContentType - ContentLength - CreationTime - ModificationTime - ResourceResolver - Do we need a resolver in this case ? So I believe in CQL context, one ROW should represent a Sling resource. If that is the case for ResourceMetadata we might need a separate column to store it since it has multiple values. I am not sure whether we can do it with CQL, but it should be possible with hector APIs may be. Appreciate your thoughts ? On Wed, Jun 19, 2013 at 1:19 AM, Dishara Wijewardana < ddwijeward...@gmail.com> wrote: > Hi Ian, > I am starting this thread to keep track on things related to the GSoC > project related milestone status updates and related discussions. > So the first task over view will be as follows as per GSoC proposal > provided. > > 1. Implementing a CassandraResourceProvider to READ from Cassandra. > Implementation Details [1] > > > > [1] : Implementation Details: > > 1.A) Write a CassanrdaResourceProviderUtil which is basically a > cassendra client which will facilitate all cassandra related operations > required by other modules (CassandraResourceProvider and > CassandraResourceResolver). > > 1.B) Implementation of CassandraResourceProvider > > 1.C) Implementation of CassandraResourceResolver > > 1.D) Implementation of CassandraResource > > > And I will start writing the CassanrdaResourceProviderUtil class which > will do basic add and get using hector API. Please provide any feedback > that will be useful to accomplish this task. > So for this how does path mapping should be done. Because for example, the > path of the cassendra node will not be same as the jcr node path. i.e > provider will ask a node path /system/myapps/test/foo and where should we > return it from Cassandra. Aren't we have to first consider the WRITE aspect > to Cassandra ? > > > -- > Thanks > /Dishara > -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
Hi Dishara, Yes. 1 resource == 1 row. The columns within that row represent the properties of the resource. I suggest that you use standard property names where appropriate (eg sling:resourceType is the Resource.resourceType etc) The Resource itself should be adaptable to a generic CassandraResource (which will probably implement Resource) which will have a map of properties containing all the columns of the cassandra row. (optimise later) A CassandraResource might look and feel like a Map or it might have a Map getProperties() method, or better still be adaptable to a Map. The essential think is dont hard code the property names in the interface of CassandraResource for the moment. ie no getContentType() and no getMimeType(), as we dont really know what a CassandraResource will store. ResourceMetadata should be built from a subset of the CassandraResource properties. You won't need to implement a ResourceResolver, only a ResourceProvider (and Factory). I would use CQL in preference to other API methods. There is one thing that hasnt been mentioned, and thats the URL -> Cassandra Row mapping. There are several ways of doing this. eg: URL = /content/cassandra// Cassandra Column Family = columnFamily Cassandra RowID = rowID or URL = /content/cassandra//remainder/of/the/path Cassandra Cassandra Column Family = mapOfColumnFamilies.get(columnFamilySelector) Cassandra RowID = function(/remainder/of/the/path) or to take that one stage further public interface CassandraMapper { String getCQL(String columnFamilySelector, String path); } URL = /content/cassandra// String cqlQuery = mapOfCassandraMappers.get(columnFamilySelector).getCQL(columnFamilySelector, remainderOfPath); Which would allow us provided one or more implementations of CassandraMapper to map between URL and CQL. HTH Ian On 23 June 2013 19:29, Dishara Wijewardana wrote: > Hi Ian, > > What is the data mapping should be between Cassandra and Sling resource. I > mean is a Sling Resource maps to a Cassandra Column ? Or Column Family ? > > Because to get this Cassandra and Sling story correct we need to finalize > this. > For an example what we eventually returns is a Sling resource. Everything > that needs to fill in to create Sling resource should be stored in > Cassandra. > In a Sling resource, > >- Path - direct sling resource path >- ResourceType - nt:cassandra >- ResourceSuperType - ? >- ResourceMetadata - we can create this on the fly with the data from >the corresponding column. At insertion, those need to be stored. > Following >are the ones which I thought might be useful by default to be set for > any >node. Please add if we need anything more. > - ContentType > - ContentLength > - CreationTime > - ModificationTime >- ResourceResolver - Do we need a resolver in this case ? > > > So I believe in CQL context, one ROW should represent a Sling resource. If > that is the case for ResourceMetadata we might need a separate column to > store it since it has multiple values. I am not sure whether we can do it > with CQL, but it should be possible with hector APIs may be. > > Appreciate your thoughts ? > > > On Wed, Jun 19, 2013 at 1:19 AM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > > > Hi Ian, > > I am starting this thread to keep track on things related to the GSoC > > project related milestone status updates and related discussions. > > So the first task over view will be as follows as per GSoC proposal > > provided. > > > > 1. Implementing a CassandraResourceProvider to READ from Cassandra. > > Implementation Details [1] > > > > > > > > [1] : Implementation Details: > > > > 1.A) Write a CassanrdaResourceProviderUtil which is basically a > > cassendra client which will facilitate all cassandra related operations > > required by other modules (CassandraResourceProvider and > > CassandraResourceResolver). > > > > 1.B) Implementation of CassandraResourceProvider > > > > 1.C) Implementation of CassandraResourceResolver > > > > 1.D) Implementation of CassandraResource > > > > > > And I will start writing the CassanrdaResourceProviderUtil class which > > will do basic add and get using hector API. Please provide any feedback > > that will be useful to accomplish this task. > > So for this how does path mapping should be done. Because for example, > the > > path of the cassendra node will not be same as the jcr node path. i.e > > provider will ask a node path /system/myapps/test/foo and where should we > > return it from Cassandra. Aren't we have to first consider the WRITE > aspect > > to Cassandra ? > > > > > > -- > > Thanks > > /Dishara > > > > > > -- > Thanks > /Dishara >
Re: [Status Update] Apache Cassandra backend for Sling
On Mon, Jun 24, 2013 at 4:02 AM, Ian Boston wrote: > Hi Dishara, > Yes. 1 resource == 1 row. > The columns within that row represent the properties of the resource. > I suggest that you use standard property names where appropriate (eg > sling:resourceType is the Resource.resourceType etc) > > The Resource itself should be adaptable to a generic CassandraResource > (which will probably implement Resource) which will have a map of > properties containing all the columns of the cassandra row. (optimise > later) A CassandraResource might look and feel like a Map > or it might have a Map getProperties() method, or better > still be adaptable to a Map. The essential think is dont hard code the > property names in the interface of CassandraResource for the moment. ie no > getContentType() and no getMimeType(), as we dont really know what a > CassandraResource will store. > > ResourceMetadata should be built from a subset of the CassandraResource > properties. > > You won't need to implement a ResourceResolver, only a ResourceProvider > (and Factory). I would use CQL in preference to other API methods. > > There is one thing that hasnt been mentioned, and thats the URL -> > Cassandra Row mapping. > There are several ways of doing this. > > eg: > URL = /content/cassandra// > Cassandra Column Family = columnFamily > Cassandra RowID = rowID > or > URL = /content/cassandra//remainder/of/the/path > Cassandra Cassandra Column Family = > mapOfColumnFamilies.get(columnFamilySelector) > Cassandra RowID = function(/remainder/of/the/path) > > or to take that one stage further > > public interface CassandraMapper { > String getCQL(String columnFamilySelector, String path); > } > Hi Ian Thank you for the detailed explanation. OK. +1 for this approach with the mentioned flexibility.But I need a small clarification. With this approach, URL = /content/cassandra/ROW-ID ROW-ID - function(/remainder/of/the/path). So you mean ROW-ID is something we have to programatically uniquely create right ? like a UUID. What is this "/remainder/of/the/path" means ? Can you give an example with real values in the context of a user who want to obtain a resource from cassandra. This is just for my understanding. > > URL = /content/cassandra// > > String cqlQuery = > > mapOfCassandraMappers.get(columnFamilySelector).getCQL(columnFamilySelector, > remainderOfPath); > > Which would allow us provided one or more implementations of > CassandraMapper to map between URL and CQL. > > > HTH > Ian > > > > > > > > > > > > > > > > > On 23 June 2013 19:29, Dishara Wijewardana > wrote: > > > Hi Ian, > > > > What is the data mapping should be between Cassandra and Sling resource. > I > > mean is a Sling Resource maps to a Cassandra Column ? Or Column Family ? > > > > Because to get this Cassandra and Sling story correct we need to finalize > > this. > > For an example what we eventually returns is a Sling resource. Everything > > that needs to fill in to create Sling resource should be stored in > > Cassandra. > > In a Sling resource, > > > >- Path - direct sling resource path > >- ResourceType - nt:cassandra > >- ResourceSuperType - ? > >- ResourceMetadata - we can create this on the fly with the data from > >the corresponding column. At insertion, those need to be stored. > > Following > >are the ones which I thought might be useful by default to be set for > > any > >node. Please add if we need anything more. > > - ContentType > > - ContentLength > > - CreationTime > > - ModificationTime > >- ResourceResolver - Do we need a resolver in this case ? > > > > > > So I believe in CQL context, one ROW should represent a Sling resource. > If > > that is the case for ResourceMetadata we might need a separate column to > > store it since it has multiple values. I am not sure whether we can do it > > with CQL, but it should be possible with hector APIs may be. > > > > Appreciate your thoughts ? > > > > > > On Wed, Jun 19, 2013 at 1:19 AM, Dishara Wijewardana < > > ddwijeward...@gmail.com> wrote: > > > > > Hi Ian, > > > I am starting this thread to keep track on things related to the GSoC > > > project related milestone status updates and related discussions. > > > So the first task over view will be as follows as per GSoC proposal > > > provided. > > > > > > 1. Implementing a CassandraResourceProvider to READ from Cassandra. > > > Implementation Details [1] > > > > > > > > > > > > [1] : Implementation Details: > > > > > > 1.A) Write a CassanrdaResourceProviderUtil which is basically a > > > cassendra client which will facilitate all cassandra related operations > > > required by other modules (CassandraResourceProvider and > > > CassandraResourceResolver). > > > > > > 1.B) Implementation of CassandraResourceProvider > > > > > > 1.C) Implementation of CassandraResourceResolver > > > > > > 1.D) Implementation of CassandraResource > > > > > > > > > And I will start writing the Cassanrda
Re: [Status Update] Apache Cassandra backend for Sling
Hi, (I might have errors in the CQL, Cassandra schema and the functions need proper escaping) Example 1: Zero depth tree wiht UUID as the rowid or key. URL /content/cassandra/pictures/13f58d5c95c70b6f then the column family is pictures and the URL -> ROWID function just results in the ROWID being 13f58d5c95c70b6f and String cql = mapOfCassandraMappers.get("pictures").getCQL("pictures", " 13f58d5c95c70b6f") System.err.println(cql); where String getCQL(String cf, String path) { return "select * from "+cf+" where rowid = '"+path+"'"; } yields: select * from pictures where rowid = '13f58d5c95c70b6f' 13f58d5c95c70b6f would be generated by the application when the user created a new picture (by upload). Example 2: User specified URL /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf String cql = mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " capacitors/electrolytic/axial/16v/10uf") System.err.println(cql); where String getCQL(String cf, String path) { MessageDigest md = MessageDigest.getInstance("SHA1"); String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); return "select * from "+cf+" where rowid = '"+rowID+"'"; } yields select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' If you want to find the parent then mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " capacitors/electrolytic/axial/16v") select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' And if the parent is stored in the property parent then select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' will generate a list of children. (Not sure about performance) Example 3: User is allowed to enter the RowID directly (identical to Example 1 URL /content/cassandra/cannesfilmfestival/TomCruiseCassino-20130402112345-ieb.jpg where String getCQL(String cf, String path) { return "select * from "+cf+" where rowid = '"+path+"'"; } yields: select * from pictures where rowid = ' TomCruiseCassino-20130402112345-ieb.jpg' Does that make sense ? Ian On 25 June 2013 05:29, Dishara Wijewardana wrote: > On Mon, Jun 24, 2013 at 4:02 AM, Ian Boston wrote: > > > Hi Dishara, > > Yes. 1 resource == 1 row. > > The columns within that row represent the properties of the resource. > > I suggest that you use standard property names where appropriate (eg > > sling:resourceType is the Resource.resourceType etc) > > > > The Resource itself should be adaptable to a generic CassandraResource > > (which will probably implement Resource) which will have a map of > > properties containing all the columns of the cassandra row. (optimise > > later) A CassandraResource might look and feel like a Map > > or it might have a Map getProperties() method, or better > > still be adaptable to a Map. The essential think is dont hard code the > > property names in the interface of CassandraResource for the moment. ie > no > > getContentType() and no getMimeType(), as we dont really know what a > > CassandraResource will store. > > > > ResourceMetadata should be built from a subset of the CassandraResource > > properties. > > > > You won't need to implement a ResourceResolver, only a ResourceProvider > > (and Factory). I would use CQL in preference to other API methods. > > > > There is one thing that hasnt been mentioned, and thats the URL -> > > Cassandra Row mapping. > > There are several ways of doing this. > > > > eg: > > URL = /content/cassandra// > > Cassandra Column Family = columnFamily > > Cassandra RowID = rowID > > or > > URL = /content/cassandra//remainder/of/the/path > > Cassandra Cassandra Column Family = > > mapOfColumnFamilies.get(columnFamilySelector) > > Cassandra RowID = function(/remainder/of/the/path) > > > > or to take that one stage further > > > > public interface CassandraMapper { > > String getCQL(String columnFamilySelector, String path); > > } > > > Hi Ian > Thank you for the detailed explanation. > > OK. +1 for this approach with the mentioned flexibility.But I need a small > clarification. With this approach, > > URL = /content/cassandra/ROW-ID > ROW-ID - function(/remainder/of/the/path). > So you mean ROW-ID is something we have to programatically uniquely create > right ? like a UUID. > > What is this "/remainder/of/the/path" means ? Can you give an example with > real values in the context of a user who want to obtain a resource from > cassandra. > This is just for my understanding. > > > > > > > URL = /content/cassandra// > > > > String cqlQuery = > > > > > mapOfCassandraMappers.get(columnFamilySelector).getCQL(columnFamilySelector, > > remainderOfPath); > > > > Which would allow us provided one or more implementations of > > CassandraMapper to map between URL and CQL. > > > > > > HTH > > Ian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 23 June 2013 19:29, Dishara Wijewardana > > wrote: > > > > > Hi Ian, > > > > > > What is the data mapping should be between Cassandra and Sling > resource. > >
Re: [Status Update] Apache Cassandra backend for Sling
On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: > Hi, > > (I might have errors in the CQL, Cassandra schema and the functions need > proper escaping) > > > Example 1: > Zero depth tree wiht UUID as the rowid or key. > > URL /content/cassandra/pictures/13f58d5c95c70b6f > > then the column family is pictures and the URL -> ROWID function just > results in the ROWID being 13f58d5c95c70b6f and > > String cql = mapOfCassandraMappers.get("pictures").getCQL("pictures", " > 13f58d5c95c70b6f") > System.err.println(cql); > > where > String getCQL(String cf, String path) { > return "select * from "+cf+" where rowid = '"+path+"'"; > } > > yields: > select * from pictures where rowid = '13f58d5c95c70b6f' > > > 13f58d5c95c70b6f would be generated by the application when the user > created a new picture (by upload). > > > > Example 2: > User specified > > URL /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf > > String cql = mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > capacitors/electrolytic/axial/16v/10uf") > System.err.println(cql); > > where > String getCQL(String cf, String path) { > MessageDigest md = MessageDigest.getInstance("SHA1"); > String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); > return "select * from "+cf+" where rowid = '"+rowID+"'"; > } > > yields > > select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' > > If you want to find the parent then > > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > capacitors/electrolytic/axial/16v") > > select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > And if the parent is stored in the property parent then > > select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > will generate a list of children. (Not sure about performance) > > > Example 3: > User is allowed to enter the RowID directly (identical to Example 1 > URL > > /content/cassandra/cannesfilmfestival/TomCruiseCassino-20130402112345-ieb.jpg > > where > String getCQL(String cf, String path) { > return "select * from "+cf+" where rowid = '"+path+"'"; > } > > yields: > select * from pictures where rowid = ' > TomCruiseCassino-20130402112345-ieb.jpg' > This should be corrected as select * from cannesfilmfestival where rowid = ' TomCruiseCassino-20130402112345-ieb.jpg' > > > Does that make sense ? > Hi Ian, I was in fact practicing some cql stuff in related to this response (with cassandra cql terminal). This is quite a wonderful explanation for a new comer like me. Thank you very much for the explanation again. Now it really makes sense. Other than the zero depth approach, I believe users will be more comfortable with Example 2 approach. Shall we go ahead with it ? > Ian > > > > > On 25 June 2013 05:29, Dishara Wijewardana > wrote: > > > On Mon, Jun 24, 2013 at 4:02 AM, Ian Boston wrote: > > > > > Hi Dishara, > > > Yes. 1 resource == 1 row. > > > The columns within that row represent the properties of the resource. > > > I suggest that you use standard property names where appropriate (eg > > > sling:resourceType is the Resource.resourceType etc) > > > > > > The Resource itself should be adaptable to a generic CassandraResource > > > (which will probably implement Resource) which will have a map of > > > properties containing all the columns of the cassandra row. (optimise > > > later) A CassandraResource might look and feel like a Map Object> > > > or it might have a Map getProperties() method, or > better > > > still be adaptable to a Map. The essential think is dont hard code the > > > property names in the interface of CassandraResource for the moment. ie > > no > > > getContentType() and no getMimeType(), as we dont really know what a > > > CassandraResource will store. > > > > > > ResourceMetadata should be built from a subset of the CassandraResource > > > properties. > > > > > > You won't need to implement a ResourceResolver, only a ResourceProvider > > > (and Factory). I would use CQL in preference to other API methods. > > > > > > There is one thing that hasnt been mentioned, and thats the URL -> > > > Cassandra Row mapping. > > > There are several ways of doing this. > > > > > > eg: > > > URL = /content/cassandra// > > > Cassandra Column Family = columnFamily > > > Cassandra RowID = rowID > > > or > > > URL = /content/cassandra//remainder/of/the/path > > > Cassandra Cassandra Column Family = > > > mapOfColumnFamilies.get(columnFamilySelector) > > > Cassandra RowID = function(/remainder/of/the/path) > > > > > > or to take that one stage further > > > > > > public interface CassandraMapper { > > > String getCQL(String columnFamilySelector, String path); > > > } > > > > > Hi Ian > > Thank you for the detailed explanation. > > > > OK. +1 for this approach with the mentioned flexibility.But I need a > small > > clarification. With this approach, > > > > URL = /content/cassandra/ROW-ID > > ROW-ID - function(/remainder/of/the/path). > > So you mean ROW-ID is somethin
Re: [Status Update] Apache Cassandra backend for Sling
On 27 June 2013 02:34, Dishara Wijewardana wrote: > On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: > > > Hi, > > > > (I might have errors in the CQL, Cassandra schema and the functions need > > proper escaping) > > > > > > Example 1: > > Zero depth tree wiht UUID as the rowid or key. > > > > URL /content/cassandra/pictures/13f58d5c95c70b6f > > > > then the column family is pictures and the URL -> ROWID function just > > results in the ROWID being 13f58d5c95c70b6f and > > > > String cql = mapOfCassandraMappers.get("pictures").getCQL("pictures", " > > 13f58d5c95c70b6f") > > System.err.println(cql); > > > > where > > String getCQL(String cf, String path) { > > return "select * from "+cf+" where rowid = '"+path+"'"; > > } > > > > yields: > > select * from pictures where rowid = '13f58d5c95c70b6f' > > > > > > 13f58d5c95c70b6f would be generated by the application when the user > > created a new picture (by upload). > > > > > > > > Example 2: > > User specified > > > > URL /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf > > > > String cql = mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > > capacitors/electrolytic/axial/16v/10uf") > > System.err.println(cql); > > > > where > > String getCQL(String cf, String path) { > > MessageDigest md = MessageDigest.getInstance("SHA1"); > > String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); > > return "select * from "+cf+" where rowid = '"+rowID+"'"; > > } > > > > yields > > > > select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' > > > > If you want to find the parent then > > > > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > > capacitors/electrolytic/axial/16v") > > > > select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > > > And if the parent is stored in the property parent then > > > > select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > > > will generate a list of children. (Not sure about performance) > > > > > > Example 3: > > User is allowed to enter the RowID directly (identical to Example 1 > > URL > > > > > /content/cassandra/cannesfilmfestival/TomCruiseCassino-20130402112345-ieb.jpg > > > > where > > String getCQL(String cf, String path) { > > return "select * from "+cf+" where rowid = '"+path+"'"; > > } > > > > yields: > > select * from pictures where rowid = ' > > TomCruiseCassino-20130402112345-ieb.jpg' > > > > This should be corrected as > select * from cannesfilmfestival where rowid = ' > TomCruiseCassino-20130402112345-ieb.jpg' > > > > > > > > Does that make sense ? > > > Hi > Hi Ian, > I was in fact practicing some cql stuff in related to this response (with > cassandra cql terminal). This is quite a wonderful explanation for a new > comer like me. Thank you very much for the explanation again. Now it really > makes sense. > excellent! > > Other than the zero depth approach, I believe users will be more > comfortable with Example 2 approach. > Shall we go ahead with it ? > Yes, go for it. It will be interesting to see how hard it is to implement and how well (or not) it works. Remember, keep it as simple as possible and dont try and and cover every use case at the expense of getting a PoC working. However, dont forget, Unit tests mocked with Mockito are a quicker way of getting to working code, than no unit test coverage. Best Regards Ian > > > > Ian > > > > > > > > > > On 25 June 2013 05:29, Dishara Wijewardana > > wrote: > > > > > On Mon, Jun 24, 2013 at 4:02 AM, Ian Boston wrote: > > > > > > > Hi Dishara, > > > > Yes. 1 resource == 1 row. > > > > The columns within that row represent the properties of the resource. > > > > I suggest that you use standard property names where appropriate (eg > > > > sling:resourceType is the Resource.resourceType etc) > > > > > > > > The Resource itself should be adaptable to a generic > CassandraResource > > > > (which will probably implement Resource) which will have a map of > > > > properties containing all the columns of the cassandra row. (optimise > > > > later) A CassandraResource might look and feel like a Map > Object> > > > > or it might have a Map getProperties() method, or > > better > > > > still be adaptable to a Map. The essential think is dont hard code > the > > > > property names in the interface of CassandraResource for the moment. > ie > > > no > > > > getContentType() and no getMimeType(), as we dont really know what a > > > > CassandraResource will store. > > > > > > > > ResourceMetadata should be built from a subset of the > CassandraResource > > > > properties. > > > > > > > > You won't need to implement a ResourceResolver, only a > ResourceProvider > > > > (and Factory). I would use CQL in preference to other API methods. > > > > > > > > There is one thing that hasnt been mentioned, and thats the URL -> > > > > Cassandra Row mapping. > > > > There are several ways of doing this. > > > > > > > > eg: > > > > URL = /content/cassandra// > > > > Cassandra
Re: [Status Update] Apache Cassandra backend for Sling
On Thu, Jun 27, 2013 at 4:26 AM, Ian Boston wrote: > On 27 June 2013 02:34, Dishara Wijewardana > wrote: > > > On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: > > > > > Hi, > > > > > > (I might have errors in the CQL, Cassandra schema and the functions > need > > > proper escaping) > > > > > > > > > Example 1: > > > Zero depth tree wiht UUID as the rowid or key. > > > > > > URL /content/cassandra/pictures/13f58d5c95c70b6f > > > > > > then the column family is pictures and the URL -> ROWID function just > > > results in the ROWID being 13f58d5c95c70b6f and > > > > > > String cql = mapOfCassandraMappers.get("pictures").getCQL("pictures", " > > > 13f58d5c95c70b6f") > > > System.err.println(cql); > > > > > > where > > > String getCQL(String cf, String path) { > > > return "select * from "+cf+" where rowid = '"+path+"'"; > > > } > > > > > > yields: > > > select * from pictures where rowid = '13f58d5c95c70b6f' > > > > > > > > > 13f58d5c95c70b6f would be generated by the application when the user > > > created a new picture (by upload). > > > > > > > > > > > > Example 2: > > > User specified > > > > > > URL /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf > > > > > > String cql = > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > > > capacitors/electrolytic/axial/16v/10uf") > > > System.err.println(cql); > > > > > > where > > > String getCQL(String cf, String path) { > > > MessageDigest md = MessageDigest.getInstance("SHA1"); > > > String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); > > > return "select * from "+cf+" where rowid = '"+rowID+"'"; > > > } > > > > > > yields > > > > > > select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' > > > > > > If you want to find the parent then > > > > > > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > > > capacitors/electrolytic/axial/16v") > > > > > > select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > > > > > And if the parent is stored in the property parent then > > > > > > select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' > > > > > > will generate a list of children. (Not sure about performance) > > > > > > > > > Example 3: > > > User is allowed to enter the RowID directly (identical to Example 1 > > > URL > > > > > > > > > /content/cassandra/cannesfilmfestival/TomCruiseCassino-20130402112345-ieb.jpg > > > > > > where > > > String getCQL(String cf, String path) { > > > return "select * from "+cf+" where rowid = '"+path+"'"; > > > } > > > > > > yields: > > > select * from pictures where rowid = ' > > > TomCruiseCassino-20130402112345-ieb.jpg' > > > > > > > This should be corrected as > > select * from cannesfilmfestival where rowid = ' > > TomCruiseCassino-20130402112345-ieb.jpg' > > > > > > > > > > > > > Does that make sense ? > > > > > > > Hi > > > > Hi Ian, > > I was in fact practicing some cql stuff in related to this response (with > > cassandra cql terminal). This is quite a wonderful explanation for a new > > comer like me. Thank you very much for the explanation again. Now it > really > > makes sense. > > > > excellent! > > > > > > Other than the zero depth approach, I believe users will be more > > comfortable with Example 2 approach. > > Shall we go ahead with it ? > > > > > Yes, go for it. It will be interesting to see how hard it is to implement > and how well (or not) it works. Remember, keep it as simple as possible and > dont try and and cover every use case at the expense of getting a PoC > working. > +1. > > However, dont forget, Unit tests mocked with Mockito are a quicker way of > getting to working code, than no unit test coverage. > > Best Regards > Ian > > > > > > > > > > > Ian > > > > > > > > > > > > > > > On 25 June 2013 05:29, Dishara Wijewardana > > > wrote: > > > > > > > On Mon, Jun 24, 2013 at 4:02 AM, Ian Boston wrote: > > > > > > > > > Hi Dishara, > > > > > Yes. 1 resource == 1 row. > > > > > The columns within that row represent the properties of the > resource. > > > > > I suggest that you use standard property names where appropriate > (eg > > > > > sling:resourceType is the Resource.resourceType etc) > > > > > > > > > > The Resource itself should be adaptable to a generic > > CassandraResource > > > > > (which will probably implement Resource) which will have a map of > > > > > properties containing all the columns of the cassandra row. > (optimise > > > > > later) A CassandraResource might look and feel like a Map > > Object> > > > > > or it might have a Map getProperties() method, or > > > better > > > > > still be adaptable to a Map. The essential think is dont hard code > > the > > > > > property names in the interface of CassandraResource for the > moment. > > ie > > > > no > > > > > getContentType() and no getMimeType(), as we dont really know what > a > > > > > CassandraResource will store. > > > > > > > > > > ResourceMetadata should be built from a subset of the > > CassandraResource > > > > > propert
Re: [Status Update] Apache Cassandra backend for Sling
Hi Ian, I am having a problem with CQL.. For example: CqlQuery** cqlQuery = new CqlQuery* *(keyspace, new StringSerializer(),new StringSerializer(), new LongSerializer(); cqlQuery.setQuery("insert into mytable (KEY,password,gender,userid) values (3,'pass1','male',34);"); QueryResult> result = cqlQuery.execute(); This will successfully insert the row with pass1,male and 34 values under rowId=3. But in sling scenario, we need to have more serializers for a query as follows. Since we have more columns. i.e CqlQuery* *cqlQuery = new CqlQuery* *(keyspace, new StringSerializer(),new StringSerializer(),new StringSerializer(),new StringSerializer()); cqlQuery.setQuery("insert into mytable (KEY,path,resourceType,resourceSuperType,metadata) values (3,'/content/cassandra/foo/bar','nt:cassandra','nt:super','metadata'); QueryResult> result = cqlQuery.execute(); Here I am using me.prettyprint.cassandra.model.CqlQuery class. Any idea how to proceed with this. Am I doing something wring or is this a limitation of the API I am using ? On Thu, Jun 27, 2013 at 7:41 AM, Dishara Wijewardana < ddwijeward...@gmail.com> wrote: > > > On Thu, Jun 27, 2013 at 4:26 AM, Ian Boston wrote: > >> On 27 June 2013 02:34, Dishara Wijewardana >> wrote: >> >> > On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: >> > >> > > Hi, >> > > >> > > (I might have errors in the CQL, Cassandra schema and the functions >> need >> > > proper escaping) >> > > >> > > >> > > Example 1: >> > > Zero depth tree wiht UUID as the rowid or key. >> > > >> > > URL /content/cassandra/pictures/13f58d5c95c70b6f >> > > >> > > then the column family is pictures and the URL -> ROWID function just >> > > results in the ROWID being 13f58d5c95c70b6f and >> > > >> > > String cql = mapOfCassandraMappers.get("pictures").getCQL("pictures", >> " >> > > 13f58d5c95c70b6f") >> > > System.err.println(cql); >> > > >> > > where >> > > String getCQL(String cf, String path) { >> > > return "select * from "+cf+" where rowid = '"+path+"'"; >> > > } >> > > >> > > yields: >> > > select * from pictures where rowid = '13f58d5c95c70b6f' >> > > >> > > >> > > 13f58d5c95c70b6f would be generated by the application when the user >> > > created a new picture (by upload). >> > > >> > > >> > > >> > > Example 2: >> > > User specified >> > > >> > > URL >> /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf >> > > >> > > String cql = >> mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " >> > > capacitors/electrolytic/axial/16v/10uf") >> > > System.err.println(cql); >> > > >> > > where >> > > String getCQL(String cf, String path) { >> > > MessageDigest md = MessageDigest.getInstance("SHA1"); >> > > String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); >> > > return "select * from "+cf+" where rowid = '"+rowID+"'"; >> > > } >> > > >> > > yields >> > > >> > > select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' >> > > >> > > If you want to find the parent then >> > > >> > > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " >> > > capacitors/electrolytic/axial/16v") >> > > >> > > select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' >> > > >> > > And if the parent is stored in the property parent then >> > > >> > > select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' >> > > >> > > will generate a list of children. (Not sure about performance) >> > > >> > > >> > > Example 3: >> > > User is allowed to enter the RowID directly (identical to Example 1 >> > > URL >> > > >> > > >> > >> /content/cassandra/cannesfilmfestival/TomCruiseCassino-20130402112345-ieb.jpg >> > > >> > > where >> > > String getCQL(String cf, String path) { >> > > return "select * from "+cf+" where rowid = '"+path+"'"; >> > > } >> > > >> > > yields: >> > > select * from pictures where rowid = ' >> > > TomCruiseCassino-20130402112345-ieb.jpg' >> > > >> > >> > This should be corrected as >> > select * from cannesfilmfestival where rowid = ' >> > TomCruiseCassino-20130402112345-ieb.jpg' >> > >> > >> > > >> > > >> > > Does that make sense ? >> > > >> > >> >> Hi >> >> >> > Hi Ian, >> > I was in fact practicing some cql stuff in related to this response >> (with >> > cassandra cql terminal). This is quite a wonderful explanation for a new >> > comer like me. Thank you very much for the explanation again. Now it >> really >> > makes sense. >> > >> >> excellent! >> >> >> > >> > Other than the zero depth approach, I believe users will be more >> > comfortable with Example 2 approach. >> > Shall we go ahead with it ? >> > >> >> >> Yes, go for it. It will be interesting to see how hard it is to implement >> and how well (or not) it works. Remember, keep it as simple as possible >> and >> dont try and and cover every use case at the expense of getting a PoC >> working. >> > +1. > >> >> However, dont forget, Unit tests mocked with Mockito are a quicker way of >> getting to working code, than no un
Re: [Status Update] Apache Cassandra backend for Sling
Hi, Have you tried the TypeInferringSerializer for the value serializer ? That claims to be detect what the column value is based on the Byte array. Failing that, I would consider making everything byte[] and using your own serializer that writes and read values to a byte[] using DataInputStream DataOutputStream. [2] Is an example of a serializer written for that purpose that was used with Cassandra over raw Thrift. Its not easy to read what it outputs to the storage layer, but it is compact and efficient. I would not use it directly as it does some very specific things like slicing large byte[]s into 1MB chunks and bypassing the 64K limit on reading and writing UTF8 strings with DataInputStream. Try the TypeInferringSerializer first. If it works great, no need to do anything more complex. Ian 1 http://hector-client.github.io/hector/source/content/API/core/0.8.0-2/me/prettyprint/cassandra/serializers/TypeInferringSerializer.html 2 https://github.com/ieb/sparsemapcontent/tree/master/core/src/main/java/org/sakaiproject/nakamura/lite/storage/spi/types On 28 June 2013 05:14, Dishara Wijewardana wrote: > Hi Ian, > I am having a problem with CQL.. > > For example: > CqlQuery** cqlQuery = new CqlQuery* > *(keyspace, new StringSerializer(),new > StringSerializer(), new LongSerializer(); > cqlQuery.setQuery("insert into mytable (KEY,password,gender,userid) > values (3,'pass1','male',34);"); > QueryResult> result = > cqlQuery.execute(); > > This will successfully insert the row with pass1,male and 34 values under > rowId=3. > > But in sling scenario, we need to have more serializers for a query as > follows. Since we have more columns. > i.e > CqlQuery* *cqlQuery = new CqlQuery* > *(keyspace, new StringSerializer(),new > StringSerializer(),new StringSerializer(),new StringSerializer()); > cqlQuery.setQuery("insert into mytable > (KEY,path,resourceType,resourceSuperType,metadata) values > (3,'/content/cassandra/foo/bar','nt:cassandra','nt:super','metadata'); > QueryResult> result = > cqlQuery.execute(); > > Here I am using me.prettyprint.cassandra.model.CqlQuery class. Any idea how > to proceed with this. > > Am I doing something wring or is this a limitation of the API I am using ? > > > On Thu, Jun 27, 2013 at 7:41 AM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > > > > > > > On Thu, Jun 27, 2013 at 4:26 AM, Ian Boston wrote: > > > >> On 27 June 2013 02:34, Dishara Wijewardana > >> wrote: > >> > >> > On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: > >> > > >> > > Hi, > >> > > > >> > > (I might have errors in the CQL, Cassandra schema and the functions > >> need > >> > > proper escaping) > >> > > > >> > > > >> > > Example 1: > >> > > Zero depth tree wiht UUID as the rowid or key. > >> > > > >> > > URL /content/cassandra/pictures/13f58d5c95c70b6f > >> > > > >> > > then the column family is pictures and the URL -> ROWID function > just > >> > > results in the ROWID being 13f58d5c95c70b6f and > >> > > > >> > > String cql = > mapOfCassandraMappers.get("pictures").getCQL("pictures", > >> " > >> > > 13f58d5c95c70b6f") > >> > > System.err.println(cql); > >> > > > >> > > where > >> > > String getCQL(String cf, String path) { > >> > > return "select * from "+cf+" where rowid = '"+path+"'"; > >> > > } > >> > > > >> > > yields: > >> > > select * from pictures where rowid = '13f58d5c95c70b6f' > >> > > > >> > > > >> > > 13f58d5c95c70b6f would be generated by the application when the user > >> > > created a new picture (by upload). > >> > > > >> > > > >> > > > >> > > Example 2: > >> > > User specified > >> > > > >> > > URL > >> /content/cassandra/catalogue/capacitors/electrolytic/axial/16v/10uf > >> > > > >> > > String cql = > >> mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > >> > > capacitors/electrolytic/axial/16v/10uf") > >> > > System.err.println(cql); > >> > > > >> > > where > >> > > String getCQL(String cf, String path) { > >> > > MessageDigest md = MessageDigest.getInstance("SHA1"); > >> > > String rowID = Base64.encode(md.finish(path.getBytes("UTF-8"))); > >> > > return "select * from "+cf+" where rowid = '"+rowID+"'"; > >> > > } > >> > > > >> > > yields > >> > > > >> > > select * from pictures where rowid = 'NzdlZmU4OTZmNGM4MzMwYzZ' > >> > > > >> > > If you want to find the parent then > >> > > > >> > > mapOfCassandraMappers.get("catalogue").getCQL("catalogue", " > >> > > capacitors/electrolytic/axial/16v") > >> > > > >> > > select * from pictures where rowid = 'ZGFzZGZzZnNkYWZzYWRmc2R' > >> > > > >> > > And if the parent is stored in the property parent then > >> > > > >> > > select * from pictures where parent = 'ZGFzZGZzZnNkYWZzYWRmc2R' > >> > > > >> > > will generate a list of children. (Not sure about performance) > >> > > > >> > > > >> > > Example 3: > >> > > User is allowed to enter the RowID directly (identical to Example 1 > >> > > URL > >> > > > >> > > > >> > > >> > /content/cassandra/cannesfilmfes
Re: [Status Update] Apache Cassandra backend for Sling
On Fri, Jun 28, 2013 at 4:37 AM, Ian Boston wrote: > Hi, > Have you tried the TypeInferringSerializer for the value serializer ? > That claims to be detect what the column value is based on the Byte array. > > Failing that, I would consider making everything byte[] and using your own > serializer that writes and read values to a byte[] using DataInputStream > DataOutputStream. > > [2] Is an example of a serializer written for that purpose that was used > with Cassandra over raw Thrift. Its not easy to read what it outputs to the > storage layer, but it is compact and efficient. I would not use it directly > as it does some very specific things like slicing large byte[]s into 1MB > chunks and bypassing the 64K limit on reading and writing UTF8 strings with > DataInputStream. > > Try the TypeInferringSerializer first. If it works great, no need to do > anything more complex. > Hi, In fact I was able to add as many params as I wanted with the same configurations. But TypeInferringSerializer is a useful one too which might need in future. Also I was thinking rather than storing resource meta data as String values, how about storing a serialized object as you mentioned ? It will be clear. But I am not sure about the performance. Because when we have multi valued columns like meta data we have to insert them in a single String as comma separated values. It is scalable if we have a Bean for Cassandra Resource ? What do you think ? And I did a first cut of this but with many TODOs ;-), where getResource method is implemented and currently all the content is printed, but I have not implemented methods in CassandraResource yet. This is just a POC to test whether the proposed model works. Apparently it works [1]. See CassandraDataPopulator class which is a plain java test class added for the moment to test the POC.(I am moving this to a proper JUnit) TODOs - I am in the process of finishing the implementation of Cassandra Resource, CassandaResource Provider and etc END to END. - Move to JUnit test framework and write more tests for each scenario where I can extend this to Mockito (I am still not clear how Mockito comes in to the picture) in near future. - Change the implementation based on the feedbacks from the community. - Parameterize the constants as much as possible to read from a property file. [1] - https://cassandra-backend-for-sling.googlecode.com/svn/trunk/main/cassandra Thanks > > > Ian > > 1 > > http://hector-client.github.io/hector/source/content/API/core/0.8.0-2/me/prettyprint/cassandra/serializers/TypeInferringSerializer.html > > 2 > > https://github.com/ieb/sparsemapcontent/tree/master/core/src/main/java/org/sakaiproject/nakamura/lite/storage/spi/types > > > On 28 June 2013 05:14, Dishara Wijewardana > wrote: > > > Hi Ian, > > I am having a problem with CQL.. > > > > For example: > > CqlQuery** cqlQuery = new CqlQuery* > > *(keyspace, new StringSerializer(),new > > StringSerializer(), new LongSerializer(); > > cqlQuery.setQuery("insert into mytable > (KEY,password,gender,userid) > > values (3,'pass1','male',34);"); > > QueryResult> result = > > cqlQuery.execute(); > > > > This will successfully insert the row with pass1,male and 34 values under > > rowId=3. > > > > But in sling scenario, we need to have more serializers for a query as > > follows. Since we have more columns. > > i.e > > CqlQuery* *cqlQuery = new CqlQuery* > > *(keyspace, new StringSerializer(),new > > StringSerializer(),new StringSerializer(),new StringSerializer()); > > cqlQuery.setQuery("insert into mytable > > (KEY,path,resourceType,resourceSuperType,metadata) values > > (3,'/content/cassandra/foo/bar','nt:cassandra','nt:super','metadata'); > > QueryResult> result = > > cqlQuery.execute(); > > > > Here I am using me.prettyprint.cassandra.model.CqlQuery class. Any idea > how > > to proceed with this. > > > > Am I doing something wring or is this a limitation of the API I am using > ? > > > > > > On Thu, Jun 27, 2013 at 7:41 AM, Dishara Wijewardana < > > ddwijeward...@gmail.com> wrote: > > > > > > > > > > > On Thu, Jun 27, 2013 at 4:26 AM, Ian Boston wrote: > > > > > >> On 27 June 2013 02:34, Dishara Wijewardana > > >> wrote: > > >> > > >> > On Tue, Jun 25, 2013 at 4:52 AM, Ian Boston wrote: > > >> > > > >> > > Hi, > > >> > > > > >> > > (I might have errors in the CQL, Cassandra schema and the > functions > > >> need > > >> > > proper escaping) > > >> > > > > >> > > > > >> > > Example 1: > > >> > > Zero depth tree wiht UUID as the rowid or key. > > >> > > > > >> > > URL /content/cassandra/pictures/13f58d5c95c70b6f > > >> > > > > >> > > then the column family is pictures and the URL -> ROWID function > > just > > >> > > results in the ROWID being 13f58d5c95c70b6f and > > >> > > > > >> > > String cql = > > mapOfCassandraMappers.get("pictures").getCQL("pictures", > > >> " > > >> > > 13f58d5c95c70b6f") > > >> > > System.err.println(cql); > > >> > > > > >> > > where
Re: [Status Update] Apache Cassandra backend for Sling
Hi Dishara, I've taken the liberty of creating a code review at [1]. This is all commits. I've emailed you separately with the comments. I think it would be good if we can get into the habit of looking at the code in this way as it often removes confusion introduced by the english language (which has many compilers ;), mine has been known to be buggy at times.). More comments inline below: (BTW, excellent progress!) Best Regards Ian 1 https://codereview.appspot.com/10811044/ On 30 June 2013 22:52, Dishara Wijewardana wrote: > On Fri, Jun 28, 2013 at 4:37 AM, Ian Boston wrote: > > > Hi, > > Have you tried the TypeInferringSerializer for the value serializer ? > > That claims to be detect what the column value is based on the Byte > array. > > > > Failing that, I would consider making everything byte[] and using your > own > > serializer that writes and read values to a byte[] using DataInputStream > > DataOutputStream. > > > > [2] Is an example of a serializer written for that purpose that was used > > with Cassandra over raw Thrift. Its not easy to read what it outputs to > the > > storage layer, but it is compact and efficient. I would not use it > directly > > as it does some very specific things like slicing large byte[]s into 1MB > > chunks and bypassing the 64K limit on reading and writing UTF8 strings > with > > DataInputStream. > > > > Try the TypeInferringSerializer first. If it works great, no need to do > > anything more complex. > > > > Hi, > In fact I was able to add as many params as I wanted with the same > configurations. But TypeInferringSerializer is a useful one too which might > need in future. > Also I was thinking rather than storing resource meta data as String > values, how about storing a serialized object as you mentioned ? I suspect that TypeInferringSerializer will do a better job of serializing than the approach I mentioned. Only consider writing your own, if there is a real and demonstrated need for it. > It will be > clear. But I am not sure about the performance. Because when we have multi > valued columns like meta data we have to insert them in a single String as > comma separated values. It is scalable if we have a Bean for Cassandra > Resource ? What do you think ? > Put one property per column in Cassandra if possible. IIRC it does a good job of serializing data, and doesnt need a pre-defined schema as traditional RDBMS's do. The serialisation I mentioned was mostly used to get schemaless storage into an RDBMS. > > And I did a first cut of this but with many TODOs ;-), where getResource > method is implemented and currently all the content is printed, but I have > not implemented methods in CassandraResource yet. This is just a POC to > test whether the proposed model works. Apparently it works [1]. Yes, this is a great start! I didn't find to many issues with the approach, as you will see from the comments on the code review. > See > CassandraDataPopulator class which is a plain java test class added for > the moment to test the POC.(I am moving this to a proper JUnit) > Good. > > TODOs > - I am in the process of finishing the implementation of Cassandra > Resource, CassandaResource Provider and etc END to END. > - Move to JUnit test framework and write more tests for each scenario > where I can extend this to Mockito (I am still not clear how Mockito comes > in to the picture) in near future. > When you write the Unit tests, if you find that you need to mock anything (ie ResourceResolver) to make your unit tests work, dont. Use Mocks. You can even Mockup concrete clases so could mockup the behaviour of the Hector API to respond in a pre-defined way to certain CQL queries. This will eliminate the need to have a real cassandra server present when doing the basic unit tests. > - Change the implementation based on the feedbacks from the community. > - Parameterize the constants as much as possible to read from a property > file. > These should come from OSGi Properties. See the comments on CassandraResoureProvider > > > [1] - > https://cassandra-backend-for-sling.googlecode.com/svn/trunk/main/cassandra > > Thanks > Excellent progress, thank you! Ian > > > > > > > Ian > > > > 1 > > > > > http://hector-client.github.io/hector/source/content/API/core/0.8.0-2/me/prettyprint/cassandra/serializers/TypeInferringSerializer.html > > > > 2 > > > > > https://github.com/ieb/sparsemapcontent/tree/master/core/src/main/java/org/sakaiproject/nakamura/lite/storage/spi/types > > > > > > On 28 June 2013 05:14, Dishara Wijewardana > > wrote: > > > > > Hi Ian, > > > I am having a problem with CQL.. > > > > > > For example: > > > CqlQuery** cqlQuery = new CqlQuery* > > > *(keyspace, new StringSerializer(),new > > > StringSerializer(), new LongSerializer(); > > > cqlQuery.setQuery("insert into mytable > > (KEY,password,gender,userid) > > > values (3,'pass1','male',34);"); > > > QueryResult> result = > > > cqlQuery.execute
Re: [Status Update] Apache Cassandra backend for Sling
Hi Ian, Does each bundle in sling made to run their junit tests separately at build time ? If so each pom should have configured to junit test cases. Where and how to define them ? On Mon, Jul 1, 2013 at 5:32 AM, Ian Boston wrote: > Hi Dishara, > > I've taken the liberty of creating a code review at [1]. This is all > commits. I've emailed you separately with the comments. I think it would be > good if we can get into the habit of looking at the code in this way as it > often removes confusion introduced by the english language (which has many > compilers ;), mine has been known to be buggy at times.). > > > More comments inline below: (BTW, excellent progress!) > > Best Regards > Ian > > > 1 https://codereview.appspot.com/10811044/ > > > > On 30 June 2013 22:52, Dishara Wijewardana > wrote: > > > On Fri, Jun 28, 2013 at 4:37 AM, Ian Boston wrote: > > > > > Hi, > > > Have you tried the TypeInferringSerializer for the value serializer ? > > > That claims to be detect what the column value is based on the Byte > > array. > > > > > > Failing that, I would consider making everything byte[] and using your > > own > > > serializer that writes and read values to a byte[] using > DataInputStream > > > DataOutputStream. > > > > > > [2] Is an example of a serializer written for that purpose that was > used > > > with Cassandra over raw Thrift. Its not easy to read what it outputs to > > the > > > storage layer, but it is compact and efficient. I would not use it > > directly > > > as it does some very specific things like slicing large byte[]s into > 1MB > > > chunks and bypassing the 64K limit on reading and writing UTF8 strings > > with > > > DataInputStream. > > > > > > Try the TypeInferringSerializer first. If it works great, no need to do > > > anything more complex. > > > > > > > Hi, > > In fact I was able to add as many params as I wanted with the same > > configurations. But TypeInferringSerializer is a useful one too which > might > > need in future. > > Also I was thinking rather than storing resource meta data as String > > values, how about storing a serialized object as you mentioned ? > > > I suspect that TypeInferringSerializer will do a better job of serializing > than the approach I mentioned. Only consider writing your own, if there is > a real and demonstrated need for it. > > > > It will be > > clear. But I am not sure about the performance. Because when we have > multi > > valued columns like meta data we have to insert them in a single String > as > > comma separated values. It is scalable if we have a Bean for Cassandra > > Resource ? What do you think ? > > > > Put one property per column in Cassandra if possible. IIRC it does a good > job of serializing data, and doesnt need a pre-defined schema as > traditional RDBMS's do. The serialisation I mentioned was mostly used to > get schemaless storage into an RDBMS. > > > > > > > And I did a first cut of this but with many TODOs ;-), where > getResource > > method is implemented and currently all the content is printed, but I > have > > not implemented methods in CassandraResource yet. This is just a POC to > > test whether the proposed model works. Apparently it works [1]. > > > Yes, this is a great start! I didn't find to many issues with the approach, > as you will see from the comments on the code review. > > > > > > See > > CassandraDataPopulator class which is a plain java test class added for > > the moment to test the POC.(I am moving this to a proper JUnit) > > > > Good. > > > > > > TODOs > > - I am in the process of finishing the implementation of Cassandra > > Resource, CassandaResource Provider and etc END to END. > > - Move to JUnit test framework and write more tests for each scenario > > where I can extend this to Mockito (I am still not clear how Mockito > comes > > in to the picture) in near future. > > > > When you write the Unit tests, if you find that you need to mock anything > (ie ResourceResolver) to make your unit tests work, dont. Use Mocks. You > can even Mockup concrete clases so could mockup the behaviour of the Hector > API to respond in a pre-defined way to certain CQL queries. This will > eliminate the need to have a real cassandra server present when doing the > basic unit tests. > > > > > > - Change the implementation based on the feedbacks from the community. > > - Parameterize the constants as much as possible to read from a property > > file. > > > > These should come from OSGi Properties. See the comments on > CassandraResoureProvider > > > > > > > > > > > > [1] - > > > https://cassandra-backend-for-sling.googlecode.com/svn/trunk/main/cassandra > > > > Thanks > > > > Excellent progress, thank you! > Ian > > > > > > > > > > > > > Ian > > > > > > 1 > > > > > > > > > http://hector-client.github.io/hector/source/content/API/core/0.8.0-2/me/prettyprint/cassandra/serializers/TypeInferringSerializer.html > > > > > > 2 > > > > > > > > > https://github.com/ieb/sparsemapcontent/tree/master/core/src/main/jav
Re: [Status Update] Apache Cassandra backend for Sling
Hi DIshara, On 3 July 2013 02:51, Dishara Wijewardana wrote: > Hi Ian, > Does each bundle in sling made to run their junit tests separately at build > time ? Yes. > If so each pom should have configured to junit test cases. > Where and how to define them ? > All you need to do is depend on he Sling parent, as you have done, include JUnit4, Slf4j simple, (and Mockito) as a dependency and put your unit tests in src/test/java. IIRC Maven's default build configuration runs unit tests. Have a look at [1] it has junit junit org.slf4j slf4j-simple Best Regards Ian 1 http://svn.apache.org/repos/asf/sling/trunk/bundles/commons/json/ > > > On Mon, Jul 1, 2013 at 5:32 AM, Ian Boston wrote: > > > Hi Dishara, > > > > I've taken the liberty of creating a code review at [1]. This is all > > commits. I've emailed you separately with the comments. I think it would > be > > good if we can get into the habit of looking at the code in this way as > it > > often removes confusion introduced by the english language (which has > many > > compilers ;), mine has been known to be buggy at times.). > > > > > > More comments inline below: (BTW, excellent progress!) > > > > Best Regards > > Ian > > > > > > 1 https://codereview.appspot.com/10811044/ > > > > > > > > On 30 June 2013 22:52, Dishara Wijewardana > > wrote: > > > > > On Fri, Jun 28, 2013 at 4:37 AM, Ian Boston wrote: > > > > > > > Hi, > > > > Have you tried the TypeInferringSerializer for the value serializer ? > > > > That claims to be detect what the column value is based on the Byte > > > array. > > > > > > > > Failing that, I would consider making everything byte[] and using > your > > > own > > > > serializer that writes and read values to a byte[] using > > DataInputStream > > > > DataOutputStream. > > > > > > > > [2] Is an example of a serializer written for that purpose that was > > used > > > > with Cassandra over raw Thrift. Its not easy to read what it outputs > to > > > the > > > > storage layer, but it is compact and efficient. I would not use it > > > directly > > > > as it does some very specific things like slicing large byte[]s into > > 1MB > > > > chunks and bypassing the 64K limit on reading and writing UTF8 > strings > > > with > > > > DataInputStream. > > > > > > > > Try the TypeInferringSerializer first. If it works great, no need to > do > > > > anything more complex. > > > > > > > > > > Hi, > > > In fact I was able to add as many params as I wanted with the same > > > configurations. But TypeInferringSerializer is a useful one too which > > might > > > need in future. > > > Also I was thinking rather than storing resource meta data as String > > > values, how about storing a serialized object as you mentioned ? > > > > > > I suspect that TypeInferringSerializer will do a better job of > serializing > > than the approach I mentioned. Only consider writing your own, if there > is > > a real and demonstrated need for it. > > > > > > > It will be > > > clear. But I am not sure about the performance. Because when we have > > multi > > > valued columns like meta data we have to insert them in a single String > > as > > > comma separated values. It is scalable if we have a Bean for Cassandra > > > Resource ? What do you think ? > > > > > > > Put one property per column in Cassandra if possible. IIRC it does a good > > job of serializing data, and doesnt need a pre-defined schema as > > traditional RDBMS's do. The serialisation I mentioned was mostly used to > > get schemaless storage into an RDBMS. > > > > > > > > > > > > And I did a first cut of this but with many TODOs ;-), where > > getResource > > > method is implemented and currently all the content is printed, but I > > have > > > not implemented methods in CassandraResource yet. This is just a POC to > > > test whether the proposed model works. Apparently it works [1]. > > > > > > Yes, this is a great start! I didn't find to many issues with the > approach, > > as you will see from the comments on the code review. > > > > > > > > > > > See > > > CassandraDataPopulator class which is a plain java test class added > for > > > the moment to test the POC.(I am moving this to a proper JUnit) > > > > > > > Good. > > > > > > > > > > TODOs > > > - I am in the process of finishing the implementation of Cassandra > > > Resource, CassandaResource Provider and etc END to END. > > > - Move to JUnit test framework and write more tests for each scenario > > > where I can extend this to Mockito (I am still not clear how Mockito > > comes > > > in to the picture) in near future. > > > > > > > When you write the Unit tests, if you find that you need to mock anything > > (ie ResourceResolver) to make your unit tests work, dont. Use Mocks. You > > can even Mockup concrete clases so could mockup the behaviour of the > Hector > > API to respond in a pre-defined way to certain CQL queries. This will > > eliminate the ne
Re: [Status Update] Apache Cassandra backend for Sling
Hi Dishara, On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana wrote: > ...Does each bundle in sling made to run their junit tests separately at build > time... Could you start new threads with new subject lines when you start a new question or discussion? Otherwise it's very hard to find things in our mailing list archives. Thanks, and keep up the good work! -Bertrand
Re: [Status Update] Apache Cassandra backend for Sling
Hi Ian, I have refactored almost all the code review changes requested. In the process of the rest of the implementation. On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz wrote: > Hi Dishara, > > On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana > wrote: > > ...Does each bundle in sling made to run their junit tests separately at > build > > time... > > Could you start new threads with new subject lines when you start a > new question or discussion? > > Otherwise it's very hard to find things in our mailing list archives. > > OK. That's a good idea. I will follow that. > Thanks, and keep up the good work! > -Bertrand > -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
Hi Ian This is regarding the sub tasks completion of the project according to the time line. I have my code locally, but yet to do some completion with some stuff(one of them is the ongoing discussion on listChildren). After that from API point of view implementation around Cassandra Resource Provider and Resource will be finish. And I have to add some more JUnit tests(have local code in to some extent already, will commit them once all done around this). So after that (after mid term) what I have to do is enhance the provider implementation to do READ operations with access control. My idea is to finish that also before the mid term. I was kind of got stuck in some OSGi stuff last days ;-). And I will make sure I will have JUnit tests to cover all the implementations before the midterm. If you would like this approach (or unless please provide your feedback on what needs to be done before midterm), please advice me on how to approach on READ with access control. Can I do it and test it with keeping my code in Google Code still ? And please add if I have missed anything. On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < ddwijeward...@gmail.com> wrote: > Hi Ian, > I have refactored almost all the code review changes requested. In the > process of the rest of the implementation. > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < > bdelacre...@apache.org> wrote: > >> Hi Dishara, >> >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana >> wrote: >> > ...Does each bundle in sling made to run their junit tests separately >> at build >> > time... >> >> Could you start new threads with new subject lines when you start a >> new question or discussion? >> >> Otherwise it's very hard to find things in our mailing list archives. >> >> OK. That's a good idea. I will follow that. > > >> Thanks, and keep up the good work! >> -Bertrand >> > > > > -- > Thanks > /Dishara > -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
Hi, On 19 July 2013 03:20, Dishara Wijewardana wrote: > Hi Ian > This is regarding the sub tasks completion of the project according to the > time line. > > I have my code locally, but yet to do some completion with some stuff(one > of them is the ongoing discussion on listChildren). > After that from API point of view implementation around Cassandra Resource > Provider and Resource will be finish. And I have to add some more JUnit > tests(have local code in to some extent already, will commit them once all > done around this). > > So after that (after mid term) what I have to do is enhance the provider > implementation to do READ operations with access control. My idea is to > finish that also before the mid term. I was kind of got stuck in some OSGi > stuff last days ;-). And I will make sure I will have JUnit tests to cover > all the implementations before the midterm. > Yes that is fine. Before adding ACLs I would like to have the code running inside Sling, inside OSGi connected to a Cassandra instance so that we can do some basic tests over http using Curl. Perhaps you are there already? Let me know when you are ready and I'll give it a go ? > If you would like this approach (or unless please provide your feedback on > what needs to be done before midterm), please advice me on how to approach > on READ with access control. Can I do it and test it with keeping my code > in Google Code still ? And please add if I have missed anything. > > On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > > > Hi Ian, > > I have refactored almost all the code review changes requested. In the > > process of the rest of the implementation. > > > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < > > bdelacre...@apache.org> wrote: > > > >> Hi Dishara, > >> > >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana > >> wrote: > >> > ...Does each bundle in sling made to run their junit tests separately > >> at build > >> > time... > >> > >> Could you start new threads with new subject lines when you start a > >> new question or discussion? > >> > >> Otherwise it's very hard to find things in our mailing list archives. > >> > >> OK. That's a good idea. I will follow that. > > > > > >> Thanks, and keep up the good work! > >> -Bertrand > >> > > > > > > > > -- > > Thanks > > /Dishara > > > > > > -- > Thanks > /Dishara >
Re: [Status Update] Apache Cassandra backend for Sling
On Fri, Jul 19, 2013 at 2:35 PM, Ian Boston wrote: > Hi, > > > On 19 July 2013 03:20, Dishara Wijewardana > wrote: > > > Hi Ian > > This is regarding the sub tasks completion of the project according to > the > > time line. > > > > I have my code locally, but yet to do some completion with some stuff(one > > of them is the ongoing discussion on listChildren). > > After that from API point of view implementation around Cassandra > Resource > > Provider and Resource will be finish. And I have to add some more JUnit > > tests(have local code in to some extent already, will commit them once > all > > done around this). > > > > So after that (after mid term) what I have to do is enhance the > provider > > implementation to do READ operations with access control. My idea is to > > finish that also before the mid term. I was kind of got stuck in some > OSGi > > stuff last days ;-). And I will make sure I will have JUnit tests to > cover > > all the implementations before the midterm. > > > > Yes that is fine. > Before adding ACLs I would like to have the code running inside Sling, > inside OSGi connected to a Cassandra instance so that we can do some basic > tests over http using Curl. > > Perhaps you are there already? Let me know when you are ready and I'll give > it a go ? > Hi Ian, That is great and +1. In fact now the code is there and I have completed, rest of the implementation on normal READ and commited. I have added 3 more tests to cover the core implementation. The tests are running which covers add/read nodes, list children,iterate/iterable children and get parent related stuff. And I have made the changes you mentioned. So now resources will not get loaded unless we do a API call with the resource. > > > > If you would like this approach (or unless please provide your feedback > on > > what needs to be done before midterm), please advice me on how to > approach > > on READ with access control. Can I do it and test it with keeping my code > > in Google Code still ? And please add if I have missed anything. > > > > On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < > > ddwijeward...@gmail.com> wrote: > > > > > Hi Ian, > > > I have refactored almost all the code review changes requested. In the > > > process of the rest of the implementation. > > > > > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < > > > bdelacre...@apache.org> wrote: > > > > > >> Hi Dishara, > > >> > > >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana > > >> wrote: > > >> > ...Does each bundle in sling made to run their junit tests > separately > > >> at build > > >> > time... > > >> > > >> Could you start new threads with new subject lines when you start a > > >> new question or discussion? > > >> > > >> Otherwise it's very hard to find things in our mailing list archives. > > >> > > >> OK. That's a good idea. I will follow that. > > > > > > > > >> Thanks, and keep up the good work! > > >> -Bertrand > > >> > > > > > > > > > > > > -- > > > Thanks > > > /Dishara > > > > > > > > > > > -- > > Thanks > > /Dishara > > > -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
On Sun, Jul 21, 2013 at 5:32 PM, Dishara Wijewardana < ddwijeward...@gmail.com> wrote: > > > On Fri, Jul 19, 2013 at 2:35 PM, Ian Boston wrote: > >> Hi, >> >> >> On 19 July 2013 03:20, Dishara Wijewardana >> wrote: >> >> > Hi Ian >> > This is regarding the sub tasks completion of the project according to >> the >> > time line. >> > >> > I have my code locally, but yet to do some completion with some >> stuff(one >> > of them is the ongoing discussion on listChildren). >> > After that from API point of view implementation around Cassandra >> Resource >> > Provider and Resource will be finish. And I have to add some more JUnit >> > tests(have local code in to some extent already, will commit them once >> all >> > done around this). >> > >> > So after that (after mid term) what I have to do is enhance the >> provider >> > implementation to do READ operations with access control. My idea is >> to >> > finish that also before the mid term. I was kind of got stuck in some >> OSGi >> > stuff last days ;-). And I will make sure I will have JUnit tests to >> cover >> > all the implementations before the midterm. >> > >> >> Yes that is fine. >> Before adding ACLs I would like to have the code running inside Sling, >> inside OSGi connected to a Cassandra instance so that we can do some basic >> tests over http using Curl. >> >> Perhaps you are there already? Let me know when you are ready and I'll >> give >> it a go ? >> > > Hi Ian, > That is great and +1. In fact now the code is there and I have completed, > rest of the implementation on normal READ and commited. > > I have added 3 more tests to cover the core implementation. The tests are > running which covers add/read nodes, list children,iterate/iterable > children and get parent related stuff. > Correction . It should be 4 more tests. Now we have 5 tests all together which covers above aspects. We should write more test around Cassandra Resource. Will work on that as well. Please let me know if any issues come across when you run this in the sling container. Excited to see whether it works in there :-). > And I have made the changes you mentioned. So now resources will not get > loaded unless we do a API call with the resource. > > >> >> >> > If you would like this approach (or unless please provide your feedback >> on >> > what needs to be done before midterm), please advice me on how to >> approach >> > on READ with access control. Can I do it and test it with keeping my >> code >> > in Google Code still ? And please add if I have missed anything. >> > >> > On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < >> > ddwijeward...@gmail.com> wrote: >> > >> > > Hi Ian, >> > > I have refactored almost all the code review changes requested. In the >> > > process of the rest of the implementation. >> > > >> > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < >> > > bdelacre...@apache.org> wrote: >> > > >> > >> Hi Dishara, >> > >> >> > >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana >> > >> wrote: >> > >> > ...Does each bundle in sling made to run their junit tests >> separately >> > >> at build >> > >> > time... >> > >> >> > >> Could you start new threads with new subject lines when you start a >> > >> new question or discussion? >> > >> >> > >> Otherwise it's very hard to find things in our mailing list archives. >> > >> >> > >> OK. That's a good idea. I will follow that. >> > > >> > > >> > >> Thanks, and keep up the good work! >> > >> -Bertrand >> > >> >> > > >> > > >> > > >> > > -- >> > > Thanks >> > > /Dishara >> > > >> > >> > >> > >> > -- >> > Thanks >> > /Dishara >> > >> > > > > -- > Thanks > /Dishara > -- Thanks /Dishara
Re: [Status Update] Apache Cassandra backend for Sling
Hi Dishara, The Unit test coverage sounds great. I will pull the code and review today. Have you tried loading the bundle into a running Sling instance ? Once you have built it you can load in 2 ways: method A mvn clean install sling:install The sling:install will post the Jar into the OSGi container over HTTP and cause it to start. method B Goto http://localhost:8080/system/console select the bundle tab and install the bundle by uploading. If you monitor the logs you should see no errors, the bundle should register and you should be able to map cassandra read only to somewhere in the resource tree. Best Regards Ian On 21 July 2013 13:08, Dishara Wijewardana wrote: > On Sun, Jul 21, 2013 at 5:32 PM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > > > > > > > On Fri, Jul 19, 2013 at 2:35 PM, Ian Boston wrote: > > > >> Hi, > >> > >> > >> On 19 July 2013 03:20, Dishara Wijewardana > >> wrote: > >> > >> > Hi Ian > >> > This is regarding the sub tasks completion of the project according > to > >> the > >> > time line. > >> > > >> > I have my code locally, but yet to do some completion with some > >> stuff(one > >> > of them is the ongoing discussion on listChildren). > >> > After that from API point of view implementation around Cassandra > >> Resource > >> > Provider and Resource will be finish. And I have to add some more > JUnit > >> > tests(have local code in to some extent already, will commit them once > >> all > >> > done around this). > >> > > >> > So after that (after mid term) what I have to do is enhance the > >> provider > >> > implementation to do READ operations with access control. My idea is > >> to > >> > finish that also before the mid term. I was kind of got stuck in some > >> OSGi > >> > stuff last days ;-). And I will make sure I will have JUnit tests to > >> cover > >> > all the implementations before the midterm. > >> > > >> > >> Yes that is fine. > >> Before adding ACLs I would like to have the code running inside Sling, > >> inside OSGi connected to a Cassandra instance so that we can do some > basic > >> tests over http using Curl. > >> > >> Perhaps you are there already? Let me know when you are ready and I'll > >> give > >> it a go ? > >> > > > > Hi Ian, > > That is great and +1. In fact now the code is there and I have completed, > > rest of the implementation on normal READ and commited. > > > > I have added 3 more tests to cover the core implementation. The tests > are > > running which covers add/read nodes, list children,iterate/iterable > > children and get parent related stuff. > > > > Correction . It should be 4 more tests. Now we have 5 tests all together > which covers above aspects. We should write more test around Cassandra > Resource. Will work on that as well. > > Please let me know if any issues come across when you run this in the sling > container. Excited to see whether it works in there :-). > > > > And I have made the changes you mentioned. So now resources will not get > > loaded unless we do a API call with the resource. > > > > > >> > >> > >> > If you would like this approach (or unless please provide your > feedback > >> on > >> > what needs to be done before midterm), please advice me on how to > >> approach > >> > on READ with access control. Can I do it and test it with keeping my > >> code > >> > in Google Code still ? And please add if I have missed anything. > >> > > >> > On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < > >> > ddwijeward...@gmail.com> wrote: > >> > > >> > > Hi Ian, > >> > > I have refactored almost all the code review changes requested. In > the > >> > > process of the rest of the implementation. > >> > > > >> > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < > >> > > bdelacre...@apache.org> wrote: > >> > > > >> > >> Hi Dishara, > >> > >> > >> > >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana > >> > >> wrote: > >> > >> > ...Does each bundle in sling made to run their junit tests > >> separately > >> > >> at build > >> > >> > time... > >> > >> > >> > >> Could you start new threads with new subject lines when you start a > >> > >> new question or discussion? > >> > >> > >> > >> Otherwise it's very hard to find things in our mailing list > archives. > >> > >> > >> > >> OK. That's a good idea. I will follow that. > >> > > > >> > > > >> > >> Thanks, and keep up the good work! > >> > >> -Bertrand > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > Thanks > >> > > /Dishara > >> > > > >> > > >> > > >> > > >> > -- > >> > Thanks > >> > /Dishara > >> > > >> > > > > > > > > -- > > Thanks > > /Dishara > > > > > > -- > Thanks > /Dishara >
Re: [Status Update] Apache Cassandra backend for Sling
On Mon, Jul 22, 2013 at 1:56 PM, Ian Boston wrote: > Hi Dishara, > > The Unit test coverage sounds great. I will pull the code and review today. > > > Have you tried loading the bundle into a running Sling instance ? > Not yet. I will try that out and let you know in one of following ways Thanks. > > Once you have built it you can load in 2 ways: > > method A > > mvn clean install sling:install > > The sling:install will post the Jar into the OSGi container over HTTP and > cause it to start. > > > method B > > Goto http://localhost:8080/system/console > > select the bundle tab and install the bundle by uploading. > > If you monitor the logs you should see no errors, the bundle should > register and you should be able to map cassandra read only to somewhere in > the resource tree. > > Best Regards > Ian > > > On 21 July 2013 13:08, Dishara Wijewardana > wrote: > > > On Sun, Jul 21, 2013 at 5:32 PM, Dishara Wijewardana < > > ddwijeward...@gmail.com> wrote: > > > > > > > > > > > On Fri, Jul 19, 2013 at 2:35 PM, Ian Boston wrote: > > > > > >> Hi, > > >> > > >> > > >> On 19 July 2013 03:20, Dishara Wijewardana > > >> wrote: > > >> > > >> > Hi Ian > > >> > This is regarding the sub tasks completion of the project according > > to > > >> the > > >> > time line. > > >> > > > >> > I have my code locally, but yet to do some completion with some > > >> stuff(one > > >> > of them is the ongoing discussion on listChildren). > > >> > After that from API point of view implementation around Cassandra > > >> Resource > > >> > Provider and Resource will be finish. And I have to add some more > > JUnit > > >> > tests(have local code in to some extent already, will commit them > once > > >> all > > >> > done around this). > > >> > > > >> > So after that (after mid term) what I have to do is enhance the > > >> provider > > >> > implementation to do READ operations with access control. My idea > is > > >> to > > >> > finish that also before the mid term. I was kind of got stuck in > some > > >> OSGi > > >> > stuff last days ;-). And I will make sure I will have JUnit tests > to > > >> cover > > >> > all the implementations before the midterm. > > >> > > > >> > > >> Yes that is fine. > > >> Before adding ACLs I would like to have the code running inside Sling, > > >> inside OSGi connected to a Cassandra instance so that we can do some > > basic > > >> tests over http using Curl. > > >> > > >> Perhaps you are there already? Let me know when you are ready and I'll > > >> give > > >> it a go ? > > >> > > > > > > Hi Ian, > > > That is great and +1. In fact now the code is there and I have > completed, > > > rest of the implementation on normal READ and commited. > > > > > > I have added 3 more tests to cover the core implementation. The tests > > are > > > running which covers add/read nodes, list children,iterate/iterable > > > children and get parent related stuff. > > > > > > > Correction . It should be 4 more tests. Now we have 5 tests all together > > which covers above aspects. We should write more test around Cassandra > > Resource. Will work on that as well. > > > > Please let me know if any issues come across when you run this in the > sling > > container. Excited to see whether it works in there :-). > > > > > > > And I have made the changes you mentioned. So now resources will not > get > > > loaded unless we do a API call with the resource. > > > > > > > > >> > > >> > > >> > If you would like this approach (or unless please provide your > > feedback > > >> on > > >> > what needs to be done before midterm), please advice me on how to > > >> approach > > >> > on READ with access control. Can I do it and test it with keeping my > > >> code > > >> > in Google Code still ? And please add if I have missed anything. > > >> > > > >> > On Thu, Jul 4, 2013 at 11:13 PM, Dishara Wijewardana < > > >> > ddwijeward...@gmail.com> wrote: > > >> > > > >> > > Hi Ian, > > >> > > I have refactored almost all the code review changes requested. In > > the > > >> > > process of the rest of the implementation. > > >> > > > > >> > > On Wed, Jul 3, 2013 at 12:52 PM, Bertrand Delacretaz < > > >> > > bdelacre...@apache.org> wrote: > > >> > > > > >> > >> Hi Dishara, > > >> > >> > > >> > >> On Tue, Jul 2, 2013 at 6:51 PM, Dishara Wijewardana > > >> > >> wrote: > > >> > >> > ...Does each bundle in sling made to run their junit tests > > >> separately > > >> > >> at build > > >> > >> > time... > > >> > >> > > >> > >> Could you start new threads with new subject lines when you > start a > > >> > >> new question or discussion? > > >> > >> > > >> > >> Otherwise it's very hard to find things in our mailing list > > archives. > > >> > >> > > >> > >> OK. That's a good idea. I will follow that. > > >> > > > > >> > > > > >> > >> Thanks, and keep up the good work! > > >> > >> -Bertrand > > >> > >> > > >> > > > > >> > > > > >> > > > > >> > > -- > > >> > > Thanks > > >> > > /Dishara > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Thanks >
Re: [Status Update] Apache Cassandra backend for Sling
On Tue, Jul 23, 2013 at 5:32 PM, Dishara Wijewardana < ddwijeward...@gmail.com> wrote: > > > On Mon, Jul 22, 2013 at 1:56 PM, Ian Boston wrote: > >> Hi Dishara, >> >> The Unit test coverage sounds great. I will pull the code and review >> today. >> >> >> Have you tried loading the bundle into a running Sling instance ? > > Not yet. I will try that out and let you know in one of following ways > Thanks. > >> >> Once you have built it you can load in 2 ways: >> >> method A >> >> mvn clean install sling:install >> >> The sling:install will post the Jar into the OSGi container over HTTP and >> cause it to start. >> > Hi Ian I started Sling launchpad and Cassandra Server as well. And uploaded the bundle through management console from http://localhost:8080/system/console . When I try to start the bundle it does nothing and no error logs in the back end as well. The I noticed following where me.prettyprint.* classes cannot loaded and hence my bundle cannot start. So ideally my bundle only contains it self only. Not included its dependencies. As I feel one option is get a me.prettyprint.* jar and install that bundle (I am not sure whether it also fails due to a similar reason). So what is the best approach to this ? Symbolic Name org.apache.sling.cassandra Version 0.0.1.SNAPSHOT Bundle Location inputstream:org.apache.sling.cassandra-0.0.1-SNAPSHOT.jar Last Modification Mon Aug 05 05:26:16 IST 2013 Description Provides a ResourceProvider implementation supporting Apache Cassandra based resources. Start Level 20 Exported Packages org.apache.sling.cassandra.resource.provider,version=0.0.1.SNAPSHOT org.apache.sling.cassandra.resource.provider.mapper,version=0.0.1.SNAPSHOT org.apache.sling.cassandra.resource.provider.util,version=0.0.1.SNAPSHOT Imported Packages com.sun.org.apache.xerces.internal.impl.dv.util -- Cannot be resolved and overwritten by Boot Delegation javax.servlet.http from org.apache.felix.http.jetty (1) me.prettyprint.cassandra.model,version=[1.0,2) -- Cannot be resolved me.prettyprint.cassandra.serializers,version=[1.0,2) -- Cannot be resolved me.prettyprint.hector.api,version=[1.0,2) -- Cannot be resolved me.prettyprint.hector.api.beans,version=[1.0,2) -- Cannot be resolved me.prettyprint.hector.api.ddl,version=[1.0,2) -- Cannot be resolved me.prettyprint.hector.api.factory,version=[1.0,2) -- Cannot be resolved me.prettyprint.hector.api.query,version=[1.0,2) -- Cannot be resolved org.apache.sling.api.resource,version=[2.3,3) from org.apache.sling.api (98) org.slf4j,version=[1.5,2) from slf4j.api (6) Manifest Headers Bnd-LastModified: 1375238009819 Build-Jdk: 1.6.0_26 Built-By: dishara Bundle-Description: Provides a ResourceProvider implementation supporting Apache Cassandra based resources. Bundle-ManifestVersion: 2 Bundle-Name: Apache Sling Cassandra Resource Provider Bundle-SymbolicName: org.apache.sling.cassandra Bundle-Version: 0.0.1.SNAPSHOT Created-By: Apache Maven Bundle Plugin Export-Package: org.apache.sling.cassandra.resource.provider.mapper; version="0.0.1.SNAPSHOT", org.apache.sling.cassandra.resource.provider; uses:="javax.servlet.http, me.prettyprint.hector.api, org.apache.sling.api.resource, org.apache.sling.cassandra.resource.provider.mapper"; version="0.0.1.SNAPSHOT", org.apache.sling.cassandra.resource.provider.util; uses:="me.prettyprint.cassandra.model, me.prettyprint.cassandra.serializers, me.prettyprint.hector.api, me.prettyprint.hector.api.query, org.apache.sling.api.resource, org.apache.sling.cassandra.resource.provider"; version="0.0.1.SNAPSHOT" Import-Package: com.sun.org.apache.xerces.internal.impl.dv.util, javax.servlet.http, me.prettyprint.cassandra.model; version="[1.0, 2)", me.prettyprint.cassandra.serializers; version="[1.0, 2)", me.prettyprint.hector.api; version="[1.0, 2)", me.prettyprint.hector.api.beans; version="[1.0, 2)", me.prettyprint.hector.api.ddl; version="[1.0, 2)", me.prettyprint.hector.api.factory; version="[1.0, 2)", me.prettyprint.hector.api.query; version="[1.0, 2)", org.apache.sling.api.resource; version="[2.3, 3)", org.slf4j; version="[1.5, 2)" Manifest-Version: 1.0 Tool: Bnd-2.1.0.20130426-122213 > >> >> method B >> >> Goto http://localhost:8080/system/console >> >> select the bundle tab and install the bundle by uploading. >> >> If you monitor the logs you should see no errors, the bundle should >> register and you should be able to map cassandra read only to somewhere in >> the resource tree. >> > Can you explain in a bit more detail how I should map a cassandra resources from sling browser http://localhost:8080/.explorer.html ? . I can add different JCR node types. How my cassandra node type get registered here. Does it load all resource providers from reflection when I implemented the provider interface. > >> Best Regards >> Ian >> >> >> On 21 July 2013 13:08, Dishara Wijewardana >> wrote: >> >> > On Sun, Jul 21, 2013 at 5:32 PM, Dishara Wijewardana < >> > ddwijeward...@gmail.com> wrote: >> > >> > > >> > > >> > > On Fri, Ju
Re: [Status Update] Apache Cassandra backend for Sling
Hi, I'll start a new thread on this for you. Ian On 5 August 2013 01:20, Dishara Wijewardana wrote: > On Tue, Jul 23, 2013 at 5:32 PM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > >> >> >> On Mon, Jul 22, 2013 at 1:56 PM, Ian Boston wrote: >> >>> Hi Dishara, >>> >>> The Unit test coverage sounds great. I will pull the code and review >>> today. >>> >>> >>> Have you tried loading the bundle into a running Sling instance ? >> >> Not yet. I will try that out and let you know in one of following ways >> Thanks. >> >>> >>> Once you have built it you can load in 2 ways: >>> >>> method A >>> >>> mvn clean install sling:install >>> >>> The sling:install will post the Jar into the OSGi container over HTTP and >>> cause it to start. >>> >> Hi Ian > > I started Sling launchpad and Cassandra Server as well. And uploaded the > bundle through management console from http://localhost:8080/system/console > . > When I try to start the bundle it does nothing and no error logs in the > back end as well. The I noticed following where me.prettyprint.* classes > cannot loaded and hence my bundle cannot start. So ideally my bundle only > contains it self only. Not included its dependencies. As I feel one > option is get a me.prettyprint.* jar and install that bundle (I am not sure > whether it also fails due to a similar reason). So what is the best > approach to this ? > > > Symbolic Name org.apache.sling.cassandra > Version 0.0.1.SNAPSHOT > Bundle Location inputstream:org.apache.sling.cassandra-0.0.1-SNAPSHOT.jar > Last Modification Mon Aug 05 05:26:16 IST 2013 > Description Provides a ResourceProvider implementation supporting Apache > Cassandra based resources. > Start Level 20 > Exported Packages > org.apache.sling.cassandra.resource.provider,version=0.0.1.SNAPSHOT > org.apache.sling.cassandra.resource.provider.mapper,version=0.0.1.SNAPSHOT > org.apache.sling.cassandra.resource.provider.util,version=0.0.1.SNAPSHOT > Imported Packages com.sun.org.apache.xerces.internal.impl.dv.util -- > Cannot be resolved and overwritten by Boot Delegation > javax.servlet.http from org.apache.felix.http.jetty (1) > me.prettyprint.cassandra.model,version=[1.0,2) -- Cannot be resolved > me.prettyprint.cassandra.serializers,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.beans,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.ddl,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.factory,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.query,version=[1.0,2) -- Cannot be resolved > org.apache.sling.api.resource,version=[2.3,3) from org.apache.sling.api (98) > org.slf4j,version=[1.5,2) from slf4j.api (6) > Manifest Headers Bnd-LastModified: 1375238009819 > Build-Jdk: 1.6.0_26 > Built-By: dishara > Bundle-Description: Provides a ResourceProvider implementation supporting > Apache Cassandra based resources. > Bundle-ManifestVersion: 2 > Bundle-Name: Apache Sling Cassandra Resource Provider > Bundle-SymbolicName: org.apache.sling.cassandra > Bundle-Version: 0.0.1.SNAPSHOT > Created-By: Apache Maven Bundle Plugin > Export-Package: org.apache.sling.cassandra.resource.provider.mapper; > version="0.0.1.SNAPSHOT", org.apache.sling.cassandra.resource.provider; > uses:="javax.servlet.http, me.prettyprint.hector.api, > org.apache.sling.api.resource, > org.apache.sling.cassandra.resource.provider.mapper"; > version="0.0.1.SNAPSHOT", > org.apache.sling.cassandra.resource.provider.util; > uses:="me.prettyprint.cassandra.model, > me.prettyprint.cassandra.serializers, me.prettyprint.hector.api, > me.prettyprint.hector.api.query, org.apache.sling.api.resource, > org.apache.sling.cassandra.resource.provider"; version="0.0.1.SNAPSHOT" > Import-Package: com.sun.org.apache.xerces.internal.impl.dv.util, > javax.servlet.http, me.prettyprint.cassandra.model; version="[1.0, 2)", > me.prettyprint.cassandra.serializers; version="[1.0, 2)", > me.prettyprint.hector.api; version="[1.0, 2)", > me.prettyprint.hector.api.beans; version="[1.0, 2)", > me.prettyprint.hector.api.ddl; version="[1.0, 2)", > me.prettyprint.hector.api.factory; version="[1.0, 2)", > me.prettyprint.hector.api.query; version="[1.0, 2)", > org.apache.sling.api.resource; version="[2.3, 3)", org.slf4j; > version="[1.5, 2)" > Manifest-Version: 1.0 > Tool: Bnd-2.1.0.20130426-122213 > > > >> >>> >>> method B >>> >>> Goto http://localhost:8080/system/console >>> >>> select the bundle tab and install the bundle by uploading. >>> >>> If you monitor the logs you should see no errors, the bundle should >>> register and you should be able to map cassandra read only to somewhere in >>> the resource tree. >>> >> > Can you explain in a bit more detail how I should map a cassandra > resources from sling browser http://localhost:8080/.explorer.html ? . I > can add different JCR node types. How my cassandra node type get registered > here. Does it load all resource provid
Re: [Status Update] Apache Cassandra backend for Sling
On Mon, Aug 5, 2013 at 5:50 AM, Dishara Wijewardana wrote: > > > On Tue, Jul 23, 2013 at 5:32 PM, Dishara Wijewardana < > ddwijeward...@gmail.com> wrote: > >> >> >> On Mon, Jul 22, 2013 at 1:56 PM, Ian Boston wrote: >> >>> Hi Dishara, >>> >>> The Unit test coverage sounds great. I will pull the code and review >>> today. >>> >>> >>> Have you tried loading the bundle into a running Sling instance ? >> >> Not yet. I will try that out and let you know in one of following ways >> Thanks. >> >>> >>> Once you have built it you can load in 2 ways: >>> >>> method A >>> >>> mvn clean install sling:install >>> >>> The sling:install will post the Jar into the OSGi container over HTTP and >>> cause it to start. >>> >> Hi Ian > > I started Sling launchpad and Cassandra Server as well. And uploaded the > bundle through management console from > http://localhost:8080/system/console. > When I try to start the bundle it does nothing and no error logs in the > back end as well. The I noticed following where me.prettyprint.* classes > cannot loaded and hence my bundle cannot start. So ideally my bundle only > contains it self only. Not included its dependencies. As I feel one > option is get a me.prettyprint.* jar and install that bundle (I am not sure > whether it also fails due to a similar reason). So what is the best > approach to this ? > > > Symbolic Name org.apache.sling.cassandra > Version 0.0.1.SNAPSHOT > Bundle Location inputstream:org.apache.sling.cassandra-0.0.1-SNAPSHOT.jar > Last Modification Mon Aug 05 05:26:16 IST 2013 > Description Provides a ResourceProvider implementation supporting Apache > Cassandra based resources. > Start Level 20 > Exported Packages > org.apache.sling.cassandra.resource.provider,version=0.0.1.SNAPSHOT > org.apache.sling.cassandra.resource.provider.mapper,version=0.0.1.SNAPSHOT > org.apache.sling.cassandra.resource.provider.util,version=0.0.1.SNAPSHOT > Imported Packages com.sun.org.apache.xerces.internal.impl.dv.util -- > Cannot be resolved and overwritten by Boot Delegation > javax.servlet.http from org.apache.felix.http.jetty (1) > me.prettyprint.cassandra.model,version=[1.0,2) -- Cannot be resolved > me.prettyprint.cassandra.serializers,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.beans,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.ddl,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.factory,version=[1.0,2) -- Cannot be resolved > me.prettyprint.hector.api.query,version=[1.0,2) -- Cannot be resolved > org.apache.sling.api.resource,version=[2.3,3) from org.apache.sling.api > (98) > org.slf4j,version=[1.5,2) from slf4j.api (6) > Manifest Headers Bnd-LastModified: 1375238009819 > Build-Jdk: 1.6.0_26 > Built-By: dishara > Bundle-Description: Provides a ResourceProvider implementation supporting > Apache Cassandra based resources. > Bundle-ManifestVersion: 2 > Bundle-Name: Apache Sling Cassandra Resource Provider > Bundle-SymbolicName: org.apache.sling.cassandra > Bundle-Version: 0.0.1.SNAPSHOT > Created-By: Apache Maven Bundle Plugin > Export-Package: org.apache.sling.cassandra.resource.provider.mapper; > version="0.0.1.SNAPSHOT", org.apache.sling.cassandra.resource.provider; > uses:="javax.servlet.http, me.prettyprint.hector.api, > org.apache.sling.api.resource, > org.apache.sling.cassandra.resource.provider.mapper"; > version="0.0.1.SNAPSHOT", > org.apache.sling.cassandra.resource.provider.util; > uses:="me.prettyprint.cassandra.model, > me.prettyprint.cassandra.serializers, me.prettyprint.hector.api, > me.prettyprint.hector.api.query, org.apache.sling.api.resource, > org.apache.sling.cassandra.resource.provider"; version="0.0.1.SNAPSHOT" > Import-Package: com.sun.org.apache.xerces.internal.impl.dv.util, > javax.servlet.http, me.prettyprint.cassandra.model; version="[1.0, 2)", > me.prettyprint.cassandra.serializers; version="[1.0, 2)", > me.prettyprint.hector.api; version="[1.0, 2)", > me.prettyprint.hector.api.beans; version="[1.0, 2)", > me.prettyprint.hector.api.ddl; version="[1.0, 2)", > me.prettyprint.hector.api.factory; version="[1.0, 2)", > me.prettyprint.hector.api.query; version="[1.0, 2)", > org.apache.sling.api.resource; version="[2.3, 3)", org.slf4j; > version="[1.5, 2)" > Manifest-Version: 1.0 > Tool: Bnd-2.1.0.20130426-122213 > > > >> >>> >>> method B >>> >>> Goto http://localhost:8080/system/console >>> >>> select the bundle tab and install the bundle by uploading. >>> >>> If you monitor the logs you should see no errors, the bundle should >>> register and you should be able to map cassandra read only to somewhere >>> in >>> the resource tree. >>> >> > Can you explain in a bit more detail how I should map a cassandra > resources from sling browser http://localhost:8080/.explorer.html ? . I > can add different JCR node types. How my cassandra node type get registered > here. Does it load all resource providers from reflection when I > impl