GitHub user alfonsonishikawa opened a pull request:
https://github.com/apache/gora/pull/135
Goraexplorer needed changes
This set of commits implements the following functionalities:
- Compile schemas from Strings and not only from files.
- Allow to configure the mapping of a HBase backed datastore when creating
the datastore
- Added a new factory method to create a datastore using class' names and
not depending on generics
- Updated Persistent to get the related schema.
- Added #clone in Persistent
- Enabled the possibility to retrieve information from the native backend
behind a datastore. This information is used in the [schema's wizard of
GoraExplorer](https://bitbucket.org/alfonsonishikawa/goraexplorer/wiki/User%20manual#markdown-header-handling-schemas-and-mappings-with-the-wizard).
- Added gora-pig module to access Gora from Pig.
[More details on
gora-pig](http://people.apache.org/~alfonsonishikawa/gora-109.html)
Please be picky on this pull request. Any doubt, anything, ask for it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/alfonsonishikawa/gora goraexplorer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/gora/pull/135.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #135
----
commit 6949760dcda2be83a3c408502d89dfdc962f3764
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2016-07-25T17:17:58Z
Creación de la rama "goraexplorer" con los .pom modificados para que
hagan referencia al repositorio maven.nishilua.com donde se aloja
0.7-goraexplorer-SNAPSHOT que soporta las funcionalidades de
goraexplorer, como por ejemplo:
- Crear un DataStore con un Configuration y Properties sin necesidad de
disponer de un DataStore<K,T> en tiempo de compilación -malditos
generics-
commit d607fdfe37c3167fb997708811effc5c94f54af3
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2016-07-28T14:33:07Z
- Compile a schema from a String, and not only from file.
- Allow to get the mapping from Configuration (key gora.mapping) as a
String, and not only from file.
- Instantiate a datastore given de datastore class, the key class and
the persistent class by name, not relying on generics.
commit b9f9f56bd29ce2fefc57acae76ad31c2f075840e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2016-09-19T17:29:58Z
Modified Persistent to return it's schema.
commit 4410c415cc322244046a2234350c8d73f8649757
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2016-09-22T21:12:23Z
Merge branch 'master' into goraexplorer
commit f3aed481b3d73f655f61e1dd35641f5a03de1415
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-02-06T20:34:54Z
Merge branch 'master' into goraexplorer
commit e6eab433cf5584bac1f418dbc0caa5427408a209
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-05-21T19:04:50Z
BUGFIX: HBaseStore endKey is exclusive (since HBase handles it
exclusive). We add a trail zero to the end key when searching to make it
inclusive to conform to Gora's quey interface.
commit 3b35c36fa6855512fe3a228b82caf32952271203
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-07-14T21:58:17Z
Actualizando el proyecto a todos los cambios de la release 0.7
commit 055d5b48cc618b41b2a2d41d95e83fcfa232d838
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-07-17T23:52:20Z
Added #clone() method to Persistent
commit b58722b587a29c8d8d069ceaafe4c2f63f6cc47e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-07-18T00:08:43Z
Updated pom.xmls to deploy
commit 43ddc91f9e0b77cbefdda850b7aadf3261505e04
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-09-22T21:12:25Z
Implemented DataStoreMetadataAnalyzer to retrieve native backend
information.
- Implemented only for HBase at this moment.
commit e3496145d2ee1d147bcf2af176b99c0ad12d43b9
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-09-26T09:50:45Z
Added missing #close() method to DataStoreMetadataAnalyzer
commit 0ae4ec44259ea0e3feec3e76671420438547fdae
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-09-26T11:27:56Z
Updated DataStoreMetadataFactory to throw ClassNotFoundException not
hiding this exceptional case.
commit bf1165fac897d7d51fd6645a0eb8dfdd4eb4165c
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-12-07T12:38:45Z
Updated Persistent with @JsonIgnore to allow jackson serialization
commit 6dc22695d51eb464df4bb27765b1af18ca757b9e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2017-12-20T21:06:10Z
Added a constructor to BeanFactoryImpl with String parameter
commit 5e5e43dad805433c533948f63599ed27706ed6cb
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-04-03T17:38:58Z
Merged with Gora 0.9 - breaking code, will fix in next commits
commit 6d40006563f96442c039b9541355e3c18390ddab
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-04-04T21:58:50Z
Fixed errors after merging
commit 0afed6a711c208fee64328cb852712e57036af5d
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-04-21T22:06:54Z
Initial commit of gora-pig
commit 8e3e3f35b903be1f73ae866f03b9c0da8c097db5
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-04-24T15:45:35Z
Refactorization of GoraStorage for LOAD command
commit 0b60c1fa734001f087c9865c49fdeaebae574557
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-04-27T20:27:04Z
Support in gora-pig of JSON configuration for GoraStorage.
The format is the following:
set job.name 'GoraPig test';
register gora/*.jar;
webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl,status",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n <field name=\\"status\\" family=\\"f\\"
qualifier=\\"st\\"/>\\n </class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
dump webpage ;
commit 4ca9bd3382d1e6b5e768f984b4ab804d6f0259bd
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-05-02T17:46:33Z
Updated GoraStorage to allow several LOADs from different origins.
Now the following script is supported:
set job.name 'GoraPig test';
register gora/*.jar;
webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n </class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
webpage2 = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "contentType",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"contentType\\" family=\\"f\\"
qualifier=\\"typ\\"/>\\n <field name=\\"baseUrl\\" family=\\"f\\"/>\\n
</class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
joined = JOIN webpage BY key, webpage2 BY key;
joined2 = FOREACH joined GENERATE webpage::key, webpage::baseUrl,
webpage2::contentType;
dump joined2 ;
commit 77be0d83d64f51553a915915ee6d56dafb6e75ed
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-05-03T14:36:54Z
Updated Pig's GoraStorage to allow STORE operations.
It performs a map-side store with this script:
**
set job.name 'GoraPig test3';
register gora/*.jar;
webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n </class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
webpage_dest = STORE webpage INTO '.' USING
org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage_dest\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage_dest\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n <field name=\\"contentType\\" family=\\"f\\"
qualifier=\\"typ\\"/>\\n</class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
**
and a reduce-side store with this other:
**
set job.name 'GoraPig test4';
register gora/*.jar;
webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n </class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
webpage2 = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "contentType",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"contentType\\" family=\\"f\\"
qualifier=\\"typ\\"/>\\n </class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
joined = JOIN webpage BY key, webpage2 BY key;
joined2 = FOREACH joined GENERATE webpage::key as key, webpage::baseUrl as
baseUrl, webpage2::contentType as contentType;
webpage_dest = STORE joined2 INTO '.' USING
org.apache.gora.pig.GoraStorage('{
"persistentClass": "admin.WebPage",
"fields": "baseUrl,contentType",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n <table name=\\"webpage_dest\\">\\n
<family name=\\"f\\" maxVersions=\\"1\\"/>\\n </table>\\n <class
table=\\"webpage_dest\\" keyClass=\\"java.lang.String\\"
name=\\"admin.WebPage\\">\\n <field name=\\"baseUrl\\" family=\\"f\\"
qualifier=\\"bas\\"/>\\n <field name=\\"contentType\\" family=\\"f\\"
qualifier=\\"typ\\"/>\\n</class>\\n</gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
**
commit 5578e731d97d974902d923da646650228a2a1446
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date: 2018-05-03T18:06:13Z
Updated GoraDeleteStorage to allow STORE that deletes rows by key.
Bugfixes in GoraStorage when storing.
Delete syntaxis, given a (key:chararray) relation:
STORE tuples_to_delete INTO '.' USING
org.apache.gora.pig.GoraDeleteStorage('{
"persistentClass": "admin.WebPage",
"goraProperties":
"gora.datastore.default=org.apache.gora.hbase.store.HBaseStore",
"mapping": "<?xml version=\\"1.0\\"
encoding=\\"UTF-8\\"?><gora-odm><table name=\\"webpage_dest\\"><family
name=\\"f\\" maxVersions=\\"1\\"/></table><class table=\\"webpage_dest\\"
keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\"></class></gora-odm>",
"configuration": {
"hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
"zookeeper.znode.parent": "/hbase-unsecure"
}
}') ;
commit 89225d7ab30ef8fdc650d74ae93454a88fe14dc9
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date: 2018-10-03T19:38:37Z
Fixed imports
commit 6cec5f66c42d5439f1c491ac2d6d9c01d831db29
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date: 2018-10-03T23:11:04Z
Merge branch 'master' into goraexplorer
commit 3b11b1c0e6852be80487963213c8d32774aa0c7e
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date: 2018-10-03T23:26:56Z
Updated version from -goraexplorer- to 0.9-SNAPSHOT
----
---
