[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Release Note: Adding the ability to get a reference to a table in the shell. Previously, all commands that acted on a table would need to take the name of the table as a string, which is annoying in an OO REPL. This patch introduces the ability to get and hold a reference to a table both on creation (via create(...)) and at will (via get_table(...)). Further, to actually make the table useful, modifications to table specific class were made so you can have a reference and just do things like put, scan, get, etc. on that table reference. To accommodate new table functionality, table specific methods are easily added (one line) in a dynamic fashion via class methods in the Table. See examples in get, put, scan, etc.. There is also a lot of admin functionality tied to a table - things like disabling, dropping, describing, etc - that were added to the table class. Now you can do things like 'table.disable' and 'table.describe'. Again these were dynamically added, so new admin functionality for a table is as simple as adding the method name to one line in the Table class. Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.1 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5815) ZKUtil.createWithParents should be transactional
[ https://issues.apache.org/jira/browse/HBASE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5815: --- Attachment: java_HBASE-5815-v0.patch Intial patch. Modifies the original method to attempt to create the parent path transactionally. ZKUtil.createWithParents should be transactional Key: HBASE-5815 URL: https://issues.apache.org/jira/browse/HBASE-5815 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Labels: zookeeper Fix For: 0.96.0, 0.94.1 Attachments: java_HBASE-5815-v0.patch Should solve a lot of tricky corner cases when you create the parent, get an update to watchers (who read ZK, but find no children) and then create the children. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5790) ZKUtil deleteRecurisively should be a recoverable operation
[ https://issues.apache.org/jira/browse/HBASE-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5790: --- Attachment: java_HBASE-5790-v1.patch Attaching new version addressing comments and now with more testing! ZKUtil deleteRecurisively should be a recoverable operation --- Key: HBASE-5790 URL: https://issues.apache.org/jira/browse/HBASE-5790 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Labels: zookeeper Fix For: 0.96.0, 0.94.1 Attachments: java_HBASE-5790-v1.patch, java_HBASE-5790.patch As of 3.4.3 Zookeeper now has full, multi-operation transaction. This means we can wholesale delete chunks of the zk tree and ensure that we don't have any pesky recursive delete issues where we delete the children of a node, but then a child joins before deletion of the parent. Even without transactions, this should be the behavior, but it is possible to make it much cleaner now that we have this new feature in zk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5790) ZKUtil deleteRecursively should be a recoverable operation
[ https://issues.apache.org/jira/browse/HBASE-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5790: --- Summary: ZKUtil deleteRecursively should be a recoverable operation (was: ZKUtil deleteRecurisively should be a recoverable operation) ZKUtil deleteRecursively should be a recoverable operation -- Key: HBASE-5790 URL: https://issues.apache.org/jira/browse/HBASE-5790 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Labels: zookeeper Fix For: 0.96.0, 0.94.1 Attachments: java_HBASE-5790-v1.patch, java_HBASE-5790.patch As of 3.4.3 Zookeeper now has full, multi-operation transaction. This means we can wholesale delete chunks of the zk tree and ensure that we don't have any pesky recursive delete issues where we delete the children of a node, but then a child joins before deletion of the parent. Even without transactions, this should be the behavior, but it is possible to make it much cleaner now that we have this new feature in zk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5790) ZKUtil deleteRecurisively should be a recoverable operation
[ https://issues.apache.org/jira/browse/HBASE-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5790: --- Attachment: java_HBASE-5790.patch Attaching patch and simple test case for recoverableZK. No need to update ZKUtil test since deleteRecurisive already covered. Wish there was a better way to test this out...tried doing a better version via Mockito, but seems to not be able to catch the actual method invocations and let me do extra work (though doAnswer should support it). Either way, did a hacked version where it dumps a node in middle while collecting children (setting a callable in the test that is called periodically in the delete), and seemed to work fine. Not recommending that we pursue the same course for general testing, but just putting this up here for peace of mind for all. ZKUtil deleteRecurisively should be a recoverable operation --- Key: HBASE-5790 URL: https://issues.apache.org/jira/browse/HBASE-5790 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Labels: zookeeper Fix For: 0.96.0, 0.94.1 Attachments: java_HBASE-5790.patch As of 3.4.3 Zookeeper now has full, multi-operation transaction. This means we can wholesale delete chunks of the zk tree and ensure that we don't have any pesky recursive delete issues where we delete the children of a node, but then a child joins before deletion of the parent. Even without transactions, this should be the behavior, but it is possible to make it much cleaner now that we have this new feature in zk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5775) ZKUtil doesn't handle deleteRecurisively cleanly
[ https://issues.apache.org/jira/browse/HBASE-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5775: --- Attachment: java_HBASE-5775.patch Attaching patch. Simple one line fix and updating associated test. ZKUtil doesn't handle deleteRecurisively cleanly Key: HBASE-5775 URL: https://issues.apache.org/jira/browse/HBASE-5775 Project: HBase Issue Type: Improvement Affects Versions: 0.94.0 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0, 0.96.0 Attachments: java_HBASE-5775.patch ZKUtil.deleteNodeRecursively()'s contract says that it handles deletion of the node and all its children. However, nothing is mentioned as to what happens if the node you are attempting to delete doesn't actually exist. Turns out, it throws a null pointer exception. I 'm proposing that we change the code s.t. it handles the case where the parent is already gone and exits cleanly, rather than failing horribly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Attachment: ruby_HBASE-5548-v3.patch Updated patch with: * more commenting! * updates for all the table commands * addition of admin utilities on a table (e.g. flush, enable, disable and drop) Think this is ready (and large enough to RB). thoughts? Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.1 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-50) Snapshot of table
[ https://issues.apache.org/jira/browse/HBASE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-50: - Affects Version/s: 0.96.0 Fix Version/s: 0.96.0 Snapshot of table - Key: HBASE-50 URL: https://issues.apache.org/jira/browse/HBASE-50 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Billy Pearson Assignee: Li Chongxin Priority: Minor Labels: gsoc Fix For: 0.96.0 Attachments: HBase Snapshot Design Report V2.pdf, HBase Snapshot Design Report V3.pdf, HBase Snapshot Implementation Plan.pdf, Snapshot Class Diagram.png Havening an option to take a snapshot of a table would be vary useful in production. What I would like to see this option do is do a merge of all the data into one or more files stored in the same folder on the dfs. This way we could save data in case of a software bug in hadoop or user code. The other advantage would be to be able to export a table to multi locations. Say I had a read_only table that must be online. I could take a snapshot of it when needed and export it to a separate data center and have it loaded there and then i would have it online at multi data centers for load balancing and failover. I understand that hadoop takes the need out of havening backup to protect from failed servers, but this does not protect use from software bugs that might delete or alter data in ways we did not plan. We should have a way we can roll back a dataset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Fix Version/s: 0.94.0 Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0, 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Fix Version/s: (was: 0.94.0) 0.94.1 Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.1 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Attachment: ruby_HBASE-5548-v2.patch Went back and forth a bunch on how I wanted to implement this (including coding up a couple different versions), hence the few days its taken me to get this patch up. Went with the current model as it helps keep concerns separate - table just does the internal work of how to do scans, puts, etc (the latter to be impl), and passes on the formatting work and specific help implementations to the command instances. To get here, had to add the ability to call a command by name in the Shell class, hence the internal_command method, but actually reuses most of the current support, even when calling the method from the table (safe views of commands, etc). Final consideration: we may want to add a method to the command class so a command instance can add the ability to add a method to a table more easily, instead of having to add the section seen at the bottom of ::Commands::Scan, where we open back up the Table class. Thoughts? Or is that work minimal enough to justify for adding new commands? If we are happy with this, I'll port the other table commands to this new model and post up a patch for review. For completeness, the other major options considered: * have table do all the formatting,etc now handled by the commands ** side effect - commands are pretty useless except for help and is a pretty big rewrite and require the table to get a formmater as well as an instance of each command ** bonus - methods like scan_internal can be made private * similar to this patch, but have the command on load register itself with the table so can do help ** minus - lots of reproduced functionality like what the Shell is currently doing with loading commands. Also, don't gain much and same problems as above ** bonus - code is a bit more centralized Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-5548:
---
Attachment: ruby_HBASE-5548-v1.patch
Attaching patch...
1) Fixed stack's comments
2) Added features:
- create now returns the table that was created
- scan now dynamically adds a scan method to htable
The latter makes sure that the table will output a scan 'nicely', and could be
a decent paradigm for adding methods to the Table class (though probably some
cleanup needs to be done as far as aliasing methods etc.). Thoughts on this new
style?
Also, here is the output from some usage:
{code}
1.9.2-p290 :006 t = create 't', 'f'
0 row(s) in 1.0470 seconds
= #Hbase::Table:0x55187eb3
@table=#Java::OrgApacheHadoopHbaseClient::HTable:0x92524b0,
@shell=#Shell::Shell:0x329b5c95
@formatter=#Shell::Formatter::Console:0x75db5427 @out=Kernel, @row_count=0,
@max_width=80, @hbase_admin=#Hbase::Admin:0x127208e4
@zk_wrapper=#Java::OrgApacheHadoopHbaseZookeeper::ZooKeeperWatcher:0xcb80017,
@zk_main=#Java::OrgApacheZookeeper::ZooKeeperMain:0x646326e5,
@formatter=#Shell::Formatter::Console:0x75db5427 @out=Kernel, @row_count=0,
@max_width=80, @conf=#Java::OrgApacheHadoopConf::Configuration:0x4332b67c,
@admin=#Java::OrgApacheHadoopHbaseClient::HBaseAdmin:0x2fbb3e9a,
@hbase=#Hbase::Hbase:0x56781dd4
@configuration=#Java::OrgApacheHadoopConf::Configuration:0x4332b67c
1.9.2-p290 :007 t.put 'r', 'f', 'v'
1.9.2-p290 :008 t1 = get_table 't'
0 row(s) in 0.0240 seconds
= #Hbase::Table:0x7194f467
@table=#Java::OrgApacheHadoopHbaseClient::HTable:0x6a39d072,
@shell=#Shell::Shell:0x329b5c95
@formatter=#Shell::Formatter::Console:0x75db5427 @out=Kernel, @row_count=0,
@max_width=80, @hbase_admin=#Hbase::Admin:0x127208e4
@zk_wrapper=#Java::OrgApacheHadoopHbaseZookeeper::ZooKeeperWatcher:0xcb80017,
@zk_main=#Java::OrgApacheZookeeper::ZooKeeperMain:0x646326e5,
@formatter=#Shell::Formatter::Console:0x75db5427 @out=Kernel, @row_count=0,
@max_width=80, @conf=#Java::OrgApacheHadoopConf::Configuration:0x4332b67c,
@admin=#Java::OrgApacheHadoopHbaseClient::HBaseAdmin:0x2fbb3e9a,
@hbase=#Hbase::Hbase:0x56781dd4
@configuration=#Java::OrgApacheHadoopConf::Configuration:0x4332b67c
1.9.2-p290 :009 t.put 'r2', 'f', 'v'
1.9.2-p290 :010 t.scan
ROW COLUMN+CELL
rcolumn=f:, timestamp=1331521351836, value=v
r2 column=f:, timestamp=1331521368969, value=v
2 row(s) in 0.0870 seconds
{code}
Add ability to get a table in the shell
---
Key: HBASE-5548
URL: https://issues.apache.org/jira/browse/HBASE-5548
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0
Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch
Currently, all the commands that operate on a table in the shell first have
to take the table as name as input.
There are two main considerations:
* It is annoying to have to write the table name every time, when you should
just be able to get a reference to a table
* the current implementation is very wasteful - it creates a new HTable for
each call (but reuses the connection since it uses the same configuration)
We should be able to get a handle to a single HTable and then operate on that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Attachment: ruby_HBASE-5528-v0.patch Initial cut at this ticket. Its a little unclean as we don't have a nice way to isolate the java methods from the ruby methods and absolutely zero formatting from the output of the different commands as that is currently handled by the individual 'command' implementations. Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Status: Patch Available (was: Open) Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v5.patch Applying fixes from RB - should be ready to go. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v7.patch Rebased against 0.94. I believe this also applies against trunk, if we want to commit it there as well. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v3.patch Simpler patch than v2 - just covers up the hfiles, .regioninfo and .tableinfo. The last was a bit more complicated than I would like as a TableDescriptor doesn't implicitly have a configuration, but it manages. The other oddity is that this patch takes a conf key when getting the permissions - did this to help keep it abstracted out, but it could debatably just drop that param and always look to using the default (and maybe have an overloaded method that takes the first to check and then falls back on hbase.data.umask if that value isn't set). Definitely up for debate here. Throwing this up on RB as well. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Optional file permission settings
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v2.patch Attaching patch for general umask application for files and directories. Essentially, the patch takes all the file and directory creation in src/main and pipes it through new methods in FSUtils, which automatically apply the file permissions if they are turned on, rather than just doing the straigh fs.create() or fs.mkdirs() calls. Unfortunately, this is a wider sweeping change than anticipated as there are a lot of places that create files and directories. However, this helps centralize the code in the future, if we want to more/fancier stuff to the files. Optional file permission settings - Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, it is often times adventageous to only allow a subset of the world to view the actual data written by hbase when scanning the raw hdfs files. This ticket covers setting permissions for files written to hdfs that are storing actual user data, as opposed to _all_ files written to hdfs as many of them contain non-identifiable metadata. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Description: Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. was: Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, it is often times adventageous to only allow a subset of the world to view the actual data written by hbase when scanning the raw hdfs files. This ticket covers setting permissions for files written to hdfs that are storing actual user data, as opposed to _all_ files written to hdfs as many of them contain non-identifiable metadata. Summary: Configurable file and directory based umask (was: Optional file permission settings) Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Optional file permission settings
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526.patch First iteration, no unit tests, but should cover the identifiable data in hdfs. Optional file permission settings - Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, it is often times adventageous to only allow a subset of the world to view the actual data written by hbase when scanning the raw hdfs files. This ticket covers setting permissions for files written to hdfs that are storing actual user data, as opposed to _all_ files written to hdfs as many of them contain non-identifiable metadata. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: java_HBASE-5162.patch Worked up an initial implementation that doesn't actually break the RPC (I think, even if client or the server has monitoring turned on), but avoids doing the exception passing back to the client, which IMHO is the least clean way to handle this as exceptions are for exceptional cases, not the common case. Right now the implementation only covers Puts, but could definitely be extended to cover other writes; I wanted to get some feedback on general style, etc. before going for the full blown implementation (and all the general cleanup associated with a 'real' patch). As far as how it is put together... Client has a backoff policy to take into consideration what the server is saying. This policy can take into account the current, max, and growth size of the write buffer for figuring out how long to sleep. The server pressure is unwrapped from the server in the Result as a MonitoredResult and then updated on the client. The next put will then take into account that pressure when attempting a put. The problem here is that the server can't tell all clients that the pressure has gone down, but that trade-off is common given traditional collision/backoff situations. The client has also been given permission to grow to a multiplier of its writeBufferSize, similar to the memstoremultiplier, allowing it to buffer more writes. If a write is within the expansion range, we want to allow the client to accept more writes while waiting/backing-off, so we launch a flusher thread that after waiting the backoff time will flush the writes (singleton). This gives us back-off as well as some flexiblilty on the client as to how much we buffer. To disable the backoff behavior, its as simple as setting the multiplier to 1, so the expansion = max. Similarly, on the RS, we make the policy for monitoring pluggable and optional to enable the blocking for a given threshold. The policy here can take into account the current number of store files, the number where we will get blocking and the current % full of the memstore (total size and max size). This allows us to ensure that we only slow down based on a given policy (yet to be implemented) but do get protection for the RS from malicious/over-eager clients. This impl may be a little over the top, but it does cover both sides of the on-the-wire story. Only things left to do are to extend this to other writes as well as creating at least one (if not two or three) tunable policies for figuring out blocking. Thoughts? Basic client pushback mechanism --- Key: HBASE-5162 URL: https://issues.apache.org/jira/browse/HBASE-5162 Project: HBase Issue Type: New Feature Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Fix For: 0.94.0 Attachments: java_HBASE-5162.patch The current blocking we do when we are close to some limits (memstores over the multiplier factor, too many store files, global memstore memory) is bad, too coarse and confusing. After hitting HBASE-5161, it really becomes obvious that we need something better. I did a little brainstorm with Stack, we came up quickly with two solutions: - Send some exception to the client, like OverloadedException, that's thrown when some situation happens like getting past the low memory barrier. It would be thrown when the client gets a handler and does some check while putting or deleting. The client would treat this a retryable exception but ideally wouldn't check .META. for a new location. It could be fancy and have multiple levels of pushback, like send the exception to 25% of the clients, and then go up if the situation persists. Should be easy to implement but we'll be using a lot more IO to send the payload over and over again (but at least it wouldn't sit in the RS's memory). - Send a message alongside a successful put or delete to tell the client to slow down a little, this way we don't have to do back and forth with the payload between the client and the server. It's a cleaner (I think) but more involved solution. In every case the RS should do very obvious things to notify the operators of this situation, through logs, web UI, metrics, etc. Other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5354) Source to standalone deployment script
[ https://issues.apache.org/jira/browse/HBASE-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5354: --- Attachment: bash_HBASE-5354.patch Attaching patch that does as described. Apply the patch and then run it from the /hbase as ./dev-support/deploy.sh -h to see the usage info. Source to standalone deployment script -- Key: HBASE-5354 URL: https://issues.apache.org/jira/browse/HBASE-5354 Project: HBase Issue Type: New Feature Components: build, scripts Affects Versions: 0.94.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Attachments: bash_HBASE-5354.patch Automating the testing of source code in a 'real' instance can be a bit of a pain, even getting it into standalone mode. Steps you need to go through: 1) Build the project 2) Copy it to the deployment directory 3) Shutdown the current cluster (if it is running) 4) Untar the tar 5) Update the configs to point to a local data cluster 6) Startup the new deployment Yeah, its not super difficult, but it would be nice to just have a script to make it button push easy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5353: --- Description: Currently, the HMaster node must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. (was: Currently, the HMaster node must be considered a 'special' node (single point of failure), meaning that the node must be protected more than the other commodity machines. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or with multiple instances and automatic failover. ) HA/Distributed HMaster via RegionServers Key: HBASE-5353 URL: https://issues.apache.org/jira/browse/HBASE-5353 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Priority: Minor Currently, the HMaster node must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5353: --- Description: Currently, the HMaster node(s) must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines or at least specially monitored. Minimally, we always need to ensure that the master is running, rather than letting the system handle that internally. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. (was: Currently, the HMaster node must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. ) HA/Distributed HMaster via RegionServers Key: HBASE-5353 URL: https://issues.apache.org/jira/browse/HBASE-5353 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Priority: Minor Currently, the HMaster node(s) must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines or at least specially monitored. Minimally, we always need to ensure that the master is running, rather than letting the system handle that internally. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5348) Constraint configuration loaded with bloat
[ https://issues.apache.org/jira/browse/HBASE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5348: --- Attachment: java_HBASE-5348.patch Fix for the creating the configuration and updating the tests to check for the proper conf. Constraint configuration loaded with bloat -- Key: HBASE-5348 URL: https://issues.apache.org/jira/browse/HBASE-5348 Project: HBase Issue Type: Bug Components: coprocessors, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Attachments: java_HBASE-5348.patch Constraints load the configuration but don't load the 'correct' configuration, but instead instantiate the default configuration (via new Configuration). It should just be Configuration(false) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5348) Constraint configuration loaded with bloat
[ https://issues.apache.org/jira/browse/HBASE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5348: --- Status: Patch Available (was: Open) Constraint configuration loaded with bloat -- Key: HBASE-5348 URL: https://issues.apache.org/jira/browse/HBASE-5348 Project: HBase Issue Type: Bug Components: coprocessors, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Attachments: java_HBASE-5348.patch Constraints load the configuration but don't load the 'correct' configuration, but instead instantiate the default configuration (via new Configuration). It should just be Configuration(false) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5348) Constraint configuration loaded with bloat
[ https://issues.apache.org/jira/browse/HBASE-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5348: --- Attachment: java_HBASE-5348.patch Updating the patch, should be good to go. Constraint configuration loaded with bloat -- Key: HBASE-5348 URL: https://issues.apache.org/jira/browse/HBASE-5348 Project: HBase Issue Type: Bug Components: coprocessors, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Attachments: java_HBASE-5348.patch, java_HBASE-5348.patch Constraints load the configuration but don't load the 'correct' configuration, but instead instantiate the default configuration (via new Configuration). It should just be Configuration(false) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5350) Fix jamon generated package names
[ https://issues.apache.org/jira/browse/HBASE-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5350: --- Attachment: jamon_HBASE-5350.patch Just had to move the jamon files down a directory, which then builds them with the right package. Passed 'mvn clean test' Fix jamon generated package names - Key: HBASE-5350 URL: https://issues.apache.org/jira/browse/HBASE-5350 Project: HBase Issue Type: Bug Components: monitoring Affects Versions: 0.92.0 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: jamon_HBASE-5350.patch Previously, jamon was creating the template files in org.apache.hbase, but it should be org.apache.hadoop.hbase, so it's in line with rest of source files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5318) Support Eclipse Indigo
[ https://issues.apache.org/jira/browse/HBASE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5318: --- Status: Patch Available (was: Reopened) Support Eclipse Indigo --- Key: HBASE-5318 URL: https://issues.apache.org/jira/browse/HBASE-5318 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Environment: Eclipse Indigo (1.4.1) which includes m2eclipse (1.0 SR1). Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Labels: maven Attachments: mvn_HBASE-5318_r0.patch, mvn_HBASE-5318_r1.patch The current 'standard' release of Eclipse (indigo) comes with m2eclipse installed. However, as of m2e v1.0, interesting lifecycle phases are now handled via a 'connector'. However, several of the plugins we use don't support connectors. This means that eclipse bails out and won't build the project or view it as 'working' even though it builds just fine from the the command line. Since Eclipse is one of the major java IDEs and that Indigo has been around for a while, we should make it easy to for new devs to pick up the code and for older devs to upgrade painlessly. The original build should not be modified in any significant way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5318) Support Eclipse Indigo
[ https://issues.apache.org/jira/browse/HBASE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5318: --- Attachment: mvn_HBASE-5318_r0.patch Attaching patch for allowing eclipse to process the dependencies properly. Support Eclipse Indigo --- Key: HBASE-5318 URL: https://issues.apache.org/jira/browse/HBASE-5318 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Environment: Eclipse Indigo (1.4.1) which includes m2eclipse (1.0 SR1). Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Labels: maven Attachments: mvn_HBASE-5318_r0.patch The current 'standard' release of Eclipse (indigo) comes with m2eclipse installed. However, as of m2e v1.0, interesting lifecycle phases are now handled via a 'connector'. However, several of the plugins we use don't support connectors. This means that eclipse bails out and won't build the project or view it as 'working' even though it builds just fine from the the command line. Since Eclipse is one of the major java IDEs and that Indigo has been around for a while, we should make it easy to for new devs to pick up the code and for older devs to upgrade painlessly. The original build should not be modified in any significant way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5070) Constraints implementation and javadoc changes
[ https://issues.apache.org/jira/browse/HBASE-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5070: --- Attachment: java_HBASE-5070-v2.patch Attaching patch with all changes as per RB and latest comments on this ticket. Should be good to go, less any more review (though I believe all concerns were addressed). Constraints implementation and javadoc changes -- Key: HBASE-5070 URL: https://issues.apache.org/jira/browse/HBASE-5070 Project: HBase Issue Type: Task Reporter: Zhihong Yu Attachments: java_HBASE-5070-v2.patch This is continuation of HBASE-4605 See Stack's comments https://reviews.apache.org/r/2579/#review3980 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5070) Constraints implementation and javadoc changes
[ https://issues.apache.org/jira/browse/HBASE-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5070: --- Attachment: java_HBASE-5070-v3.patch this time, attached with --no-prefix (I really have to make that the default...) Constraints implementation and javadoc changes -- Key: HBASE-5070 URL: https://issues.apache.org/jira/browse/HBASE-5070 Project: HBase Issue Type: Task Reporter: Zhihong Yu Attachments: java_HBASE-5070-v2.patch, java_HBASE-5070-v3.patch This is continuation of HBASE-4605 See Stack's comments https://reviews.apache.org/r/2579/#review3980 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v7.patch
Updated patch with fixed javadocs as per Ted's comments.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605-v6.txt, 4605.v7, constraint_as_cp.txt,
java_Constraint_v2.patch, java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch,
java_HBASE-4605_v3.patch, java_HBASE-4605_v5.patch, java_HBASE-4605_v7.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v5.patch
For Hadoop-QA
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch,
java_HBASE-4605_v5.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v5.patch
Ok, this time --no-prefix (duh).
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch,
java_HBASE-4605_v5.patch, java_HBASE-4605_v5.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v2.patch
oops, wrong file. This should be right.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_HBASE-4605_v3.patch
Forgot --no-prefix. Let's try this again.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Summary: Constraints (was: Add constraints as a top-level feature)
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4605) Add constraints as a top-level feature
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-4605:
---
Attachment: java_Constraint_v2.patch
Constraint implementation that is just added as a coprocessor. Not implemented
as a Precondition for ease, though it could be ported over to that fairly
easily. Basically, putting this up for posterity since the consensus seems to
be pursuing #1 above.
Also, is there a better way to pass back exceptions from coprocessors? Right
now, the exception causes a retry which is a huge timeout problem
Add constraints as a top-level feature
--
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4559) Refactor TestAvroServer into an integration test
[ https://issues.apache.org/jira/browse/HBASE-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-4559: --- Attachment: java_HBASE_4559.txt Refactor TestAvroServer into an integration test Key: HBASE-4559 URL: https://issues.apache.org/jira/browse/HBASE-4559 Project: HBase Issue Type: Improvement Components: test Reporter: Jesse Yates Attachments: java_HBASE_4559.txt TestAvroServer is a beefy test, spins up a mini cluster, does a large series of manipulations and then spins it down. It take about 2 mins to run on a local machine, which on the high side for a 'unit' test. This is part of the implentation discussed in http://search-hadoop.com/m/L9OzBNEOJK1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4561) Update Maven documentation in book
[ https://issues.apache.org/jira/browse/HBASE-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-4561: --- Attachment: book_HBASE-4561.txt Adding fix. Updates that maven 3 is standard and moves info about integration tests up to be with the rest of the maven commands Update Maven documentation in book -- Key: HBASE-4561 URL: https://issues.apache.org/jira/browse/HBASE-4561 Project: HBase Issue Type: Improvement Components: documentation Reporter: Jesse Yates Priority: Minor Attachments: book_HBASE-4561.txt The maven documentation is a little out of date and has recently led to some confusion about tests. This would cleanup the maven documents in the book to be more explicit about how maven should be used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4454) Add failsafe plugin to build and rename integration tests
[ https://issues.apache.org/jira/browse/HBASE-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-4454: --- Attachment: mvn_HBASE-4454.patch Patch uploaded to separate out running integration tests. IntegrationTests must be named as **/IntegrationTest*.java. They can be run with the command: 'mvn verfy'. Since verify is part of the standard build phases, under assembly, package, etc, integration tests will be run automatically when doing a full build. @Stack: should I open up a separate ticket, new patch version, or just add another patch for updating documentation? Do we even need to update the docs for this? Add failsafe plugin to build and rename integration tests - Key: HBASE-4454 URL: https://issues.apache.org/jira/browse/HBASE-4454 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Attachments: mvn_HBASE-4454.patch Add the maven-failsafe-plugin to the build process so we can run integration tests with mvn verify. This will also involve a renaming of integration tests to conform to a new integration test regex. This is a stopgap measure while we until break them out into their own module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
