[jira] [Resolved] (HBASE-18877) Move Type out of KeyValue to Cell
[
https://issues.apache.org/jira/browse/HBASE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai resolved HBASE-18877.
Resolution: Won't Fix
User can use {{CellBuilder}} and {{CellBuilder#DataType}} to build the {{Cell}}
now. All of these class are IA.Public, so this issue is not necessary for my
story. Just close this as "Won't Fix".
> Move Type out of KeyValue to Cell
> -
>
> Key: HBASE-18877
> URL: https://issues.apache.org/jira/browse/HBASE-18877
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0
>
>
> We allow user to create custom cell but the valid code of type -
> {{KeyValue#Type}} - is declared as IA.Private. We should make it
> IA.Public...Or move it to {{Cell}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
To those who commented on the hbase refguide back in 2012...
I know, it was just one or two of you, but FYI, We used disqus to catch comments on the old docbook refguide. It turns out disqus had a compromise of a database from 2012 [1]. For those of you who left a comment back then, your login may have been exposed. The disqus site says: Potential Impact For Users: - Right now there isn’t any evidence of unauthorized logins occurring in relation to this. No plain text passwords were exposed, but it is possible for this data to be decrypted (even if unlikely). As a security precaution, we have reset the passwords for all affected users. We recommend that all users change passwords on other services if they are shared. - Email addresses are in plain text here, so it’s possible that affected users may receive spam or unwanted emails. - At this time, we do not believe that this data is widely distributed or readily available. We can also confirm that the most recent data that was exposed is from July, 2012. See [1] for more detail. FYI. Thanks, St.Ack 1. https://blog.disqus.com/security-alert-user-info-breach
[jira] [Created] (HBASE-18965) Create alternate API to processRowsWithLock() that doesn't take RowProcessor as an argument
Umesh Agashe created HBASE-18965: Summary: Create alternate API to processRowsWithLock() that doesn't take RowProcessor as an argument Key: HBASE-18965 URL: https://issues.apache.org/jira/browse/HBASE-18965 Project: HBase Issue Type: Sub-task Components: Coprocessors Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Create alternate API to processRowsWithLock() that doesn't take RowProcessor as an argument. Also write example showing how coprocessors and batchMutate() can be used instead of RowProcessors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18964) Deprecate RowProcessor and processRowsWithLocks() APIs that take RowProcessor as an argument
Umesh Agashe created HBASE-18964: Summary: Deprecate RowProcessor and processRowsWithLocks() APIs that take RowProcessor as an argument Key: HBASE-18964 URL: https://issues.apache.org/jira/browse/HBASE-18964 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Deprecate RowProcessor and processRowsWithLocks() APIs that take RowProcessor as an argument -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18963) Remove MultiRowMutationProcessor and implement mutateRows... methods using batchMutate()
Umesh Agashe created HBASE-18963: Summary: Remove MultiRowMutationProcessor and implement mutateRows... methods using batchMutate() Key: HBASE-18963 URL: https://issues.apache.org/jira/browse/HBASE-18963 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Remove MultiRowMutationProcessor and implement mutateRows... methods using batchMutate() -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18962) Support atomic BatchOperations
Umesh Agashe created HBASE-18962: Summary: Support atomic BatchOperations Key: HBASE-18962 URL: https://issues.apache.org/jira/browse/HBASE-18962 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Support all mutations in BatchOperations to be applied atomically (all or none) by locking all rows corresponding to mutations exclusively. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18961) doMiniBatchMutate() is big, split it into smaller methods
Umesh Agashe created HBASE-18961: Summary: doMiniBatchMutate() is big, split it into smaller methods Key: HBASE-18961 URL: https://issues.apache.org/jira/browse/HBASE-18961 Project: HBase Issue Type: Sub-task Components: Coprocessors Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Split doMiniBatchMutate() and improve readability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18960) A few bug fixes and minor improvements around batchMutate()
Umesh Agashe created HBASE-18960: Summary: A few bug fixes and minor improvements around batchMutate() Key: HBASE-18960 URL: https://issues.apache.org/jira/browse/HBASE-18960 Project: HBase Issue Type: Sub-task Components: Coprocessors Affects Versions: 2.0.0-alpha-3 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-4 Batch validation and preparation can be done before we start iterating over batch operations for writes. observedExceptions, familyCellMaps and durability can be stored in BatchOperation as they are batch wide. For all operations, done by preBatchMutate() CP hook, operation status needs to be SUCCESS. Modify and use doWALAppend() from doMiniBatchMutate(). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18959) Backport HBASE-18874 (HMaster abort message will be skipped if Throwable is passed null) to branch-1
Pankaj Kumar created HBASE-18959: Summary: Backport HBASE-18874 (HMaster abort message will be skipped if Throwable is passed null) to branch-1 Key: HBASE-18959 URL: https://issues.apache.org/jira/browse/HBASE-18959 Project: HBase Issue Type: Bug Affects Versions: 1.4.0, 1.3.2, 1.5.0, 1.2.7 Reporter: Pankaj Kumar Assignee: Pankaj Kumar Priority: Minor Backport HBASE-18874 to branch-1/1.4/1.3/1.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18958) Remove the IS annotation from SpaceLimitingException
Chia-Ping Tsai created HBASE-18958:
--
Summary: Remove the IS annotation from SpaceLimitingException
Key: HBASE-18958
URL: https://issues.apache.org/jira/browse/HBASE-18958
Project: HBase
Issue Type: Bug
Reporter: Chia-Ping Tsai
Priority: Trivial
{code}
@InterfaceAudience.Public
@InterfaceStability.Evolving
public class SpaceLimitingException extends QuotaExceededException {
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: Looking for input on an alpha-4 thorny item
I think at least at the Store level we want to admit the possibility of alternate implementations. Region is pushing it. It would be ideal of course to have nice clean interfaces which could be mocked or given to new implementations, but the legacy aspects of the code base probably make that more work than worth the effort unless someone is really going to try going all the way to a new type of Region. On Fri, Oct 6, 2017 at 9:33 AM, Stack wrote: > On Thu, Oct 5, 2017 at 9:30 AM, Josh Elser wrote: > > > (I think I understand the problems enough to comment, but, admittedly, my > > 5minute read is probably lacking) > > > > > Thanks for chiming-in Josh. > > > > > I think the only argument against what you all have outlined here is if, > > in the future, we have some intent to create new implementations of > HRegion > > or HStore. If that's the case, Region could still be the CP's "view" of > > what a region is, we could introduce another interface which defines what > > the internal/private "view" of a region is, and then we plug in the > > implementation of choice (e.g. HRegion as we know it now would become > > something like HRegionImpl). > > > > > > Currently our codebase entertains the 'fantasy' that it is possible to plug > in alternate Region and Store implementations; there is a config that > allows you stipulate another Region class. As part of the Duo/Anoop > pare-back of CPs on RegionServer-side, it was noted that we should disabuse > ourselves of all such notions. It just doesn't work. We've not expended the > effort to keep clean Region/Store Interfaces (Witness this note on > confusion around intent of Region Interface for example). There are no > tests. > > > > > However, I'm struggling to come up with a concrete use-case as to why we > > would need that extra level of indirection. As such, I think the below > > proposal makes sense. > > > > > Yeah. We could do the work to backfill an 'HRegion Interface', but years > later, there have been no takers. It'd be a fun project for sure but would > have to have good justification (Justification would have to include why > use HBase at all and not something like Apache Helix instead). > > Good stuff, > S > > > > > > > The distillation of a tricky issue is quite appreciated! > > > > > > On 10/4/17 6:51 PM, Andrew Purtell wrote: > > > >> I think it is fine to rebrand these interfaces as for coprocessors and > tag > >> them LP(COPROC): > >> > >> Region (use HRegion in internals) > >> > >> Store (use HStore in internals) > >> > >> MasterServices (use HMaster in internals) > >> > >> RegionServerServices (use HRegionServer in internals) > >> > >> and pare them down. This is not a complete list, just a handful of > >> examples. > >> > >> Some specific points of feedback I have had: > >> > >> In Region, it would be good for CPs to be able to schedule async flushes > >> and compactions, poll or wait for completion of a specific request, or > >> wait > >> for all pending flushes and compactions to complete. There is a Phoenix > >> use > >> case for this in indexing. > >> > >> Security coprocessors use MasterServices to create their system tables. > >> Maybe this can be replaced by using the normal admin API for same. > >> > >> When removing access to internal services, consider if there are client > >> API > >> equivalents that the CP can use, and if embedded calls to such client > APIs > >> from the coprocessor context would be a good idea. CP invocation of an > >> internal service is simply an in-process method call. That's good and > bad, > >> right? The bad part, direct access, is the thing we want to restrict, > the > >> motivation for this work (in part). But the good thing is it avoids a > lot > >> of the fat client logic unnecessary for all-in-process service > invocation, > >> which might even not work correctly. Removing everything is as drastic > as > >> allowing CPs access to everything. It could be fine to drastically pare > >> down, but please consider it. > >> > >> Some changes have been proposed that removes access to metrics (e.g. > >> RegionMetrics, MasterMetrics). Right now coprocessors can bypass core > >> function and replace it. Until and unless we remove the bypass semantic > >> (under discussion) we should continue to allow CPs access to metrics > >> objects so they can update metrics as expected by admins and users when > >> replacing functionality (via bypass). Metrics are a public facing API. I > >> agree this is kind of dodgy. I believe we should remove the bypass > >> semantic. Once that is done, coprocessors can only mix in additional > >> functionality. No more cause to touch core metrics. They can export > their > >> own metrics if so desired. > >> > >> > >> On Wed, Oct 4, 2017 at 3:15 PM, Stack wrote: > >> > >> > >> A bunch of us are making good progress on the next alpha release, > >> > >>> hbase-2.0.0-alpha-4. The theme for this release is "Fixing the > >>> Coprocessor > >>> API", mostly undoin
Re: If possible read families from tables and (more important) qualifiers?
On Thu, Oct 5, 2017 at 8:57 AM, Ted Yu wrote: > See Enis' comment: > > https://issues.apache.org/jira/browse/HBASE-14850? > focusedCommentId=15840799&page=com.atlassian.jira. > plugin.system.issuetabpanels:comment-tabpanel#comment-15840799 > > though it was half year old. > > That is not a status by my reading. That is an overview of how the client is implemented. A state of C++ cliient would be appreciated; what is to done, what works, when we think it will be done. Thanks, S > On Thu, Oct 5, 2017 at 8:53 AM, Sean Busbey wrote: > > > I believe there is an umbrella jira tracking the progress of the > > feature on its branch. I haven't kept up with development enough to know > if > > it's accurate and/or if things have reached the stage of explicitly > > documenting things. > > > > @Ted Y might be able to provide a pointer? > > > > On Thu, Oct 5, 2017 at 2:03 AM, Andrzej wrote: > > > W dniu 05.10.2017 o 08:01, Sean Busbey pisze: > > >> > > >> Presuming this is the using the C++ client that is under active > > >> development and not yet in a release, please limit discussion of it to > > >> the dev@hbase list. > > >> > > >> (I have sent this to the dev list and moved the user@ list to bcc) > > > > > > > > > Is anywhere features of native client?, what is working, what yet no. > > > I want use this client because is very fast, whereas in my tests, > Thrift > > is > > > slow. > > >
Re: Looking for input on an alpha-4 thorny item
On Thu, Oct 5, 2017 at 9:30 AM, Josh Elser wrote: > (I think I understand the problems enough to comment, but, admittedly, my > 5minute read is probably lacking) > > Thanks for chiming-in Josh. > I think the only argument against what you all have outlined here is if, > in the future, we have some intent to create new implementations of HRegion > or HStore. If that's the case, Region could still be the CP's "view" of > what a region is, we could introduce another interface which defines what > the internal/private "view" of a region is, and then we plug in the > implementation of choice (e.g. HRegion as we know it now would become > something like HRegionImpl). > > Currently our codebase entertains the 'fantasy' that it is possible to plug in alternate Region and Store implementations; there is a config that allows you stipulate another Region class. As part of the Duo/Anoop pare-back of CPs on RegionServer-side, it was noted that we should disabuse ourselves of all such notions. It just doesn't work. We've not expended the effort to keep clean Region/Store Interfaces (Witness this note on confusion around intent of Region Interface for example). There are no tests. > However, I'm struggling to come up with a concrete use-case as to why we > would need that extra level of indirection. As such, I think the below > proposal makes sense. > > Yeah. We could do the work to backfill an 'HRegion Interface', but years later, there have been no takers. It'd be a fun project for sure but would have to have good justification (Justification would have to include why use HBase at all and not something like Apache Helix instead). Good stuff, S > The distillation of a tricky issue is quite appreciated! > > > On 10/4/17 6:51 PM, Andrew Purtell wrote: > >> I think it is fine to rebrand these interfaces as for coprocessors and tag >> them LP(COPROC): >> >> Region (use HRegion in internals) >> >> Store (use HStore in internals) >> >> MasterServices (use HMaster in internals) >> >> RegionServerServices (use HRegionServer in internals) >> >> and pare them down. This is not a complete list, just a handful of >> examples. >> >> Some specific points of feedback I have had: >> >> In Region, it would be good for CPs to be able to schedule async flushes >> and compactions, poll or wait for completion of a specific request, or >> wait >> for all pending flushes and compactions to complete. There is a Phoenix >> use >> case for this in indexing. >> >> Security coprocessors use MasterServices to create their system tables. >> Maybe this can be replaced by using the normal admin API for same. >> >> When removing access to internal services, consider if there are client >> API >> equivalents that the CP can use, and if embedded calls to such client APIs >> from the coprocessor context would be a good idea. CP invocation of an >> internal service is simply an in-process method call. That's good and bad, >> right? The bad part, direct access, is the thing we want to restrict, the >> motivation for this work (in part). But the good thing is it avoids a lot >> of the fat client logic unnecessary for all-in-process service invocation, >> which might even not work correctly. Removing everything is as drastic as >> allowing CPs access to everything. It could be fine to drastically pare >> down, but please consider it. >> >> Some changes have been proposed that removes access to metrics (e.g. >> RegionMetrics, MasterMetrics). Right now coprocessors can bypass core >> function and replace it. Until and unless we remove the bypass semantic >> (under discussion) we should continue to allow CPs access to metrics >> objects so they can update metrics as expected by admins and users when >> replacing functionality (via bypass). Metrics are a public facing API. I >> agree this is kind of dodgy. I believe we should remove the bypass >> semantic. Once that is done, coprocessors can only mix in additional >> functionality. No more cause to touch core metrics. They can export their >> own metrics if so desired. >> >> >> On Wed, Oct 4, 2017 at 3:15 PM, Stack wrote: >> >> >> A bunch of us are making good progress on the next alpha release, >> >>> hbase-2.0.0-alpha-4. The theme for this release is "Fixing the >>> Coprocessor >>> API", mostly undoing access accidentally granted Coprocessors. I'm >>> talking >>> out loud about a particularly awkward item here rather than in a comment >>> up >>> in JIRA so the airing sees a broader audience. Interested in any opinions >>> or input you might have. >>> >>> TL;DR MasterService/RegionServerService and Region, etc., Interfaces >>> were >>> overloaded serving two, disparate roles; a load of refactoring has to be >>> done to undo the damage. Suggestions for how to avoid making same mistake >>> in future? >>> >>> I'm working on "HBASE-12260 MasterServices - remove from coprocessor API >>> (Discuss)". MasterServices started out as a subset of the Master >>> functionality. The id
Re: Looking for input on an alpha-4 thorny item
On Wed, Oct 4, 2017 at 3:51 PM, Andrew Purtell wrote: > I think it is fine to rebrand these interfaces as for coprocessors and tag > them LP(COPROC): > > Region (use HRegion in internals) > > Store (use HStore in internals) > > MasterServices (use HMaster in internals) > > RegionServerServices (use HRegionServer in internals) > > and pare them down. This is not a complete list, just a handful of > examples. > > Will do. Just to say that it means a bunch of refactoring undoing reliance on Interfaces (We have too many services and managers!) > Some specific points of feedback I have had: > > In Region, it would be good for CPs to be able to schedule async flushes > and compactions, poll or wait for completion of a specific request, or wait > for all pending flushes and compactions to complete. There is a Phoenix use > case for this in indexing. > > The means are in place. Will make sure it happens. > Security coprocessors use MasterServices to create their system tables. > Maybe this can be replaced by using the normal admin API for same. > > Yes. Working on this too. > When removing access to internal services, consider if there are client API > equivalents that the CP can use, and if embedded calls to such client APIs > from the coprocessor context would be a good idea. CP invocation of an > internal service is simply an in-process method call. That's good and bad, > right? The bad part, direct access, is the thing we want to restrict, the > motivation for this work (in part). But the good thing is it avoids a lot > of the fat client logic unnecessary for all-in-process service invocation, > which might even not work correctly. Removing everything is as drastic as > allowing CPs access to everything. It could be fine to drastically pare > down, but please consider it. > > The short-circuit of RPC (and PB) we have where when a Client is on the target host, the invocation is direct helps here. Moving the RegionServerGroup feature over to the new cut-down Coprocessor API has turned up some holes in our Admin Interface -- i.e. there is no getServers method unless you go via ClusterReport -- that I'm backfilling so RSGroups can use Admin Interface instead of now-removed CP API. But also, have had to add back a few items to MasterServices in a more constrained form. > Some changes have been proposed that removes access to metrics (e.g. > RegionMetrics, MasterMetrics). Right now coprocessors can bypass core > function and replace it. Until and unless we remove the bypass semantic > (under discussion) we should continue to allow CPs access to metrics > objects so they can update metrics as expected by admins and users when > replacing functionality (via bypass). Metrics are a public facing API. I > agree this is kind of dodgy. I believe we should remove the bypass > semantic. Once that is done, coprocessors can only mix in additional > functionality. No more cause to touch core metrics. They can export their > own metrics if so desired. > > > Will be back to ping you on this one. Yeah, I think it dodgy CPs updating internal metrics. Will look at the radical removal of bypass too... Will be back. Thanks Andrew, S > On Wed, Oct 4, 2017 at 3:15 PM, Stack wrote: > > > A bunch of us are making good progress on the next alpha release, > > hbase-2.0.0-alpha-4. The theme for this release is "Fixing the > Coprocessor > > API", mostly undoing access accidentally granted Coprocessors. I'm > talking > > out loud about a particularly awkward item here rather than in a comment > up > > in JIRA so the airing sees a broader audience. Interested in any opinions > > or input you might have. > > > > TL;DR MasterService/RegionServerService and Region, etc., Interfaces > were > > overloaded serving two, disparate roles; a load of refactoring has to be > > done to undo the damage. Suggestions for how to avoid making same mistake > > in future? > > > > I'm working on "HBASE-12260 MasterServices - remove from coprocessor API > > (Discuss)". MasterServices started out as a subset of the Master > > functionality. The idea back then was that certain Services and Managers > > could make do w/ less-than-full-access to the HMaster process. If so, we > > could test the Service and Manager without having to standup a full > HMaster > > instance (This usually required our putting up a cluster too). If > > MasterServices had but a few methods, a Mock would be easy, making > testing > > easier still. We did the same thing on the RegionServer side where we had > > RegionServerServices. > > > > MasterServices (and RegionServerServices) were also exposed to > > Coprocessors. The idea was that CPs could ask-for particular Master > > functions via MasterService. In this role MasterServices was meant to > > constrain what CPs had access to. > > > > MasterServices therefore had two consumers; one internal, the other not. > > > > With time, MasterServices got fat as internal Services and Managers > needed > > more of
[jira] [Created] (HBASE-18957) add test that establishes branch-1 behavior for filterlist w/OR
Sean Busbey created HBASE-18957: --- Summary: add test that establishes branch-1 behavior for filterlist w/OR Key: HBASE-18957 URL: https://issues.apache.org/jira/browse/HBASE-18957 Project: HBase Issue Type: Sub-task Components: Filters Reporter: Sean Busbey Assignee: Peter Somogyi Priority: Critical we need a test that shows the expected behavior for filter lists that rely on OR prior to our filterlist improvements so we have a baseline to show compatibility (and/or document incompatibilities that end up being introduced). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18956) ruby-lint report "undefined method java_import"
Guanghao Zhang created HBASE-18956:
--
Summary: ruby-lint report "undefined method java_import"
Key: HBASE-18956
URL: https://issues.apache.org/jira/browse/HBASE-18956
Project: HBase
Issue Type: Sub-task
Reporter: Guanghao Zhang
In HBASE-18909, I add a new java_import for admin.rb. But the ruby-lint report
"undefined method java_import".
{code}
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:22:13: undefined
method java
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:23:1: undefined
method java_import
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:49:29: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:273:27: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:287:17: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:317:17: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1009:30: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1015:35: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1016:9: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1022:28: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1028:33: undefined
constant Pattern
/testptch/hbase/hbase-shell/src/main/ruby/hbase/admin.rb:E:1029:9: undefined
constant Pattern
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
