Re: Custom filter appears to just hang
Do you know of any third party filters that are posted somewhere where I could do that? On Mon, Sep 16, 2019 at 5:55 PM Stack wrote: > On Sun, Sep 15, 2019 at 9:35 AM Mike Thomsen > wrote: > > > So that's just it, I don't see any errors at all on either the client > side > > or the server side. I'm running this against HBase 2.0.5, and I didn't > see > > anything showing up. Granted, I'm running server in local mode with just > > "bin/hbase master start" and letting it manage ZK and create one > > regionserver that way. As I said, I can read and write data just fine > with > > this setup until I throw the filter into the mix and then it mysteriously > > hangs. > > > > I've uploaded the whole project here, including the protobuf section: > > > > https://github.com/MikeThomsen/hbase-test > > > > Thanks, > > > > > This stuff can be tricky to dev. If mismatch in rpc or error formulating > response, the exceptions can be dropped and not make it out in log or on > client-side. Can you printf your filter code? Or use an existing working > one and slowly pull it over to do as you want. > S > > > > > Mike > > > > On Fri, Sep 13, 2019 at 12:33 PM Esteban Gutierrez > > wrote: > > > > > Hi Mike, > > > > > > What errors do you see in the RegionServer or Client logs? Did you > > > generated the Java class files for SimpleFilter from a proto file? The > > > HBase Definitive Guide as a good example about how to write and deploy > > > custom filters: > > > > > > > > > https://www.oreilly.com/library/view/hbase-the-definitive/9781492024255/ch04.html > > > see Example 4-23 and Example 4-24. > > > > > > But depending on the version of HBase you are using you might have to > > also > > > take a look a the Protobuf section in the HBase Reference Guide to see > > > where you should create the proto files and which jar should be used > > > (shaded vs non-shaded) : https://hbase.apache.org/book.html#protobuf > > > > > > thanks, > > > esteban. > > > > > > -- > > > Cloudera, Inc. > > > > > > > > > > > > On Fri, Sep 13, 2019 at 6:54 AM Mike Thomsen > > > wrote: > > > > > > > https://gist.github.com/MikeThomsen/ed3e742d13adeb63d45ceca7a09bf176 > > > > > > > > I'm running this in standalone mode, with the jar file in the HBase > lib > > > > folder. When I run the client side of it, it gets to the print > > statement > > > > that says Fetching and then just hangs. When I take the filter > off > > > the > > > > Get, it pulls the row just fine. Can someone give me some help on why > > it > > > > might be hanging? > > > > > > > > Thanks, > > > > > > > > Mike > > > > > > > > > >
Re: Custom filter appears to just hang
So that's just it, I don't see any errors at all on either the client side or the server side. I'm running this against HBase 2.0.5, and I didn't see anything showing up. Granted, I'm running server in local mode with just "bin/hbase master start" and letting it manage ZK and create one regionserver that way. As I said, I can read and write data just fine with this setup until I throw the filter into the mix and then it mysteriously hangs. I've uploaded the whole project here, including the protobuf section: https://github.com/MikeThomsen/hbase-test Thanks, Mike On Fri, Sep 13, 2019 at 12:33 PM Esteban Gutierrez wrote: > Hi Mike, > > What errors do you see in the RegionServer or Client logs? Did you > generated the Java class files for SimpleFilter from a proto file? The > HBase Definitive Guide as a good example about how to write and deploy > custom filters: > > https://www.oreilly.com/library/view/hbase-the-definitive/9781492024255/ch04.html > see Example 4-23 and Example 4-24. > > But depending on the version of HBase you are using you might have to also > take a look a the Protobuf section in the HBase Reference Guide to see > where you should create the proto files and which jar should be used > (shaded vs non-shaded) : https://hbase.apache.org/book.html#protobuf > > thanks, > esteban. > > -- > Cloudera, Inc. > > > > On Fri, Sep 13, 2019 at 6:54 AM Mike Thomsen > wrote: > > > https://gist.github.com/MikeThomsen/ed3e742d13adeb63d45ceca7a09bf176 > > > > I'm running this in standalone mode, with the jar file in the HBase lib > > folder. When I run the client side of it, it gets to the print statement > > that says Fetching and then just hangs. When I take the filter off > the > > Get, it pulls the row just fine. Can someone give me some help on why it > > might be hanging? > > > > Thanks, > > > > Mike > > >
Custom filter appears to just hang
https://gist.github.com/MikeThomsen/ed3e742d13adeb63d45ceca7a09bf176 I'm running this in standalone mode, with the jar file in the HBase lib folder. When I run the client side of it, it gets to the print statement that says Fetching and then just hangs. When I take the filter off the Get, it pulls the row just fine. Can someone give me some help on why it might be hanging? Thanks, Mike
Re: Sorting keys with HFile.Writer
Thanks. Is there any value in me specifying the comparator?
On Tue, Jan 1, 2019 at 9:07 AM 张铎(Duo Zhang) wrote:
> Yes, the keys should be sorted, before passing them to HFile.Writer. The
> way we build index for HFile is based on this assumption, that the keys are
> sorted.
>
> Mike Thomsen 于2019年1月1日周二 下午9:36写道：
>
> > I took some of the code from the JUnit test for setting up a HFile writer
> > and it looked like it should work.
> >
> > Path f = new Path("/", "test-something.hfile");
> > HFileContext context = new HFileContextBuilder()
> > .withBlockSize(4096)
> > .withIncludesTags(true)
> > .withCompression(Compression.Algorithm.NONE).build();
> > HFile.Writer w = HFile.getWriterFactory(conf, cacheConf)
> > .withPath(FileSystem.get(conf), f)
> > .withFileContext(context)
> > .withComparator(KeyValue.COMPARATOR)
> > .create();
> > //KeyValue(byte[] row, byte[] family, byte[] qualifier, long timestamp,
> > byte[] value)
> > Map records = new HashMap();
> > for (int x = 0; x < 1; x++) {
> > String uuid = UUID.randomUUID().toString();
> > KeyValue value = new KeyValue(uuid.getBytes(), "some-fam".getBytes(),
> > "test".getBytes(), System.currentTimeMillis(), "hi".getBytes());
> > w.append(value);
> > }
> >
> > That threw this exception:
> >
> > java.io.IOException: Added a key not lexically larger than previous.
> > Current cell =
> >
> >
> 8faf82ba-5fed-4332-9731-0ebf4d4494f9/some-fam:test/1546348994160/Put/vlen=2/seqid=0,
> > lastCell =
> >
> >
> e32ec727-d946-4f67-a3c6-315a27c76408/some-fam:test/1546348994160/Put/vlen=2/seqid=0
> > at
> > org.apache.hadoop.hbase.io
> > .hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
> > at
> > org.apache.hadoop.hbase.io
> > .hfile.HFileWriterV2.append(HFileWriterV2.java:267)
> > at
> > org.apache.hadoop.hbase.io
> > .hfile.HFileWriterV3.append(HFileWriterV3.java:87)
> > at HFileTest.main(HFileTest.java:39)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > When I manually sorted the keys, it worked fine for creating and closing
> > the new HFile. Did I miss something or is this expected behavior from the
> > writer? I'm testing this against HBase 1.2.9.
> >
> > Thanks,
> >
> > Mike
> >
>
Sorting keys with HFile.Writer
I took some of the code from the JUnit test for setting up a HFile writer
and it looked like it should work.
Path f = new Path("/", "test-something.hfile");
HFileContext context = new HFileContextBuilder()
.withBlockSize(4096)
.withIncludesTags(true)
.withCompression(Compression.Algorithm.NONE).build();
HFile.Writer w = HFile.getWriterFactory(conf, cacheConf)
.withPath(FileSystem.get(conf), f)
.withFileContext(context)
.withComparator(KeyValue.COMPARATOR)
.create();
//KeyValue(byte[] row, byte[] family, byte[] qualifier, long timestamp,
byte[] value)
Map records = new HashMap();
for (int x = 0; x < 1; x++) {
String uuid = UUID.randomUUID().toString();
KeyValue value = new KeyValue(uuid.getBytes(), "some-fam".getBytes(),
"test".getBytes(), System.currentTimeMillis(), "hi".getBytes());
w.append(value);
}
That threw this exception:
java.io.IOException: Added a key not lexically larger than previous.
Current cell =
8faf82ba-5fed-4332-9731-0ebf4d4494f9/some-fam:test/1546348994160/Put/vlen=2/seqid=0,
lastCell =
e32ec727-d946-4f67-a3c6-315a27c76408/some-fam:test/1546348994160/Put/vlen=2/seqid=0
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:267)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at HFileTest.main(HFileTest.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
When I manually sorted the keys, it worked fine for creating and closing
the new HFile. Did I miss something or is this expected behavior from the
writer? I'm testing this against HBase 1.2.9.
Thanks,
Mike
Re: AntsDB is released with MySQL compatibility for HBase
That's my understanding as well, and why so many projects choose ASL or BSD/MIT now. I am not a lawyer, but my understanding is that you also cannot change the license to ASL now, even if you remove all of the GPL'd code, without getting written permission from every contributor to the code base. The good part is that your clients won't have to worry about GPL contamination as long as they can load a driver like Connector/J at runtime. So it's not a show stopper from trying out AntsDB. On Tue, May 8, 2018 at 11:53 AM Water Guo wrote: > I am not an expert with software license. But my gut’s feeling is I can’t > claim Apache if I am using code from MySQL JDBC driver (GPL), can I? > > > On May 8, 2018, at 11:49 AM, Stack wrote: > > > > On Mon, May 7, 2018 at 11:58 AM, Water Guo wrote: > > > >> Not it is not. AntsDB has no dependence on MySQL code. It is written in > >> Java and uses a couple of open source libraries. > >> > >> > > Sorry. There seems to be a misunderstanding. I was just asking if why > > antsdb has a GPL license rather than say an Apache one: > > https://github.com/waterguo/antsdb/blob/master/LICENSE.txt > > > > Thanks, > > S > > > > > >>> Sounds great Water. Lets take it for a spin. Quick question, why the > "GNU > >>> Affero General Public License, version 3" Is it up from mysql? Thanks, > >>> S > >> > >> On Mon, May 7, 2018 at 10:09 AM, Water Guo > wrote: > >> > >>> Dear HBase Community, > >>> > >>> I’d like to take this opportunity to introduce my open source project > >>> AntsDB. It is a database virtualization software that brings MySQL > >>> compatibility to HBase. It means you can use any MySQL bindings such as > >>> JDBC, ODBC, PHP, Perl to manipulate data in HBase. It supports most > MySQL > >>> DDLs and all DMLs, transaction control, table locks, row locks etc. Up > to > >>> date applications such as MySQL console, MySQL command lines, > >> BenchmarkSQL, > >>> MediaWiki, SonarQube, DBeaver, SquirrelSQL and many others can run > >> directly > >>> on HBase using AntsDB layer. The project is hosted at > >>> https://github.com/waterguo/antsdb. > >>> > >>> AntsDB is designed to support high concurrency, low latency > applications. > >>> It uses local storage as cache so it can further reduce the latency of > >>> HBase. We have benchmarked AntsDB using YCSB. The result is at > >>> http://www.antsdb.com/?p=171. > >>> > >>> People always ask me how it is different from Phoenix. While Phoenix is > >>> building a powerful SQL layer for HBase, we want to focus on backward > >>> compatibility. We want to have applications built for MySQL can be used > >>> directly on HBase. And people who are familiar with traditional > >> relational > >>> database can adopt HBase/Hadoop stack with ease. > >>> > >>> I’d be very glad if you find the project is useful and your feedback is > >>> very welcome. > >>> > >>> Thanks > >>> -water > >>> > >> > >> > >
Re: Visibility auths only visible to hbase user
It was actually a configuration problem in the client. I had accidentally deleted part of the configuration and didn't realize it. Sorry about the confusion. On Sun, May 6, 2018 at 2:08 PM Ted Yu wrote: > For the open source download, can you tell us which release you downloaded > ? > > Did you install it on the docker image ? > > Please share hbase-site.xml (thru pastebin) if possible. > > Thanks > > On Sun, May 6, 2018 at 8:04 AM, Mike Thomsen > wrote: > > > Ted, > > > > As I mentioned, I tried this with HDP and a copy downloaded from > > hbase.apache.org which is why I'm scratching my head on this... > > > > Thanks, > > > > Mike > > > > On Sun, May 6, 2018 at 10:58 AM Ted Yu wrote: > > > > > Please use vendor forum for vendor specific question(s). > > > > > > To my knowledge, this feature works in Apache hbase releases. > > > > > > Cheers > > > > > > On Sun, May 6, 2018 at 7:55 AM, Mike Thomsen > > > wrote: > > > > > > > I've tried this in the HDP docker sandbox and outside that with a > basic > > > > installation of HBase. > > > > > > > > su - hbase > > > > hbase shell > > > > set_auths 'test_user', ['OPEN', 'BILLING', 'PII'] > > > > get_auths 'test_user' > > > > OPEN > > > > BILLING > > > > PII > > > > > > > > So far so good. > > > > > > > > su - test_user > > > > hbase shell > > > > get_auths 'test_user' > > > > 0 returned > > > > > > > > If su back to hbase and check, that user reports OPEN, BILLING and > PII > > > are > > > > set for test_user. > > > > > > > > Weird thing is I had it working under a previous iteration of running > > the > > > > HDP sandbox, but it won't work in either environment now. > > > > > > > > Any ideas? > > > > > > > > Thanks, > > > > > > > > Mike > > > > > > > > > >
Re: Visibility auths only visible to hbase user
Ted, As I mentioned, I tried this with HDP and a copy downloaded from hbase.apache.org which is why I'm scratching my head on this... Thanks, Mike On Sun, May 6, 2018 at 10:58 AM Ted Yu wrote: > Please use vendor forum for vendor specific question(s). > > To my knowledge, this feature works in Apache hbase releases. > > Cheers > > On Sun, May 6, 2018 at 7:55 AM, Mike Thomsen > wrote: > > > I've tried this in the HDP docker sandbox and outside that with a basic > > installation of HBase. > > > > su - hbase > > hbase shell > > set_auths 'test_user', ['OPEN', 'BILLING', 'PII'] > > get_auths 'test_user' > > OPEN > > BILLING > > PII > > > > So far so good. > > > > su - test_user > > hbase shell > > get_auths 'test_user' > > 0 returned > > > > If su back to hbase and check, that user reports OPEN, BILLING and PII > are > > set for test_user. > > > > Weird thing is I had it working under a previous iteration of running the > > HDP sandbox, but it won't work in either environment now. > > > > Any ideas? > > > > Thanks, > > > > Mike > > >
Visibility auths only visible to hbase user
I've tried this in the HDP docker sandbox and outside that with a basic installation of HBase. su - hbase hbase shell set_auths 'test_user', ['OPEN', 'BILLING', 'PII'] get_auths 'test_user' OPEN BILLING PII So far so good. su - test_user hbase shell get_auths 'test_user' 0 returned If su back to hbase and check, that user reports OPEN, BILLING and PII are set for test_user. Weird thing is I had it working under a previous iteration of running the HDP sandbox, but it won't work in either environment now. Any ideas? Thanks, Mike
Re: New installation of HBase ignoring cell visibility (even though it's active)
Ted/Anoop
https://issues.apache.org/jira/browse/HBASE-20080
On Sat, Feb 24, 2018 at 12:12 PM, Ted Yu wrote:
> bq. a warning message in the shell should be displayed if simple auth and
> cell visibility are in use together.
>
> Makes sense.
>
> Please log a JIRA.
>
> On Sat, Feb 24, 2018 at 9:06 AM, Mike Thomsen
> wrote:
>
> > Ted/Anoop,
> >
> > I realized what the problem was. When I installed HBase previously (and
> had
> > this working) it was on a Linux machine with a package that created the
> > hbase superuser and loaded the services with that. I was starting HBase
> on
> > the Mac my company just gave me and was using my user account. Ergo, I
> was
> > scanning as the superuser...
> >
> > Sorry about that. I'd suggest for newbs like me that a warning message in
> > the shell should be displayed if simple auth and cell visibility are in
> use
> > together.
> >
> > BTW, the reason I have been trying to get this work is that I'm working
> on
> > a patch for NiFi to integrate visibility label support.
> >
> > Thanks,
> >
> > Mike
> >
> > On Sat, Feb 24, 2018 at 10:51 AM, Ted Yu wrote:
> >
> > > I noted that SIMPLE_AUTHENTICATION was returned.
> > > Here is related code for getSecurityCapabilities():
> > >
> > > if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
> > >
> > > capabilities.add(SecurityCapabilitiesResponse.Capability.
> > > SECURE_AUTHENTICATION);
> > >
> > > } else {
> > >
> > > capabilities.add(SecurityCapabilitiesResponse.Capability.
> > > SIMPLE_AUTHENTICATION);
> > >
> > > }
> > > Did "hbase.security.authentication" have value of "kerberos" ?
> > >
> > > If it does, please pastebin your hbase-site.xml
> > >
> > > Thanks
> > >
> > > On Sat, Feb 24, 2018 at 4:59 AM, Mike Thomsen
> > > wrote:
> > >
> > > > (hbase-site.xml is attached)
> > > >
> > > > I reinstalled HBase on my development machine and the console shows
> > that
> > > > it's just ignoring the cell visibility settings even though it shows
> > > > they're active:
> > > >
> > > > hbase(main):001:0> list
> > > > TABLE
> > > >
> > > >
> > > >
> > > > 0 row(s) in 0.1630 seconds
> > > >
> > > > => []
> > > > hbase(main):002:0> get_auths "michaelthomsen"
> > > > 0 row(s) in 10.1940 seconds
> > > >
> > > > hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> > > > 0 row(s) in 0.1900 seconds
> > > >
> > > > hbase(main):004:0> set_auths "michaelthomsen", ["U", "PII", "PHI"]
> > > > 0 row(s) in 0.0500 seconds
> > > >
> > > > hbase(main):005:0> get_auths "michaelthomsen"
> > > > U
> > > >
> > > >
> > > >
> > > > PII
> > > >
> > > >
> > > >
> > > > PHI
> > > >
> > > >
> > > >
> > > > 0 row(s) in 0.0470 seconds
> > > >
> > > > hbase(main):006:0> create 'test', {NAME => 'prop' }
> > > > 0 row(s) in 1.2780 seconds
> > > >
> > > > => Hbase::Table - test
> > > > hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> > > > 0 row(s) in 0.0500 seconds
> > > >
> > > > hbase(main):008:0> scan "test"
> > > > ROW
> > > > COLUMN+CELL
> > > >
> > > >
> > > > user1 column=prop:name,
> > > > timestamp=1519476818510, value=John Smith
> > > >
> > > > 1 row(s) in 0.0260 seconds
> > > >
> > > > hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop"
> }
> > > > 1 row(s) in 0.0130 seconds
> > > >
> > > > hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> > > > ROW
> > > > COLUMN+CELL
> > > >
> > > >
> > > > user1 column=prop:name,
> > > > timestamp=1519476818510, value=John Smith
> > > >
> > > > 1 row(s) in 0.0180 seconds
> > > >
> > > > hbase(main):011:0> list_security_capabilities
> > > > SIMPLE_AUTHENTICATION
> > > > CELL_VISIBILITY
> > > >
> > > > => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> > > > hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> > > > ROW
> > > > COLUMN+CELL
> > > >
> > > >
> > > > user1 column=prop:name,
> > > > timestamp=1519476818510, value=John Smith
> > > >
> > > > 1 row(s) in 0.0060 seconds
> > > >
> > > >
> > > > I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> > > > sense because it's allowing me to assign authorizations to my simple
> > auth
> > > > user, set_visibility runs successfully and it's reporting that cell
> > > > visibility is enabled.
> > > >
> > > >
> > >
> >
>
Incrementally load HFiles outside of MR/Spark
I'm looking into creating HFiles directly from NiFi using the HBase API. It seems pretty straight forward: 1. Open a HFile.Writer pointing to a file path in HDFS. 2. Write the cells with the HFile API. 3. Call the incremental loader API to have it tell HBase to load the generated segments. Is that right? If so, are there any gotchas that I should be aware of? Thanks, Mike
Re: New installation of HBase ignoring cell visibility (even though it's active)
Ted/Anoop,
I realized what the problem was. When I installed HBase previously (and had
this working) it was on a Linux machine with a package that created the
hbase superuser and loaded the services with that. I was starting HBase on
the Mac my company just gave me and was using my user account. Ergo, I was
scanning as the superuser...
Sorry about that. I'd suggest for newbs like me that a warning message in
the shell should be displayed if simple auth and cell visibility are in use
together.
BTW, the reason I have been trying to get this work is that I'm working on
a patch for NiFi to integrate visibility label support.
Thanks,
Mike
On Sat, Feb 24, 2018 at 10:51 AM, Ted Yu wrote:
> I noted that SIMPLE_AUTHENTICATION was returned.
> Here is related code for getSecurityCapabilities():
>
> if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
>
> capabilities.add(SecurityCapabilitiesResponse.Capability.
> SECURE_AUTHENTICATION);
>
> } else {
>
> capabilities.add(SecurityCapabilitiesResponse.Capability.
> SIMPLE_AUTHENTICATION);
>
> }
> Did "hbase.security.authentication" have value of "kerberos" ?
>
> If it does, please pastebin your hbase-site.xml
>
> Thanks
>
> On Sat, Feb 24, 2018 at 4:59 AM, Mike Thomsen
> wrote:
>
> > (hbase-site.xml is attached)
> >
> > I reinstalled HBase on my development machine and the console shows that
> > it's just ignoring the cell visibility settings even though it shows
> > they're active:
> >
> > hbase(main):001:0> list
> > TABLE
> >
> >
> >
> > 0 row(s) in 0.1630 seconds
> >
> > => []
> > hbase(main):002:0> get_auths "michaelthomsen"
> > 0 row(s) in 10.1940 seconds
> >
> > hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> > 0 row(s) in 0.1900 seconds
> >
> > hbase(main):004:0> set_auths "michaelthomsen", ["U", "PII", "PHI"]
> > 0 row(s) in 0.0500 seconds
> >
> > hbase(main):005:0> get_auths "michaelthomsen"
> > U
> >
> >
> >
> > PII
> >
> >
> >
> > PHI
> >
> >
> >
> > 0 row(s) in 0.0470 seconds
> >
> > hbase(main):006:0> create 'test', {NAME => 'prop' }
> > 0 row(s) in 1.2780 seconds
> >
> > => Hbase::Table - test
> > hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> > 0 row(s) in 0.0500 seconds
> >
> > hbase(main):008:0> scan "test"
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0260 seconds
> >
> > hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
> > 1 row(s) in 0.0130 seconds
> >
> > hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0180 seconds
> >
> > hbase(main):011:0> list_security_capabilities
> > SIMPLE_AUTHENTICATION
> > CELL_VISIBILITY
> >
> > => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> > hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0060 seconds
> >
> >
> > I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> > sense because it's allowing me to assign authorizations to my simple auth
> > user, set_visibility runs successfully and it's reporting that cell
> > visibility is enabled.
> >
> >
>
Re: New installation of HBase ignoring cell visibility (even though it's active)
Hi Anoop,
Looks like it's running as michaelthomsen:
hbase(main):013:0> whoami
michaelthomsen (auth:SIMPLE)
groups: staff, everyone, localaccounts, _appserverusr, admin,
_appserveradm, _lpadmin, com.apple.sharepoint.group.2, _appstore,
_lpoperator, _developer, com.apple.access_ftp,
com.apple.access_screensharing, com.apple.access_ssh,
com.apple.sharepoint.group.1
On Sat, Feb 24, 2018 at 9:31 AM, Anoop John wrote:
> As which user you are running the scan shell command? It has to be as
> michaelthomasen as u have set auths to this uname. Looks like the user
> running the command is the hbase super user who started the RS process.
> Then all cells will be returned irrespective of its visibility and scan
> auths.
>
> Anoop
>
> On Saturday, February 24, 2018, Mike Thomsen
> wrote:
>
> > (hbase-site.xml is attached)
> >
> > I reinstalled HBase on my development machine and the console shows that
> > it's just ignoring the cell visibility settings even though it shows
> > they're active:
> >
> > hbase(main):001:0> list
> > TABLE
> >
> >
> >
> > 0 row(s) in 0.1630 seconds
> >
> > => []
> > hbase(main):002:0> get_auths "michaelthomsen"
> > 0 row(s) in 10.1940 seconds
> >
> > hbase(main):003:0> add_labels ["U", "PII", "PHI"]
> > 0 row(s) in 0.1900 seconds
> >
> > hbase(main):004:0> set_auths "michaelthomsen", ["U", "PII", "PHI"]
> > 0 row(s) in 0.0500 seconds
> >
> > hbase(main):005:0> get_auths "michaelthomsen"
> > U
> >
> >
> >
> > PII
> >
> >
> >
> > PHI
> >
> >
> >
> > 0 row(s) in 0.0470 seconds
> >
> > hbase(main):006:0> create 'test', {NAME => 'prop' }
> > 0 row(s) in 1.2780 seconds
> >
> > => Hbase::Table - test
> > hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
> > 0 row(s) in 0.0500 seconds
> >
> > hbase(main):008:0> scan "test"
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0260 seconds
> >
> > hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
> > 1 row(s) in 0.0130 seconds
> >
> > hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0180 seconds
> >
> > hbase(main):011:0> list_security_capabilities
> > SIMPLE_AUTHENTICATION
> > CELL_VISIBILITY
> >
> > => ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
> > hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
> > ROW
> > COLUMN+CELL
> >
> >
> > user1 column=prop:name,
> > timestamp=1519476818510, value=John Smith
> >
> > 1 row(s) in 0.0060 seconds
> >
> >
> > I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no
> > sense because it's allowing me to assign authorizations to my simple auth
> > user, set_visibility runs successfully and it's reporting that cell
> > visibility is enabled.
> >
> >
>
New installation of HBase ignoring cell visibility (even though it's active)
(hbase-site.xml is attached)
I reinstalled HBase on my development machine and the console shows that
it's just ignoring the cell visibility settings even though it shows
they're active:
hbase(main):001:0> list
TABLE
0 row(s) in 0.1630 seconds
=> []
hbase(main):002:0> get_auths "michaelthomsen"
0 row(s) in 10.1940 seconds
hbase(main):003:0> add_labels ["U", "PII", "PHI"]
0 row(s) in 0.1900 seconds
hbase(main):004:0> set_auths "michaelthomsen", ["U", "PII", "PHI"]
0 row(s) in 0.0500 seconds
hbase(main):005:0> get_auths "michaelthomsen"
U
PII
PHI
0 row(s) in 0.0470 seconds
hbase(main):006:0> create 'test', {NAME => 'prop' }
0 row(s) in 1.2780 seconds
=> Hbase::Table - test
hbase(main):007:0> put "test", "user1", "prop:name", "John Smith"
0 row(s) in 0.0500 seconds
hbase(main):008:0> scan "test"
ROW
COLUMN+CELL
user1 column=prop:name,
timestamp=1519476818510, value=John
Smith
1 row(s) in 0.0260 seconds
hbase(main):009:0> set_visibility "test", "PII", { COLUMNS => "prop" }
1 row(s) in 0.0130 seconds
hbase(main):010:0> scan "test", { AUTHORIZATIONS => ["PHI", "U"] }
ROW
COLUMN+CELL
user1 column=prop:name,
timestamp=1519476818510, value=John
Smith
1 row(s) in 0.0180 seconds
hbase(main):011:0> list_security_capabilities
SIMPLE_AUTHENTICATION
CELL_VISIBILITY
=> ["SIMPLE_AUTHENTICATION", "CELL_VISIBILITY"]
hbase(main):012:0> scan "test", { AUTHORIZATIONS => [] }
ROW
COLUMN+CELL
user1 column=prop:name,
timestamp=1519476818510, value=John
Smith
1 row(s) in 0.0060 seconds
I'm running this on a Mac w/out HDFS. It's HBase 1.3.1. This makes no sense
because it's allowing me to assign authorizations to my simple auth user,
set_visibility runs successfully and it's reporting that cell visibility is
enabled.
hbase.zookeeper.quorum
localhost
hbase.cluster.distributed
true
hbase.rootdir
file:data/hbase
hbase.master.info.bindAddress
localhost
hbase.regionserver.info.bindAddress
localhost
hbase.coprocessor.region.classes
org.apache.hadoop.hbase.security.visibility.VisibilityController
hbase.coprocessor.master.classes
org.apache.hadoop.hbase.security.visibility.VisibilityController
Re: HBaseTestingUtility with visibility labels enabled
Thanks, that got me a little further, but it seems like even though the
security capabilities say that cell visibility is enabled, the scanner
ignores the authorization list I pass. This is the relevant code:
private static HBaseTestingUtility UTILITY;
private static final String[] FULL_AUTHS = new String[] { "PII", "PHI",
"UNRESTRICTED" };
private static final String[] PII_AUTHS = new String[] { "PII",
"UNRESTRICTED" };
private static final String[] PHI_AUTHS = new String[] { "PHI",
"UNRESTRICTED" };
private static final String USER_ID = "john.smith";
private static final String TABLE_NAME = "label_test_table";
private static final String FAM = "prop";
private static final String ROW = "test-row-id";
private static final String DEFAULT_VISIBILITY = "PII|UNRESTRICTED";
private static final String CLIENT_VAL = "hbaseClient";
@BeforeClass
public static void setup() throws Throwable {
UTILITY = new HBaseTestingUtility();///conf);
Configuration conf = UTILITY.getConfiguration();
SecureTestUtil.enableSecurity(conf);
conf.set("hbase.coprocessor.region.classes",
"org.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController");
conf.set("hbase.coprocessor.master.classes",
"org.apache.hadoop.hbase.security.access.AccessController,org.apache.hadoop.hbase.security.visibility.VisibilityController");
UTILITY.startMiniCluster();
UTILITY.waitTableEnabled(LABELS_TABLE_NAME.getName(), 5);
UTILITY.waitTableEnabled(AccessControlLists.ACL_TABLE_NAME.getName(),
5);
List capabilities =
UTILITY.getConnection().getAdmin()
.getSecurityCapabilities();
Assert.assertTrue("CELL_VISIBILITY capability is missing",
capabilities.contains(SecurityCapability.CELL_VISIBILITY));
LOCAL_USER = User.createUserForTesting(conf, USER_ID, new
String[]{});
VisibilityClient.addLabels(UTILITY.getConnection(), FULL_AUTHS);
VisibilityClient.setAuths(UTILITY.getConnection(), new
String[]{"PHI"}, USER_ID);
clientService = new
IntegrationTestClientService(UTILITY.getConnection());
UTILITY.createTable(TABLE_NAME.getBytes(), FAM.getBytes());
}
@Test
public void testGetHBase() throws Exception {
Table table =
UTILITY.getConnection().getTable(TableName.valueOf(TABLE_NAME));
Put put = new Put(ROW.getBytes());
put.addColumn(FAM.getBytes(), "fullName".getBytes(),
"john.smith2".getBytes());
put.setCellVisibility(new CellVisibility("PII&UNRESTRICTED"));
//Also tried PII|UNRESTRICTED
table.put(put);
table.close();
LOCAL_USER.runAs((PrivilegedExceptionAction) () -> {
Scan scan = new Scan();
scan.setAuthorizations(new Authorizations("PHI"));
ResultScanner scanner = table.getScanner(scan);
Result res = scanner.next();
Assert.assertTrue("It was not null", res == null);
return null;
});
}
(A chunk of that is copy pasta from trying to figure out why things weren't
working along the way)
Any ideas?
Thanks,
Mike
On Thu, Feb 22, 2018 at 3:14 PM, Ted Yu wrote:
> labels table is created by VisibilityController#postStartMaster().
>
> You can add the following call in the @BeforeClass method:
>
>
> TEST_UTIL.waitTableEnabled(LABELS_TABLE_NAME.getName(), 5);
>
>
> See TestVisibilityLabelsWithACL for complete example.
>
>
>
> On Thu, Feb 22, 2018 at 12:07 PM, Mike Thomsen
> wrote:
>
> > I'm trying to spin up a mini cluster for integration testing. Can someone
> > give me an idea of what I'm doing wrong?
> >
> > public static void main(String[] args) throws Throwable {
> >
> > Configuration conf =
> > org.apache.hadoop.hbase.HBaseConfiguration.create();
> > conf.set("hbase.coprocessor.region.classes",
> > "org.apache.hadoop.hbase.security.visibility.VisibilityController");
> > conf.set("hbase.coprocessor.master.classes",
> > "org.apache.hadoop.hbase.security.visibility.VisibilityController");
> >
> > utility = new HBaseTestingUtility(conf);
> >
> > utility.startMiniCluster();
> >
> > VisibilityClient.addLabels(utility.getConnection(), new
> String[]{
> > "X", "Y", "Z" });
> > }
> >
> > That results in this:
> >
> > org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
> >
> > Thanks,
> >
> > Mike
> >
>
HBaseTestingUtility with visibility labels enabled
I'm trying to spin up a mini cluster for integration testing. Can someone
give me an idea of what I'm doing wrong?
public static void main(String[] args) throws Throwable {
Configuration conf =
org.apache.hadoop.hbase.HBaseConfiguration.create();
conf.set("hbase.coprocessor.region.classes",
"org.apache.hadoop.hbase.security.visibility.VisibilityController");
conf.set("hbase.coprocessor.master.classes",
"org.apache.hadoop.hbase.security.visibility.VisibilityController");
utility = new HBaseTestingUtility(conf);
utility.startMiniCluster();
VisibilityClient.addLabels(utility.getConnection(), new String[]{
"X", "Y", "Z" });
}
That results in this:
org.apache.hadoop.hbase.TableNotFoundException: hbase:labels
Thanks,
Mike
Re: Deletes with cell visibility labels
Thanks. I had a feeling you were going to say that.
On Thu, Sep 21, 2017 at 12:40 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> So the problem that you are facing here is that you don know what is the
> visibility labels associated with a row that was added during 'PUT'. Now to
> form the delete you are not sure what are the exact labels so that the
> 'PUT' remains masked.
>
> But in terms of a generic use case that is the way you can exactly mask a
> PUT with visibility labels right?
> Say if a row is having a sensitive info then it is marked with PRIVATE &
> SECRET labels, then if you need to remove that row ( so that it is later
> changed to PUBLIC) it is always better you specify the ROW with exact
> labels.
> The current impl is that we mask only those PUTs which matches exactly with
> the deletes visibility labels.
>
> And to answer your question in a simple way
> Since you are not sure what were the labels added for a PUT you need to re
> run the algo that generated the labels and add it with deletes if that
> specific row needs to be masked.
>
> Regards
> Ram
>
> On Thu, Sep 21, 2017 at 9:53 PM, Mike Thomsen
> wrote:
>
> > Yes, I realized my mistake shortly after posting. So my question is how
> do
> > you form a proper delete? Is the expected behavior roughly...
> >
> > 1. Get the row.
> > 2. Rerun the algorithm that computed the visibility label on the row.
> > 3. Build a list of deletes.
> >
> > Is that what we're expected to do here? Or is there a simpler way of
> > handling this?
> >
> > Thanks,
> >
> > Mike
> >
> > On Thu, Sep 21, 2017 at 12:18 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Hi Thomson
> > >
> > > I think you are saying that the shell allows you to specify
> > > delete 'tablename', 'row', 'family', ts1,
> {VISIBILITY=>'PRIVATE|SECRET'}
> > > but the java client does not allow to do it? I doubt it.
> > >
> > > In case of mutations like puts and deletes what we pass is the
> > visibility
> > > labels. Now when you do a scan that is where we specify the
> AUTHORIZTIONs
> > > so that only those cells with visibility cells as passed in the
> > > AUTHORIZATIONS are returned back.
> > >
> > > Hope you find this useful. Let us know if you need further inputs.
> > >
> > > Regards
> > > Ram
> > >
> > > On Thu, Sep 21, 2017 at 6:04 PM, Mike Thomsen
> > > wrote:
> > >
> > > > According to the javadocs and some examples I've seen, it looks like
> > with
> > > > the Java client you have to know the visibility label of the cell you
> > > want
> > > > to delete. You cannot just pass a token list like you can in the
> shell
> > > > (delete TABLE, ROW, COLUMN, {AUTHORIZATIONS => ["token", "token"]})
> > > >
> > > > Is this true or am I missing something?
> > > >
> > > > Thanks,
> > > >
> > > > Mike
> > > >
> > >
> >
>
Re: Deletes with cell visibility labels
Yes, I realized my mistake shortly after posting. So my question is how do
you form a proper delete? Is the expected behavior roughly...
1. Get the row.
2. Rerun the algorithm that computed the visibility label on the row.
3. Build a list of deletes.
Is that what we're expected to do here? Or is there a simpler way of
handling this?
Thanks,
Mike
On Thu, Sep 21, 2017 at 12:18 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Hi Thomson
>
> I think you are saying that the shell allows you to specify
> delete 'tablename', 'row', 'family', ts1, {VISIBILITY=>'PRIVATE|SECRET'}
> but the java client does not allow to do it? I doubt it.
>
> In case of mutations like puts and deletes what we pass is the visibility
> labels. Now when you do a scan that is where we specify the AUTHORIZTIONs
> so that only those cells with visibility cells as passed in the
> AUTHORIZATIONS are returned back.
>
> Hope you find this useful. Let us know if you need further inputs.
>
> Regards
> Ram
>
> On Thu, Sep 21, 2017 at 6:04 PM, Mike Thomsen
> wrote:
>
> > According to the javadocs and some examples I've seen, it looks like with
> > the Java client you have to know the visibility label of the cell you
> want
> > to delete. You cannot just pass a token list like you can in the shell
> > (delete TABLE, ROW, COLUMN, {AUTHORIZATIONS => ["token", "token"]})
> >
> > Is this true or am I missing something?
> >
> > Thanks,
> >
> > Mike
> >
>
Deletes with cell visibility labels
According to the javadocs and some examples I've seen, it looks like with
the Java client you have to know the visibility label of the cell you want
to delete. You cannot just pass a token list like you can in the shell
(delete TABLE, ROW, COLUMN, {AUTHORIZATIONS => ["token", "token"]})
Is this true or am I missing something?
Thanks,
Mike
Solution to my NPE issue I just posted
It appeared to be a problem with HBase in the HortonWorks sandbox. No idea what was going on. Tried another Docker image, this seems to work. Docker Compose configuration: version: '3' services: hbase-docker: container_name: hbase-docker image: "dajobe/hbase-docker" ports: - 16010:16010 - 2181:2181 - 6:6 - 60010:60010 - 60020:60020 - 60030:60030 - 9090:9090 - 9095:9095 hostname: hbase-docker Docker image source: https://github.com/dajobe/hbase-docker Updated the hbase-site.xml to look like this: hbase.zookeeper.quorum hbase-docker hbase.rootdir file:data/hbase hbase.master.info.bindAddress hbase-docker hbase.regionserver.info.bindAddress hbase-docker hbase.coprocessor.region.classes org.apache.hadoop.hbase.security.visibility.VisibilityController hbase.coprocessor.master.classes org.apache.hadoop.hbase.security.visibility.VisibilityController Hope someone else finds that useful.
NullPointerException thrown in the client driver when trying to scan
I wrote a simple Groovy script to test the connection:
@Grab(group='org.apache.hbase', module='hbase-client', version='1.1.5')
@Grab(group='org.hamcrest', module='hamcrest-core', version='1.2')
@Grab(group='org.apache.hbase', module='hbase-common', version='1.1.5')
@Grab(group='org.apache.hadoop', module='hadoop-common', version='2.7.2')
@Grab(group='org.apache.curator', module='curator-client',
version='3.2.0')
@Grab(group='org.apache.curator', module='curator-framework',
version='3.2.0')
@Grab(group='org.apache.curator', module='curator-recipes',
version='3.2.0')
@Grab(group='org.mortbay.jetty', module='jetty', version='6.1.26')
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.client.Admin
import org.apache.hadoop.hbase.client.Connection
import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.HTableDescriptor
import org.apache.hadoop.hbase.HColumnDescriptor
import org.apache.hadoop.hbase.TableName
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.hbase.security.visibility.Authorizations
def log = { msg ->
println "${Calendar.instance.time}\t$msg"
}
Configuration conf = HBaseConfiguration.create();
log "Created configuration"
conf.addResource('/vagrant/workspace/hbase-config/hbase-site.xml')
Connection conn = ConnectionFactory.createConnection(conf);
log "Got conn"
Admin admin = conn.getAdmin();
log "Got admin"
def table = conn.getTable(TableName.valueOf('test_table'))
log "Got table"
assert table
def scan = new Scan()
scan.setAuthorizations(new Authorizations(["public", "private]))
try {
def scanner = table.getScanner(scan)
def fam = 'prop'.bytes
def qual = 'msg'.bytes
log "Got scanner, scanning..."
scanner.each { result ->
println new String(result.row)
// println "\t" + new String(result.getValue(fam, qual))
}
} catch (Exception ex) {
ex.printStackTrace()
}
When I run it, this is the output:
Wed Sep 06 14:01:25 EDT 2017Got conn
Wed Sep 06 14:01:25 EDT 2017Got admin
Wed Sep 06 14:01:25 EDT 2017Got table
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:161)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at org.apache.hadoop.hbase.client.Table$getScanner.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at hbase_test.run(hbase_test.groovy:41)
at
groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:263)
at groovy.lang.GroovyShell.run(GroovyShell.java:518)
at groovy.lang.GroovyShell.run(GroovyShell.java:507)
at groovy.ui.GroovyMain.processOnce(GroovyMain.java:653)
at groovy.ui.GroovyMain.run(GroovyMain.java:384)
at groovy.ui.GroovyMain.process(GroovyMain.java:370)
at groovy.ui.GroovyMain.processArgs(GroovyMain.java:129)
at groovy.ui.GroovyMain.main(GroovyMain.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1250)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1156)
at
org.apache
Visibility labels without Kerberos
Is it possible to use visibility labels without Kerberos? Our admins are still figuring out how to set up Kerberos, and we just need something really simple to get started like being able to set a list of tokens on a scanner and go with that. Is that possible? Thanks, Mike
Re: Fully distributed mode on Windows
You probably shouldn't be using Windows servers for this. For example, Cloudera doesn't list any version of Windows at all as a supported OS: http://www.cloudera.com/documentation/cdh/5-0-x/CDH5-Requirements-and-Supported-Versions/cdhrsv_os.html You'd almost certainly be better off setting up a few Linux VMs on those servers and running it that way. On Fri, Aug 12, 2016 at 10:36 PM, Xiaochun Liang wrote: > Hi, > > Does Hbase support fully distributed mode on Windows? > > I am now trying to setup Hbase on Windows servers. I followed the steps in > https://hbase.apache.org/cygwin.html, and can launch a standalone Hbase > successfully on Windows. Next step I want to setup Hbase on Windows servers > with distributed mode. However I saw followings in 'start-hbase.cmd': > > "if "%distMode%"=="true" ( > @echo This is not implemented yet. Stay tuned." > > I am wondering if distributed mode of Hbase is supported on Windows or not. > > Anyone has experience to setup fully-distributed Hbase on Windows? > > Regards, > xiaochun >
Lily and HBase cell level security
Does anyone have experience with using cell level security and Lily? Doing some basic research on Google didn't yield anything positive. Thanks, Mike
Re: Can a put delete other values?
The behavior I think I observed was one column family hitting its max version and the entire row, all column families, being wiped. Is that expected? On Sun, Dec 13, 2015 at 6:30 PM, Ted Yu wrote: > The Maximum Number of Versions for a Column Family applies to the row. > In your case, subsequent writes cause the max versions to be reached, > leading to the behavior you observed. > > See also http://hbase.apache.org/book.html#specify.number.of.versions > > Cheers > > > On Sun, Dec 13, 2015 at 2:39 PM, Mike Thomsen > wrote: > > > I think our TTL is whatever is or functionally is "forever." We have a > max > > of 3 versions on those column families. When I do that put several times > > (it's in a Storm topology, so it might get sent a dozen times if another > > bolt fails), will that overwrite just the affected cells or affect > > everything in the column family or even the entire row? > > > > Thanks, > > > > Mike > > > > On Sun, Dec 13, 2015 at 5:14 PM, Ted Yu wrote: > > > > > The put for q4, q5, q6 and q7 wouldn't overwrite existing rows. > > > > > > When were the columns q1 to q3 written ? > > > What is the TTL for your table ? > > > > > > Thanks > > > > > > On Sun, Dec 13, 2015 at 12:36 PM, Mike Thomsen > > > > wrote: > > > > > > > I noticed that our test data set is suddenly missing a lot of data, > > and I > > > > am wondering if it's because I'm misunderstanding how HBase handles > > puts. > > > > > > > > Suppose we have 3 families: ca, cb and cc. > > > > > > > > Before on row abcde-fgh-ijkl > > > > > > > > ca:q1= > > > > ca:q2= > > > > ca:q3= > > > > cb:q1= > > > > cb:q2= > > > > cc:q1= > > > > > > > > We send a put that adds q4, q5, q6 and q7. > > > > > > > > Now the row looks like this: > > > > ca:q4= > > > > ca:q5= > > > > ca:q6= > > > > ca:q7= > > > > > > > > I thought a PUT would just add data. It looks like it's overwriting > the > > > > entire row. > > > > > > > > > >
Re: Can a put delete other values?
I think our TTL is whatever is or functionally is "forever." We have a max of 3 versions on those column families. When I do that put several times (it's in a Storm topology, so it might get sent a dozen times if another bolt fails), will that overwrite just the affected cells or affect everything in the column family or even the entire row? Thanks, Mike On Sun, Dec 13, 2015 at 5:14 PM, Ted Yu wrote: > The put for q4, q5, q6 and q7 wouldn't overwrite existing rows. > > When were the columns q1 to q3 written ? > What is the TTL for your table ? > > Thanks > > On Sun, Dec 13, 2015 at 12:36 PM, Mike Thomsen > wrote: > > > I noticed that our test data set is suddenly missing a lot of data, and I > > am wondering if it's because I'm misunderstanding how HBase handles puts. > > > > Suppose we have 3 families: ca, cb and cc. > > > > Before on row abcde-fgh-ijkl > > > > ca:q1= > > ca:q2= > > ca:q3= > > cb:q1= > > cb:q2= > > cc:q1= > > > > We send a put that adds q4, q5, q6 and q7. > > > > Now the row looks like this: > > ca:q4= > > ca:q5= > > ca:q6= > > ca:q7= > > > > I thought a PUT would just add data. It looks like it's overwriting the > > entire row. > > >
Can a put delete other values?
I noticed that our test data set is suddenly missing a lot of data, and I am wondering if it's because I'm misunderstanding how HBase handles puts. Suppose we have 3 families: ca, cb and cc. Before on row abcde-fgh-ijkl ca:q1= ca:q2= ca:q3= cb:q1= cb:q2= cc:q1= We send a put that adds q4, q5, q6 and q7. Now the row looks like this: ca:q4= ca:q5= ca:q6= ca:q7= I thought a PUT would just add data. It looks like it's overwriting the entire row.
Re: Question about reading data very recently written to hbase
I'm almost certain we're not doing that. Thanks for that! On Mon, Oct 5, 2015 at 4:48 PM, Vladimir Rodionov wrote: > HBase writes are consistent. Writes are available immediately only after > table's flush on a client side. > (HTable.flushCommits()) > > -Vlad > > > > On Mon, Oct 5, 2015 at 1:30 PM, Mike Thomsen > wrote: > > > My team has a set of web services that read data from HBase and prepare > it > > to be exported as a report. The first call is an AJAX call that reads all > > of the requested rows, generates the report pieces and returns a JSON map > > to the calling web app saying what level of success it had in building up > > the data for the report. If the data can be exported, it then display a > > link to another web service that builds the report and sends it as a file > > download to the user. > > > > In the first call, it's pretty quick between reading the data and > > immediately writing a copy of the report data to a table in HBase. So the > > turn around time between the two web service calls is usually about five > > seconds or less. > > > > We sometimes get errors back from HBase on the first attempt because it > > appears to have not committed the data from the Java client API to the > > server. Something like that. It can't immediately find the data we just > > generated. If we wait a few more seconds and try it again, everything > works > > just right. It just appears to be the case that the data isn't flushed > into > > a state where it's accessible yet. > > > > To be frank, we're HBase novices and we're probably missing something > > pretty simple and obvious. Is this to be expected? If so, does anyone > have > > a recommendation for how we can mitigate this I/O behavior? > > > > Thanks, > > > > Mike > > >
Question about reading data very recently written to hbase
My team has a set of web services that read data from HBase and prepare it to be exported as a report. The first call is an AJAX call that reads all of the requested rows, generates the report pieces and returns a JSON map to the calling web app saying what level of success it had in building up the data for the report. If the data can be exported, it then display a link to another web service that builds the report and sends it as a file download to the user. In the first call, it's pretty quick between reading the data and immediately writing a copy of the report data to a table in HBase. So the turn around time between the two web service calls is usually about five seconds or less. We sometimes get errors back from HBase on the first attempt because it appears to have not committed the data from the Java client API to the server. Something like that. It can't immediately find the data we just generated. If we wait a few more seconds and try it again, everything works just right. It just appears to be the case that the data isn't flushed into a state where it's accessible yet. To be frank, we're HBase novices and we're probably missing something pretty simple and obvious. Is this to be expected? If so, does anyone have a recommendation for how we can mitigate this I/O behavior? Thanks, Mike
