>>I have Column Family with very large number of column qualifiers (>
50,000). Each column qualifier is 8 bytes long.
When u say u have 50000 qualifiers in a CF, means u will have those many
cells coming under that CF per row. So am not getting what is the
qualifier length limit as such coming. Per qualifier, you will have a diff
cell and its qualifier.
-Anoop-
On Thu, Sep 24, 2015 at 1:13 AM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:
> Yes, the comment is incorrect.
>
> hbase.client.keyvalue.maxsize controls max key-value size, but its
> unlimited in a master (I was wrong about 1MB, this is probably for older
> versions of HBase)
>
>
> -Vlad
>
> On Wed, Sep 23, 2015 at 11:45 AM, Gaurav Agarwal <gau...@arkin.net> wrote:
>
> > Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
> > Just to make sure that I understand correct, are you suggesting that the
> > following comment is incorrect in Cell.java?
> >
> > /**
> > * Contiguous raw bytes that may start at any index in the containing
> > array. Max length is
> > * Short.MAX_VALUE which is 32,767 bytes.
> > * @return The array containing the qualifier bytes.
> > */
> > byte[] getQualifierArray();
> >
> > On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal <gau...@arkin.net>
> wrote:
> >
> > > Thanks Vlad. Could you please point me the KV size setting (default
> 1MB)?
> > > Just to make sure that I understand correct - the following comment is
> > > incorrect in Cell.java:
> > >
> > > /**
> > > * Contiguous raw bytes that may start at any index in the containing
> > > array. Max length is
> > > * Short.MAX_VALUE which is 32,767 bytes.
> > > * @return The array containing the qualifier bytes.
> > > */
> > > byte[] getQualifierArray();
> > >
> > > On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> > > vladrodio...@gmail.com> wrote:
> > >
> > >> Check KeyValue class (Cell's implementation). getQualifierArray()
> > returns
> > >> kv's backing array. There is no SHORT limit on a size of this array,
> but
> > >> there are other limits in HBase - maximum KV size, for example, which
> > is
> > >> configurable, but, by default, is 1MB. Having 50K qualifiers is a bad
> > >> idea.
> > >> Consider redesigning your data model and use rowkey instead.
> > >>
> > >> -Vlad
> > >>
> > >> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >>
> > >> > Please take a look at HBASE-11544 which is in hbase 1.1
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <gau...@arkin.net>
> > >> wrote:
> > >> >
> > >> > > Hi All,
> > >> > >
> > >> > > I have Column Family with very large number of column qualifiers
> (>
> > >> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
> > >> when I
> > >> > > do a scan operation to fetch some rows, the client side Cell
> object
> > >> does
> > >> > > not have enough space allocated in it to hold all the
> > columnQaulifiers
> > >> > for
> > >> > > a given row and hence I cannot read all the columns back for a
> given
> > >> row.
> > >> > >
> > >> > > Please see the code snippet that I am using:
> > >> > >
> > >> > > final ResultScanner rs = htable.getScanner(scan);
> > >> > > for (Result row = rs.next(); row != null; row = rs.next()) {
> > >> > > final Cell[] cells = row.rawCells();
> > >> > > if (cells != null) {
> > >> > > for (final Cell cell : cells) {
> > >> > > final long c = Bytes.toLong(
> > >> > > *cell.getQualifierArray()*,
> > >> > cell.getQualifierOffset(),
> > >> > > cell.getQualifierLength());
> > >> > > final long v = Bytes.toLong(cell.getValueArray(),
> > >> > > cell.getValueOffset());
> > >> > > points.put(c, v);
> > >> > > }
> > >> > > }
> > >> > > }
> > >> > >
> > >> > > The cell.getQualifierArray() method says that it's 'Max length is
> > >> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold
> > around
> > >> > > 4,000 columnQualfiers.
> > >> > >
> > >> > > Is there an alternate API that I should be using or am I missing
> > some
> > >> > > setting here? Note that in worst case I need to read all the
> > >> > > columnQualifiers in a row and I may or may not know a subset to
> > fetch
> > >> in
> > >> > > advance.
> > >> > >
> > >> > > Even if this is not possible in a single call, is there a way to
> > >> cursor
> > >> > > through the columnQualifiers?
> > >> > >
> > >> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x
> > if
> > >> > there
> > >> > > is an API in the newer version.
> > >> > >
> > >> > > --cheers, gaurav
> > >> > >
> > >> > > --
> > >> > > --cheers, gaurav
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > --cheers, gaurav
> > >
> >
> >
> >
> > --
> > --cheers, gaurav
> >
>
