After spending more time I realised that my understanding and my question
(was invalid).
I am still trying to get more information regarding the problem and will
update the thread once I have a better handle on the problem.
Apologies for the confusion..
On Thu, Sep 24, 2015 at 10:32 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Am not sure whether you have tried it. the scan API has got an API called
> 'batching'. Did you try it? So per row if there are more columns you can
> still limit the amount of data being sent to the client. I think the main
> issue you are facing is that the qualifiers getting returned are more in
> number and so the client is not able to accept them?
>
> 'Short.MAX_VALUE which is 32,767 bytes.'
> This comment applies for the qualifier length ie. the name that you specify
> for the qualifier not on the number of qualifiers.
>
> Regards
> Ram
>
> On Thu, Sep 24, 2015 at 8:52 AM, Anoop John <anoop.hb...@gmail.com> wrote:
>
> > >>I have Column Family with very large number of column qualifiers (>
> > 50,000). Each column qualifier is 8 bytes long.
> >
> > When u say u have 50000 qualifiers in a CF, means u will have those many
> > cells coming under that CF per row. So am not getting what is the
> > qualifier length limit as such coming. Per qualifier, you will have a
> diff
> > cell and its qualifier.
> >
> > -Anoop-
> >
> >
> > On Thu, Sep 24, 2015 at 1:13 AM, Vladimir Rodionov <
> vladrodio...@gmail.com
> > >
> > wrote:
> >
> > > Yes, the comment is incorrect.
> > >
> > > hbase.client.keyvalue.maxsize controls max key-value size, but its
> > > unlimited in a master (I was wrong about 1MB, this is probably for
> older
> > > versions of HBase)
> > >
> > >
> > > -Vlad
> > >
> > > On Wed, Sep 23, 2015 at 11:45 AM, Gaurav Agarwal <gau...@arkin.net>
> > wrote:
> > >
> > > > Thanks Vlad. Could you please point me the KV size setting (default
> > 1MB)?
> > > > Just to make sure that I understand correct, are you suggesting that
> > the
> > > > following comment is incorrect in Cell.java?
> > > >
> > > > /**
> > > > * Contiguous raw bytes that may start at any index in the
> containing
> > > > array. Max length is
> > > > * Short.MAX_VALUE which is 32,767 bytes.
> > > > * @return The array containing the qualifier bytes.
> > > > */
> > > > byte[] getQualifierArray();
> > > >
> > > > On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal <gau...@arkin.net>
> > > wrote:
> > > >
> > > > > Thanks Vlad. Could you please point me the KV size setting (default
> > > 1MB)?
> > > > > Just to make sure that I understand correct - the following comment
> > is
> > > > > incorrect in Cell.java:
> > > > >
> > > > > /**
> > > > > * Contiguous raw bytes that may start at any index in the
> > containing
> > > > > array. Max length is
> > > > > * Short.MAX_VALUE which is 32,767 bytes.
> > > > > * @return The array containing the qualifier bytes.
> > > > > */
> > > > > byte[] getQualifierArray();
> > > > >
> > > > > On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> > > > > vladrodio...@gmail.com> wrote:
> > > > >
> > > > >> Check KeyValue class (Cell's implementation). getQualifierArray()
> > > > returns
> > > > >> kv's backing array. There is no SHORT limit on a size of this
> array,
> > > but
> > > > >> there are other limits in HBase - maximum KV size, for example,
> > which
> > > > is
> > > > >> configurable, but, by default, is 1MB. Having 50K qualifiers is a
> > bad
> > > > >> idea.
> > > > >> Consider redesigning your data model and use rowkey instead.
> > > > >>
> > > > >> -Vlad
> > > > >>
> > > > >> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Please take a look at HBASE-11544 which is in hbase 1.1
> > > > >> >
> > > > >> > Cheers
> > > > >> >
> > > > >> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <
> > gau...@arkin.net>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi All,
> > > > >> > >
> > > > >> > > I have Column Family with very large number of column
> qualifiers
> > > (>
> > > > >> > > 50,000). Each column qualifier is 8 bytes long. The problem is
> > the
> > > > >> when I
> > > > >> > > do a scan operation to fetch some rows, the client side Cell
> > > object
> > > > >> does
> > > > >> > > not have enough space allocated in it to hold all the
> > > > columnQaulifiers
> > > > >> > for
> > > > >> > > a given row and hence I cannot read all the columns back for a
> > > given
> > > > >> row.
> > > > >> > >
> > > > >> > > Please see the code snippet that I am using:
> > > > >> > >
> > > > >> > > final ResultScanner rs = htable.getScanner(scan);
> > > > >> > > for (Result row = rs.next(); row != null; row = rs.next()) {
> > > > >> > > final Cell[] cells = row.rawCells();
> > > > >> > > if (cells != null) {
> > > > >> > > for (final Cell cell : cells) {
> > > > >> > > final long c = Bytes.toLong(
> > > > >> > > *cell.getQualifierArray()*,
> > > > >> > cell.getQualifierOffset(),
> > > > >> > > cell.getQualifierLength());
> > > > >> > > final long v = Bytes.toLong(cell.getValueArray(),
> > > > >> > > cell.getValueOffset());
> > > > >> > > points.put(c, v);
> > > > >> > > }
> > > > >> > > }
> > > > >> > > }
> > > > >> > >
> > > > >> > > The cell.getQualifierArray() method says that it's 'Max length
> > is
> > > > >> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only
> hold
> > > > around
> > > > >> > > 4,000 columnQualfiers.
> > > > >> > >
> > > > >> > > Is there an alternate API that I should be using or am I
> missing
> > > > some
> > > > >> > > setting here? Note that in worst case I need to read all the
> > > > >> > > columnQualifiers in a row and I may or may not know a subset
> to
> > > > fetch
> > > > >> in
> > > > >> > > advance.
> > > > >> > >
> > > > >> > > Even if this is not possible in a single call, is there a way
> to
> > > > >> cursor
> > > > >> > > through the columnQualifiers?
> > > > >> > >
> > > > >> > > I am presently using Hbase 0.96 client but can switch to Hbase
> > 1.x
> > > > if
> > > > >> > there
> > > > >> > > is an API in the newer version.
> > > > >> > >
> > > > >> > > --cheers, gaurav
> > > > >> > >
> > > > >> > > --
> > > > >> > > --cheers, gaurav
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --cheers, gaurav
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > --cheers, gaurav
> > > >
> > >
> >
>
--
--cheers, gaurav
