subject:"How to manually generate a salted row key\?"

Re: How to manually generate a salted row key?

2016-09-14 Thread Marica Tan

Hi Josh,

It turns out the salt bucket in the create table script is different than
what we're using in the generation of row key :D

Anyway thanks for the help :)


Marica

On Wed, Sep 14, 2016 at 3:33 PM, Marica Tan  wrote:

> Hi Josh,
>
> It works without salting.  Tried the getSaltedKey but still the same
> problem.
>
> The generated row key when doing an upsert via phoenix sqlline, is
> different and I can query the one created via phoenix.
>
> via phoenix: \x09O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA
>
> via hbase api: \x15O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA
>
>
> Btw, I modified the code a little:
>
> def generateRowKey(nBuckets: Integer, compositeKeys: String*): Array[Byte] = {
>   val keys = 
> compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
>  b) => {
> a += QueryConstants.SEPARATOR_BYTE_ARRAY
> a += convertToByteArray(b)
>   })
>
>   updateSaltingByte(keys, nBuckets)
> }
>
> def convertToByteArray(key: String): Array[Byte] = key match {
>   case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
>   case _ => ByteUtil.EMPTY_BYTE_ARRAY
> }
>
> def updateSaltingByte(rowKey: ArrayBuffer[Array[Byte]], nBuckets: Integer): 
> Array[Byte] = {
>   if (nBuckets > 0) {
> SaltingUtil.getSaltedKey(new 
> ImmutableBytesWritable(ByteUtil.concat(Array.fill[Byte](1)(0), rowKey.toSeq: 
> _*)), nBuckets);
>
>   } else {
> ByteUtil.concat(rowKey.head, rowKey.tail.toSeq: _*)
>
>   }
> }
>
>
> From what I understand, phoenix will just combine those keys separated by
> a zero byte. Am I doing it wrong?
>
> I would greatly appreciate any help you can give me to solve this problem.
>
>
> Thanks!
> --
> Marica
>
> On Wed, Sep 14, 2016 at 8:15 AM, Marica Tan  wrote:
>
>> Hi Josh,
>>
>> Haven't tried without salting yet. The problem we encountered is only on
>> the firstKey, when we do select with a where condition on the secondKey or
>> thirdKey, it returns the correct result. So I'm guessing it has something
>> to do with the salted key.
>>
>> Will try the getSaltedKey first and hoping that it will fix the problem :)
>>
>> Thanks for the help!
>>
>> --
>> Marica
>>
>> On 2016-09-13 21:50 (+0800), Josh Mahonin  wrote:
>> > Hi Marica,>
>> >
>> > Are you able to successfully write your rowkey without salting? If not,
>> it>
>> > could be that your 'generateRowKey' function is the culprit.>
>> >
>> > FWIW, we have some code that does something similar, though we use>
>> > 'getSaltedKey':>
>> >
>> > // If salting, we need to prepend an empty byte to 'rowKey', then fill
>> it>
>> > if (saltBuckets > 0) {>
>> > rowKey = ByteUtil.concat(new byte[1], rowKey);>
>> > rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),>
>> > saltBuckets);>
>> > }>
>> >
>> > Good luck,>
>> >
>> > Josh>
>>
>> >
>> > On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan 
>> wrote:>
>> >
>> > > Hi,>
>> > >>
>> > > We have a table created via phoenix with salt bucket, but we're using>
>> > > HBase API to insert records since we need to manually set the HBase
>> version>
>> > > and I believe that it isn't possible via phoenix.>
>> > >>
>> > > Our table has a composite key (firstKey varchar, secondKey varchar,>
>> > > thirdKey varchar) and when we do a select query with a where
>> condition on>
>> > > the firstKey, not all records are retrieved.>
>> > >>
>> > > We checked the value of the firstKey and it should return 10 records,
>> but>
>> > > we're only getting 7.>
>> > >>
>> > > If we do a where firstKey = 'someValue' we get 7>
>> > > If we do a where firstKey like '%someValue' we get 10>
>> > >>
>> > > So we think the main culprit is the way we generate the row key.
>> Here's>
>> > > the code:>
>> > >>
>> > > def generateRowKey(nBuckets: Integer, compositeKeys: String*):>
>> > > Array[Byte] = {>
>> > >>
>> > > val keys = compositeKeys.tail.foldLeft(Ar
>> rayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a, b)
>> => {>
>> > > a += QueryConstants.SEPARATOR_BYTE_ARRAY>
>> > > a += convertToByteArray(b)>
>> > > })>
>> > >>
>> > > val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
>> keys.toSeq: _*)>
>> > >>
>> > > updateSaltingByte(rowKey, nBuckets)>
>> > >>
>> > > rowKey>
>> > > }>
>> > >>
>> > > def convertToByteArray(key: String): Array[Byte] = key match {>
>> > > case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)>
>> > > case _ => ByteUtil.EMPTY_BYTE_ARRAY>
>> > > }>
>> > >>
>> > > def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit =
>> {>
>> > > if (nBuckets > 0) {>
>> > > rowKey(0) = SaltingUtil.getSaltingByte(rowKey,>
>> > > SaltingUtil.NUM_SALTING_BYTES, rowKey.length -
>> SaltingUtil.NUM_SALTING_BYTES, nBuckets)>
>> > > }>
>> > > }>
>> > >>
>> > >>
>> > > Btw, we're using phoenix 4.4>
>> > >>
>> > >>
>> > > Thanks,>
>> > > -->
>> > > Marica Tan>
>> > >>
>> >
>>
>
>

Re: How to manually generate a salted row key?

2016-09-14 Thread Marica Tan

Hi Josh,

It works without salting.  Tried the getSaltedKey but still the same
problem.

The generated row key when doing an upsert via phoenix sqlline, is
different and I can query the one created via phoenix.

via phoenix: \x09O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA

via hbase api: \x15O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA


Btw, I modified the code a little:

def generateRowKey(nBuckets: Integer, compositeKeys: String*): Array[Byte] = {
  val keys = 
compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
b) => {
a += QueryConstants.SEPARATOR_BYTE_ARRAY
a += convertToByteArray(b)
  })

  updateSaltingByte(keys, nBuckets)
}

def convertToByteArray(key: String): Array[Byte] = key match {
  case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
  case _ => ByteUtil.EMPTY_BYTE_ARRAY
}

def updateSaltingByte(rowKey: ArrayBuffer[Array[Byte]], nBuckets:
Integer): Array[Byte] = {
  if (nBuckets > 0) {
SaltingUtil.getSaltedKey(new
ImmutableBytesWritable(ByteUtil.concat(Array.fill[Byte](1)(0),
rowKey.toSeq: _*)), nBuckets);

  } else {
ByteUtil.concat(rowKey.head, rowKey.tail.toSeq: _*)

  }
}


>From what I understand, phoenix will just combine those keys separated by a
zero byte. Am I doing it wrong?

I would greatly appreciate any help you can give me to solve this problem.


Thanks!
--
Marica

On Wed, Sep 14, 2016 at 8:15 AM, Marica Tan  wrote:

> Hi Josh,
>
> Haven't tried without salting yet. The problem we encountered is only on
> the firstKey, when we do select with a where condition on the secondKey or
> thirdKey, it returns the correct result. So I'm guessing it has something
> to do with the salted key.
>
> Will try the getSaltedKey first and hoping that it will fix the problem :)
>
> Thanks for the help!
>
> --
> Marica
>
> On 2016-09-13 21:50 (+0800), Josh Mahonin  wrote:
> > Hi Marica,>
> >
> > Are you able to successfully write your rowkey without salting? If not,
> it>
> > could be that your 'generateRowKey' function is the culprit.>
> >
> > FWIW, we have some code that does something similar, though we use>
> > 'getSaltedKey':>
> >
> > // If salting, we need to prepend an empty byte to 'rowKey', then fill
> it>
> > if (saltBuckets > 0) {>
> > rowKey = ByteUtil.concat(new byte[1], rowKey);>
> > rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),>
> > saltBuckets);>
> > }>
> >
> > Good luck,>
> >
> > Josh>
>
> >
> > On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan 
> wrote:>
> >
> > > Hi,>
> > >>
> > > We have a table created via phoenix with salt bucket, but we're using>
> > > HBase API to insert records since we need to manually set the HBase
> version>
> > > and I believe that it isn't possible via phoenix.>
> > >>
> > > Our table has a composite key (firstKey varchar, secondKey varchar,>
> > > thirdKey varchar) and when we do a select query with a where condition
> on>
> > > the firstKey, not all records are retrieved.>
> > >>
> > > We checked the value of the firstKey and it should return 10 records,
> but>
> > > we're only getting 7.>
> > >>
> > > If we do a where firstKey = 'someValue' we get 7>
> > > If we do a where firstKey like '%someValue' we get 10>
> > >>
> > > So we think the main culprit is the way we generate the row key.
> Here's>
> > > the code:>
> > >>
> > > def generateRowKey(nBuckets: Integer, compositeKeys: String*):>
> > > Array[Byte] = {>
> > >>
> > > val keys = compositeKeys.tail.foldLeft(Ar
> rayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a, b) =>
> {>
> > > a += QueryConstants.SEPARATOR_BYTE_ARRAY>
> > > a += convertToByteArray(b)>
> > > })>
> > >>
> > > val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
> keys.toSeq: _*)>
> > >>
> > > updateSaltingByte(rowKey, nBuckets)>
> > >>
> > > rowKey>
> > > }>
> > >>
> > > def convertToByteArray(key: String): Array[Byte] = key match {>
> > > case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)>
> > > case _ => ByteUtil.EMPTY_BYTE_ARRAY>
> > > }>
> > >>
> > > def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit =
> {>
> > > if (nBuckets > 0) {>
> > > rowKey(0) = SaltingUtil.getSaltingByte(rowKey,>
> > > SaltingUtil.NUM_SALTING_BYTES, rowKey.length -
> SaltingUtil.NUM_SALTING_BYTES, nBuckets)>
> > > }>
> > > }>
> > >>
> > >>
> > > Btw, we're using phoenix 4.4>
> > >>
> > >>
> > > Thanks,>
> > > -->
> > > Marica Tan>
> > >>
> >
>

Re: How to manually generate a salted row key?

2016-09-13 Thread Marica Tan


Hi Josh,

Haven't tried without salting yet. The problem we encountered is only on 
the firstKey, when we do select with a where condition on the secondKey 
or thirdKey, it returns the correct result. So I'm guessing it has 
something to do with the salted key.


Will try the getSaltedKey first and hoping that it will fix the problem :)

Thanks for the help!

--
Marica

On 2016-09-13 21:50 (+0800), Josh Mahonin  wrote:
> Hi Marica,>
>
> Are you able to successfully write your rowkey without salting? If 
not, it>

> could be that your 'generateRowKey' function is the culprit.>
>
> FWIW, we have some code that does something similar, though we use>
> 'getSaltedKey':>
>
> // If salting, we need to prepend an empty byte to 'rowKey', then 
fill it>

> if (saltBuckets > 0) {>
> rowKey = ByteUtil.concat(new byte[1], rowKey);>
> rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),>
> saltBuckets);>
> }>
>
> Good luck,>
>
> Josh>
>
> On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan  
wrote:>

>
> > Hi,>
> >>
> > We have a table created via phoenix with salt bucket, but we're using>
> > HBase API to insert records since we need to manually set the HBase 
version>

> > and I believe that it isn't possible via phoenix.>
> >>
> > Our table has a composite key (firstKey varchar, secondKey varchar,>
> > thirdKey varchar) and when we do a select query with a where 
condition on>

> > the firstKey, not all records are retrieved.>
> >>
> > We checked the value of the firstKey and it should return 10 
records, but>

> > we're only getting 7.>
> >>
> > If we do a where firstKey = 'someValue' we get 7>
> > If we do a where firstKey like '%someValue' we get 10>
> >>
> > So we think the main culprit is the way we generate the row key. 
Here's>

> > the code:>
> >>
> > def generateRowKey(nBuckets: Integer, compositeKeys: String*):>
> > Array[Byte] = {>
> >>
> > val keys = 
compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a, 
b) => {>

> > a += QueryConstants.SEPARATOR_BYTE_ARRAY>
> > a += convertToByteArray(b)>
> > })>
> >>
> > val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY, 
keys.toSeq: _*)>

> >>
> > updateSaltingByte(rowKey, nBuckets)>
> >>
> > rowKey>
> > }>
> >>
> > def convertToByteArray(key: String): Array[Byte] = key match {>
> > case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)>
> > case _ => ByteUtil.EMPTY_BYTE_ARRAY>
> > }>
> >>
> > def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit 
= {>

> > if (nBuckets > 0) {>
> > rowKey(0) = SaltingUtil.getSaltingByte(rowKey,>
> > SaltingUtil.NUM_SALTING_BYTES, rowKey.length - 
SaltingUtil.NUM_SALTING_BYTES, nBuckets)>

> > }>
> > }>
> >>
> >>
> > Btw, we're using phoenix 4.4>
> >>
> >>
> > Thanks,>
> > -->
> > Marica Tan>
> >>
>

Re: How to manually generate a salted row key?

2016-09-13 Thread Josh Mahonin

Hi Marica,

Are you able to successfully write your rowkey without salting? If not, it
could be that your 'generateRowKey' function is the culprit.

FWIW, we have some code that does something similar, though we use
'getSaltedKey':

// If salting, we need to prepend an empty byte to 'rowKey', then fill it
if (saltBuckets > 0) {
rowKey = ByteUtil.concat(new byte[1], rowKey);
rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),
saltBuckets);
}

Good luck,

Josh

On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan  wrote:

> Hi,
>
> We have a table created via phoenix with salt bucket, but we're using
> HBase API to insert records since we need to manually set the HBase version
> and I believe that it isn't possible via phoenix.
>
> Our table has a composite key (firstKey varchar, secondKey varchar,
> thirdKey varchar) and when we do a select query with a where condition on
> the firstKey, not all records are retrieved.
>
> We checked the value of the firstKey and it should return 10 records, but
> we're only getting 7.
>
> If we do a where firstKey = 'someValue' we get 7
> If we do a where firstKey like '%someValue' we get 10
>
> So we think the main culprit is the way we generate the row key. Here's
> the code:
>
>  def generateRowKey(nBuckets: Integer, compositeKeys: String*):
> Array[Byte] = {
>
>   val keys = 
> compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
>  b) => {
> a += QueryConstants.SEPARATOR_BYTE_ARRAY
> a += convertToByteArray(b)
>   })
>
>   val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY, 
> keys.toSeq: _*)
>
>   updateSaltingByte(rowKey, nBuckets)
>
>   rowKey
> }
>
> def convertToByteArray(key: String): Array[Byte] = key match {
>   case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
>   case _ => ByteUtil.EMPTY_BYTE_ARRAY
> }
>
> def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit = {
>   if (nBuckets > 0) {
> rowKey(0) = SaltingUtil.getSaltingByte(rowKey,
>   SaltingUtil.NUM_SALTING_BYTES, rowKey.length - 
> SaltingUtil.NUM_SALTING_BYTES, nBuckets)
>   }
> }
>
>
> Btw, we're using phoenix 4.4
>
>
> Thanks,
> --
> Marica Tan
>

How to manually generate a salted row key?

2016-09-13 Thread Marica Tan

Hi,

We have a table created via phoenix with salt bucket, but we're using HBase
API to insert records since we need to manually set the HBase version and I
believe that it isn't possible via phoenix.

Our table has a composite key (firstKey varchar, secondKey varchar,
thirdKey varchar) and when we do a select query with a where condition on
the firstKey, not all records are retrieved.

We checked the value of the firstKey and it should return 10 records, but
we're only getting 7.

If we do a where firstKey = 'someValue' we get 7
If we do a where firstKey like '%someValue' we get 10

So we think the main culprit is the way we generate the row key. Here's the
code:

 def generateRowKey(nBuckets: Integer, compositeKeys: String*): Array[Byte]
= {

  val keys = 
compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
b) => {
a += QueryConstants.SEPARATOR_BYTE_ARRAY
a += convertToByteArray(b)
  })

  val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
keys.toSeq: _*)

  updateSaltingByte(rowKey, nBuckets)

  rowKey
}

def convertToByteArray(key: String): Array[Byte] = key match {
  case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
  case _ => ByteUtil.EMPTY_BYTE_ARRAY
}

def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit = {
  if (nBuckets > 0) {
rowKey(0) = SaltingUtil.getSaltingByte(rowKey,
  SaltingUtil.NUM_SALTING_BYTES, rowKey.length -
SaltingUtil.NUM_SALTING_BYTES, nBuckets)
  }
}


Btw, we're using phoenix 4.4


Thanks,
--
Marica Tan

Re: How to manually generate a salted row key?

Re: How to manually generate a salted row key?

Re: How to manually generate a salted row key?

Re: How to manually generate a salted row key?

How to manually generate a salted row key?

5 matches

Site Navigation

Mail list logo

Footer information