subject:"DRILL\-4199\: Add Support for HBase 1.X \- planning to merge"

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-22 Thread qiang li

Hi all,

Just to report my test progress.

As the SQL always throw schema changes exception. I reformatted my SQL, now
it will not throw exception.

I changed SQL
*from *
select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
convert_from(`ref0`.`v`.`v`,'UTF8') as v
from hbase.`offers_nation_idx` as `nation` join hbase.offers_ref0 as `ref0`
on BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v`
where `nation`.row_key  > '0br' and `nation`.row_key  < '0bs' limit 10

*to *
select t1.uid, t1.v from
  (select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as
uid, convert_from(`ref0`.`v`.`v`,'UTF8') as v
   from hbase.offers_ref0 as `ref0` where `ref0`.row_key > '0' and
`ref0`.row_key < '1' ) t1
join
  (select CONVERT_FROM(nation.`v`.`v`, 'BIGINT_BE') uid
   from hbase.`offers_nation_idx` as `nation` where `nation`.row_key  >
'0br' and `nation`.row_key  < '0bs' ) t2
on t1.uid = t2.uid limit 10

Now, my work can going on. But I think the first SQL should working too,
hope someone can take a look at it.






2016-06-23 10:59 GMT+08:00 qiang li :

> Dear all,
>
> Today ,I retest it again step by step. This time I get exception in both
> drillbit.log and drillbit.out, also I tested with released 1.6.0, its throw
> Hash join does not support schema changes exception too.
>
>
> I have reattached the error msg to 4734, Could some one help me solve this
> issue, we are very need this function working.
>
> Thanks
>
> 2016-06-22 9:39 GMT+08:00 qiang li :
>
>> Thanks Aman.
>>
>> I think the second is not related to hbase too. Its
>> happened occasionally. I wil give you more detail when its happen again.
>>
>> Hi Aditya, there is no error in drillbit.out, but have error in
>> drillbit.log, I have attached to the DRILL-4734
>>  , please check it .
>>
>>
>>
>> 2016-06-22 4:31 GMT+08:00 Aman Sinha :
>>
>>> Qiang,
>>> The last issue mentioned about failure to start drillbit seems unrelated
>>> to HBase isn't it ?  For now, I am going with the assumption that it is not
>>> a regression, so does not impact the 1.7 release.  Of course, if you find
>>> out otherwise, we would need to address it.
>>> Regarding the SchemaChangeException you reported, I have created a JIRA:
>>> DRILL-4734.
>>>
>>>
>>> -Aman
>>>
>>> On Tue, Jun 21, 2016 at 10:04 AM, Aditya 
>>> wrote:
>>>
 There should be another stack trace in drillbit.out when this happens,
 could you please check that file.

 On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:

 > Another issue is some time when I restart the node, the node can not
 be
 > startup.
 >
 > Here is the exception.
 > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
 >
 jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
 > took 2800ms
 > 2016-06-20 19:10:18,313 [main] INFO
 o.a.d.e.s.s.PersistentStoreRegistry -
 > Using the configured PStoreProvider class:
 >
 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
 > 2016-06-20 19:10:19,221 [main] INFO
 o.apache.drill.exec.server.Drillbit -
 > Construction completed (1529 ms).
 > 2016-06-20 19:10:31,136 [main] WARN
 o.apache.drill.exec.server.Drillbit -
 > Failure on close()
 > java.lang.NullPointerException: null
 > at
 > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
 > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
 > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
 > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > at
 org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
 > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
 > 2016-06-20 19:10:31,137 [main] INFO
 o.apache.drill.exec.server.Drillbit -
 > Shutdown completed (1914 ms).
 >
 > I did nothing and start it at next day, then it can startup.
 >
 > 2016-06-21 9:48 GMT+08:00 qiang li :
 >
 >> Hi Aman,
 >>
 >> I did not fully test with the old version.
 >>
 >> Cloud you please help me create the JIRA issue,  I think my account
 have
 >> not the privilege, my account is griffinli and can not find the
 place to
 >> create new issue. Below is the explain detail for the same SQL in
 diffe

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-22 Thread qiang li

Dear all,

Today ,I retest it again step by step. This time I get exception in both
drillbit.log and drillbit.out, also I tested with released 1.6.0, its throw
Hash join does not support schema changes exception too.


I have reattached the error msg to 4734, Could some one help me solve this
issue, we are very need this function working.

Thanks

2016-06-22 9:39 GMT+08:00 qiang li :

> Thanks Aman.
>
> I think the second is not related to hbase too. Its happened occasionally.
> I wil give you more detail when its happen again.
>
> Hi Aditya, there is no error in drillbit.out, but have error in
> drillbit.log, I have attached to the DRILL-4734
>  , please check it .
>
>
>
> 2016-06-22 4:31 GMT+08:00 Aman Sinha :
>
>> Qiang,
>> The last issue mentioned about failure to start drillbit seems unrelated
>> to HBase isn't it ?  For now, I am going with the assumption that it is not
>> a regression, so does not impact the 1.7 release.  Of course, if you find
>> out otherwise, we would need to address it.
>> Regarding the SchemaChangeException you reported, I have created a JIRA:
>> DRILL-4734.
>>
>>
>> -Aman
>>
>> On Tue, Jun 21, 2016 at 10:04 AM, Aditya  wrote:
>>
>>> There should be another stack trace in drillbit.out when this happens,
>>> could you please check that file.
>>>
>>> On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:
>>>
>>> > Another issue is some time when I restart the node, the node can not be
>>> > startup.
>>> >
>>> > Here is the exception.
>>> > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
>>> >
>>> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
>>> > took 2800ms
>>> > 2016-06-20 19:10:18,313 [main] INFO
>>> o.a.d.e.s.s.PersistentStoreRegistry -
>>> > Using the configured PStoreProvider class:
>>> >
>>> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
>>> > 2016-06-20 19:10:19,221 [main] INFO
>>> o.apache.drill.exec.server.Drillbit -
>>> > Construction completed (1529 ms).
>>> > 2016-06-20 19:10:31,136 [main] WARN
>>> o.apache.drill.exec.server.Drillbit -
>>> > Failure on close()
>>> > java.lang.NullPointerException: null
>>> > at
>>> > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
>>> > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
>>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
>>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > at
>>> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>>> > 2016-06-20 19:10:31,137 [main] INFO
>>> o.apache.drill.exec.server.Drillbit -
>>> > Shutdown completed (1914 ms).
>>> >
>>> > I did nothing and start it at next day, then it can startup.
>>> >
>>> > 2016-06-21 9:48 GMT+08:00 qiang li :
>>> >
>>> >> Hi Aman,
>>> >>
>>> >> I did not fully test with the old version.
>>> >>
>>> >> Cloud you please help me create the JIRA issue,  I think my account
>>> have
>>> >> not the privilege, my account is griffinli and can not find the place
>>> to
>>> >> create new issue. Below is the explain detail for the same SQL in
>>> different
>>> >> nodes of cluster.
>>> >>
>>> >>
>>> >> This is the correct plan which only have two nodes:
>>> >> 0: jdbc:drill:zk=xxx:> explain plan for select
>>> >> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>>> >> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from
>>> hbase.`offers_nation_idx` as
>>> >> `nation` join hbase.offers_ref0 as `ref0` on
>>> >>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where
>>> `nation`.row_key
>>> >>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>>> >> +--+--+
>>> >> | text | json |
>>> >> +--+--+
>>> >> | 00-00Screen
>>> >> 00-01  Project(uid=[$0], v=[$1])
>>> >> 00-02SelectionVectorRemover
>>> >> 00-03  Limit(fetch=[10])
>>> >> 00-04UnionExchange
>>> >> 01-01  SelectionVectorRemover
>>> >> 01-02Limit(fetch=[10])
>>> >> 01-03
>>> Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>>> >> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>>> >> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
>>> >> row_key0=[$0], v0=[$1], $f2=[$2])
>>> >> 01-05  HashJoin(condition=[=($2, $5)],
>>> >> joinType=[inner])
>>> >> 01-07

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-21 Thread qiang li

Thanks Aman.

I think the second is not related to hbase too. Its happened occasionally.
I wil give you more detail when its happen again.

Hi Aditya, there is no error in drillbit.out, but have error in
drillbit.log, I have attached to the DRILL-4734
 , please check it .



2016-06-22 4:31 GMT+08:00 Aman Sinha :

> Qiang,
> The last issue mentioned about failure to start drillbit seems unrelated
> to HBase isn't it ?  For now, I am going with the assumption that it is not
> a regression, so does not impact the 1.7 release.  Of course, if you find
> out otherwise, we would need to address it.
> Regarding the SchemaChangeException you reported, I have created a JIRA:
> DRILL-4734.
>
>
> -Aman
>
> On Tue, Jun 21, 2016 at 10:04 AM, Aditya  wrote:
>
>> There should be another stack trace in drillbit.out when this happens,
>> could you please check that file.
>>
>> On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:
>>
>> > Another issue is some time when I restart the node, the node can not be
>> > startup.
>> >
>> > Here is the exception.
>> > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
>> >
>> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
>> > took 2800ms
>> > 2016-06-20 19:10:18,313 [main] INFO
>> o.a.d.e.s.s.PersistentStoreRegistry -
>> > Using the configured PStoreProvider class:
>> >
>> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
>> > 2016-06-20 19:10:19,221 [main] INFO
>> o.apache.drill.exec.server.Drillbit -
>> > Construction completed (1529 ms).
>> > 2016-06-20 19:10:31,136 [main] WARN
>> o.apache.drill.exec.server.Drillbit -
>> > Failure on close()
>> > java.lang.NullPointerException: null
>> > at
>> > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
>> > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at
>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at
>> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
>> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at
>> org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at
>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at
>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>> > 2016-06-20 19:10:31,137 [main] INFO
>> o.apache.drill.exec.server.Drillbit -
>> > Shutdown completed (1914 ms).
>> >
>> > I did nothing and start it at next day, then it can startup.
>> >
>> > 2016-06-21 9:48 GMT+08:00 qiang li :
>> >
>> >> Hi Aman,
>> >>
>> >> I did not fully test with the old version.
>> >>
>> >> Cloud you please help me create the JIRA issue,  I think my account
>> have
>> >> not the privilege, my account is griffinli and can not find the place
>> to
>> >> create new issue. Below is the explain detail for the same SQL in
>> different
>> >> nodes of cluster.
>> >>
>> >>
>> >> This is the correct plan which only have two nodes:
>> >> 0: jdbc:drill:zk=xxx:> explain plan for select
>> >> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>> >> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from
>> hbase.`offers_nation_idx` as
>> >> `nation` join hbase.offers_ref0 as `ref0` on
>> >>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where
>> `nation`.row_key
>> >>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>> >> +--+--+
>> >> | text | json |
>> >> +--+--+
>> >> | 00-00Screen
>> >> 00-01  Project(uid=[$0], v=[$1])
>> >> 00-02SelectionVectorRemover
>> >> 00-03  Limit(fetch=[10])
>> >> 00-04UnionExchange
>> >> 01-01  SelectionVectorRemover
>> >> 01-02Limit(fetch=[10])
>> >> 01-03
>> Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>> >> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>> >> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
>> >> row_key0=[$0], v0=[$1], $f2=[$2])
>> >> 01-05  HashJoin(condition=[=($2, $5)],
>> >> joinType=[inner])
>> >> 01-07Project(row_key=[$0], v=[$1],
>> >> $f2=[BYTE_SUBSTR($0, -8, 8)])
>> >> 01-09  Scan(groupscan=[HBaseGroupScan
>> >> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
>> >> stopRow=null, filter=null], columns=[`*`]]])
>> >> 01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
>> >> 01-08  *BroadcastExchange*
>>
>> >> 02-01Project(row_key=[$0], v=[$1],
>> >> ITEM=[ITEM($1, 'v')])
>>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-21 Thread Aman Sinha

Qiang,
The last issue mentioned about failure to start drillbit seems unrelated to
HBase isn't it ?  For now, I am going with the assumption that it is not a
regression, so does not impact the 1.7 release.  Of course, if you find out
otherwise, we would need to address it.
Regarding the SchemaChangeException you reported, I have created a JIRA:
DRILL-4734.


-Aman

On Tue, Jun 21, 2016 at 10:04 AM, Aditya  wrote:

> There should be another stack trace in drillbit.out when this happens,
> could you please check that file.
>
> On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:
>
> > Another issue is some time when I restart the node, the node can not be
> > startup.
> >
> > Here is the exception.
> > ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
> >
> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
> > took 2800ms
> > 2016-06-20 19:10:18,313 [main] INFO  o.a.d.e.s.s.PersistentStoreRegistry
> -
> > Using the configured PStoreProvider class:
> >
> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
> > 2016-06-20 19:10:19,221 [main] INFO  o.apache.drill.exec.server.Drillbit
> -
> > Construction completed (1529 ms).
> > 2016-06-20 19:10:31,136 [main] WARN  o.apache.drill.exec.server.Drillbit
> -
> > Failure on close()
> > java.lang.NullPointerException: null
> > at
> > org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
> > ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at
> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at
> > org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> > ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> > [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> > 2016-06-20 19:10:31,137 [main] INFO  o.apache.drill.exec.server.Drillbit
> -
> > Shutdown completed (1914 ms).
> >
> > I did nothing and start it at next day, then it can startup.
> >
> > 2016-06-21 9:48 GMT+08:00 qiang li :
> >
> >> Hi Aman,
> >>
> >> I did not fully test with the old version.
> >>
> >> Cloud you please help me create the JIRA issue,  I think my account have
> >> not the privilege, my account is griffinli and can not find the place to
> >> create new issue. Below is the explain detail for the same SQL in
> different
> >> nodes of cluster.
> >>
> >>
> >> This is the correct plan which only have two nodes:
> >> 0: jdbc:drill:zk=xxx:> explain plan for select
> >> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
> >> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from
> hbase.`offers_nation_idx` as
> >> `nation` join hbase.offers_ref0 as `ref0` on
> >>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where
> `nation`.row_key
> >>  > '0br' and `nation`.row_key  < '0bs' limit 10;
> >> +--+--+
> >> | text | json |
> >> +--+--+
> >> | 00-00Screen
> >> 00-01  Project(uid=[$0], v=[$1])
> >> 00-02SelectionVectorRemover
> >> 00-03  Limit(fetch=[10])
> >> 00-04UnionExchange
> >> 01-01  SelectionVectorRemover
> >> 01-02Limit(fetch=[10])
> >> 01-03
> Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
> >> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
> >> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
> >> row_key0=[$0], v0=[$1], $f2=[$2])
> >> 01-05  HashJoin(condition=[=($2, $5)],
> >> joinType=[inner])
> >> 01-07Project(row_key=[$0], v=[$1],
> >> $f2=[BYTE_SUBSTR($0, -8, 8)])
> >> 01-09  Scan(groupscan=[HBaseGroupScan
> >> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
> >> stopRow=null, filter=null], columns=[`*`]]])
> >> 01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
> >> 01-08  *BroadcastExchange*
> >> 02-01Project(row_key=[$0], v=[$1],
> >> ITEM=[ITEM($1, 'v')])
> >> 02-02  Scan(groupscan=[HBaseGroupScan
> >> [HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
> >> startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
> >> (GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
> >> `v`.`v`]]])
> >>
> >>
> >> This is the plan that fails which have more than 5 nodes:
> >> 0: jdbc:drill:zk=xxx:> explain plan for select
> >> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
> >>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-21 Thread Aditya

There should be another stack trace in drillbit.out when this happens,
could you please check that file.

On Mon, Jun 20, 2016 at 8:14 PM, qiang li  wrote:

> Another issue is some time when I restart the node, the node can not be
> startup.
>
> Here is the exception.
> ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
> took 2800ms
> 2016-06-20 19:10:18,313 [main] INFO  o.a.d.e.s.s.PersistentStoreRegistry -
> Using the configured PStoreProvider class:
> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
> 2016-06-20 19:10:19,221 [main] INFO  o.apache.drill.exec.server.Drillbit -
> Construction completed (1529 ms).
> 2016-06-20 19:10:31,136 [main] WARN  o.apache.drill.exec.server.Drillbit -
> Failure on close()
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
> ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> 2016-06-20 19:10:31,137 [main] INFO  o.apache.drill.exec.server.Drillbit -
> Shutdown completed (1914 ms).
>
> I did nothing and start it at next day, then it can startup.
>
> 2016-06-21 9:48 GMT+08:00 qiang li :
>
>> Hi Aman,
>>
>> I did not fully test with the old version.
>>
>> Cloud you please help me create the JIRA issue,  I think my account have
>> not the privilege, my account is griffinli and can not find the place to
>> create new issue. Below is the explain detail for the same SQL in different
>> nodes of cluster.
>>
>>
>> This is the correct plan which only have two nodes:
>> 0: jdbc:drill:zk=xxx:> explain plan for select
>> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
>> `nation` join hbase.offers_ref0 as `ref0` on
>>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>> +--+--+
>> | text | json |
>> +--+--+
>> | 00-00Screen
>> 00-01  Project(uid=[$0], v=[$1])
>> 00-02SelectionVectorRemover
>> 00-03  Limit(fetch=[10])
>> 00-04UnionExchange
>> 01-01  SelectionVectorRemover
>> 01-02Limit(fetch=[10])
>> 01-03  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
>> row_key0=[$0], v0=[$1], $f2=[$2])
>> 01-05  HashJoin(condition=[=($2, $5)],
>> joinType=[inner])
>> 01-07Project(row_key=[$0], v=[$1],
>> $f2=[BYTE_SUBSTR($0, -8, 8)])
>> 01-09  Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
>> stopRow=null, filter=null], columns=[`*`]]])
>> 01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
>> 01-08  *BroadcastExchange*
>> 02-01Project(row_key=[$0], v=[$1],
>> ITEM=[ITEM($1, 'v')])
>> 02-02  Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
>> startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
>> (GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
>> `v`.`v`]]])
>>
>>
>> This is the plan that fails which have more than 5 nodes:
>> 0: jdbc:drill:zk=xxx:> explain plan for select
>> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
>> `nation` join hbase.offers_ref0 as `ref0` on
>>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>> +--+--+
>> | text | json |
>> +--+--+
>> | 00-00Screen
>> 00-01  Project(uid=[$0], v=[$1])
>> 00-02SelectionVectorRemover
>> 00-03  Limit(fetch=[10])
>> 00-04UnionExchange
>> 01-01  SelectionVectorRemover
>> 01-02Limit(fetch=[10])
>> 01-03  Project(uid=[CONVERT

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-20 Thread qiang li

Another issue is some time when I restart the node, the node can not be
startup.

Here is the exception.
ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
took 2800ms
2016-06-20 19:10:18,313 [main] INFO  o.a.d.e.s.s.PersistentStoreRegistry -
Using the configured PStoreProvider class:
'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
2016-06-20 19:10:19,221 [main] INFO  o.apache.drill.exec.server.Drillbit -
Construction completed (1529 ms).
2016-06-20 19:10:31,136 [main] WARN  o.apache.drill.exec.server.Drillbit -
Failure on close()
java.lang.NullPointerException: null
at
org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at
org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
2016-06-20 19:10:31,137 [main] INFO  o.apache.drill.exec.server.Drillbit -
Shutdown completed (1914 ms).

I did nothing and start it at next day, then it can startup.

2016-06-21 9:48 GMT+08:00 qiang li :

> Hi Aman,
>
> I did not fully test with the old version.
>
> Cloud you please help me create the JIRA issue,  I think my account have
> not the privilege, my account is griffinli and can not find the place to
> create new issue. Below is the explain detail for the same SQL in different
> nodes of cluster.
>
>
> This is the correct plan which only have two nodes:
> 0: jdbc:drill:zk=xxx:> explain plan for select
> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
> `nation` join hbase.offers_ref0 as `ref0` on
>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>  > '0br' and `nation`.row_key  < '0bs' limit 10;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(uid=[$0], v=[$1])
> 00-02SelectionVectorRemover
> 00-03  Limit(fetch=[10])
> 00-04UnionExchange
> 01-01  SelectionVectorRemover
> 01-02Limit(fetch=[10])
> 01-03  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
> row_key0=[$0], v0=[$1], $f2=[$2])
> 01-05  HashJoin(condition=[=($2, $5)],
> joinType=[inner])
> 01-07Project(row_key=[$0], v=[$1],
> $f2=[BYTE_SUBSTR($0, -8, 8)])
> 01-09  Scan(groupscan=[HBaseGroupScan
> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
> stopRow=null, filter=null], columns=[`*`]]])
> 01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
> 01-08  *BroadcastExchange*
> 02-01Project(row_key=[$0], v=[$1],
> ITEM=[ITEM($1, 'v')])
> 02-02  Scan(groupscan=[HBaseGroupScan
> [HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
> startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
> (GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
> `v`.`v`]]])
>
>
> This is the plan that fails which have more than 5 nodes:
> 0: jdbc:drill:zk=xxx:> explain plan for select
> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
> `nation` join hbase.offers_ref0 as `ref0` on
>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>  > '0br' and `nation`.row_key  < '0bs' limit 10;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(uid=[$0], v=[$1])
> 00-02SelectionVectorRemover
> 00-03  Limit(fetch=[10])
> 00-04UnionExchange
> 01-01  SelectionVectorRemover
> 01-02Limit(fetch=[10])
> 01-03  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
> 01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
> row_key0=[$0], v0=[$1], $f2=[$2])
> 01-05  HashJoin(condition=[=($2, $5)],
> joinType=[inner])
> 01-07Pr

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-20 Thread qiang li

Hi Aman,

I did not fully test with the old version.

Cloud you please help me create the JIRA issue,  I think my account have
not the privilege, my account is griffinli and can not find the place to
create new issue. Below is the explain detail for the same SQL in different
nodes of cluster.


This is the correct plan which only have two nodes:
0: jdbc:drill:zk=xxx:> explain plan for select
CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
`nation` join hbase.offers_ref0 as `ref0` on
 BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
 > '0br' and `nation`.row_key  < '0bs' limit 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(uid=[$0], v=[$1])
00-02SelectionVectorRemover
00-03  Limit(fetch=[10])
00-04UnionExchange
01-01  SelectionVectorRemover
01-02Limit(fetch=[10])
01-03  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
-8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
row_key0=[$0], v0=[$1], $f2=[$2])
01-05  HashJoin(condition=[=($2, $5)], joinType=[inner])
01-07Project(row_key=[$0], v=[$1],
$f2=[BYTE_SUBSTR($0, -8, 8)])
01-09  Scan(groupscan=[HBaseGroupScan
[HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
stopRow=null, filter=null], columns=[`*`]]])
01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
01-08  *BroadcastExchange*
02-01Project(row_key=[$0], v=[$1],
ITEM=[ITEM($1, 'v')])
02-02  Scan(groupscan=[HBaseGroupScan
[HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
(GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
`v`.`v`]]])


This is the plan that fails which have more than 5 nodes:
0: jdbc:drill:zk=xxx:> explain plan for select
CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
`nation` join hbase.offers_ref0 as `ref0` on
 BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
 > '0br' and `nation`.row_key  < '0bs' limit 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(uid=[$0], v=[$1])
00-02SelectionVectorRemover
00-03  Limit(fetch=[10])
00-04UnionExchange
01-01  SelectionVectorRemover
01-02Limit(fetch=[10])
01-03  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
-8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
01-04Project(row_key=[$3], v=[$4], ITEM=[$5],
row_key0=[$0], v0=[$1], $f2=[$2])
01-05  HashJoin(condition=[=($2, $5)], joinType=[inner])
01-07Project(row_key=[$0], v=[$1], $f2=[$2])
01-09  *HashToRandomExchange*(dist0=[[$2]])
02-01UnorderedMuxExchange
04-01  Project(row_key=[$0], v=[$1], $f2=[$2],
E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($2)])
04-02Project(row_key=[$0], v=[$1],
$f2=[BYTE_SUBSTR($0, -8, 8)])
04-03  Scan(groupscan=[HBaseGroupScan
[HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
stopRow=null, filter=null], columns=[`*`]]])
01-06Project(row_key0=[$0], v0=[$1], ITEM=[$2])
01-08  Project(row_key=[$0], v=[$1], ITEM=[$2])
01-10*HashToRandomExchange*(dist0=[[$2]])
03-01  UnorderedMuxExchange
05-01Project(row_key=[$0], v=[$1],
ITEM=[$2], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($2)])
05-02  Project(row_key=[$0], v=[$1],
ITEM=[ITEM($1, 'v')])
05-03Scan(groupscan=[HBaseGroupScan
[HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
(GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
`v`.`v`]]])

The difference is use *BroadcastExchange *and  *HashToRandomExchange.*

You can create the JIRA and send me the link .

Thanks.


2016-06-20 23:44 GMT+08:00 Aman Sinha :

> Hi Qiang,
> were you seeing this same issue with the prior HBase version also ?  (I
> would think this is not a regression).  It would be best to create a new
> JIRA and attach the EXPLAIN plans for the successful and failed runs.  With
> more nodes some minor fragments of the hash join may be getting empty input
> batches and I am guessing that has something to do with the
> SchemaChangeException.   Someone would need to debu

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-20 Thread qiang li

Thanks Aditya.

By the way, I found another issue.

Let say I have two tables.

offers_ref0 : rowkey salt(1byte)+long uid(8 byte ) , family: v,  qualifier:
v(string)
offers_nation_idx: rowkey salt(1byte) + string, family:v, qualifier: v(long
8 byte)

there is the SQL:

select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
`nation` join hbase.offers_ref0 as `ref0` on
CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') =
CONVERT_FROM(nation.`v`.`v`,'BIGINT_BE') where `nation`.row_key  > '0br'
and `nation`.row_key  < '0bs' limit 10

When I execute the query with single node or less than 5 nodes, its working
good. But when I execute it in cluster which have about 14 nodes, its throw
a exception:

First time will throw this exception:
*Caused by: java.sql.SQLException: SYSTEM ERROR: SchemaChangeException:
Hash join does not support schema changes*

Then if I query again, it will always throw below exception:
*Query Failed: An Error Occurred*
*org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
IllegalStateException: Failure while reading vector. Expected vector class
of org.apache.drill.exec.vector.NullableIntVector but was holding vector
class org.apache.drill.exec.vector.complex.MapVector, field=
v(MAP:REQUIRED)[v(VARBINARY:OPTIONAL)[$bits$(UINT1:REQUIRED),
v(VARBINARY:OPTIONAL)[$offsets$(UINT4:REQUIRED)]]] Fragment 12:4 [Error Id:
06c6eae4-0822-4714-b0bf-a6e04ebfec79 on xxx:31010]*

Its very strange, and I do not know how to solve it.
I tried add node to the cluster one by one, it will reproduce when I added
5 nodes. Can anyone help me solve this issue?




2016-06-17 4:39 GMT+08:00 Aditya :

> https://issues.apache.org/jira/browse/DRILL-4727
>
> On Thu, Jun 16, 2016 at 11:39 AM, Aman Sinha  wrote:
>
>> Qiang/Aditya can you create a JIRA for this and mark it for 1.7.  thanks.
>>
>> On Thu, Jun 16, 2016 at 11:25 AM, Aditya  wrote:
>>
>> > Thanks for reporting, I'm looking into it and will post a patch soon.
>> >
>> > On Wed, Jun 15, 2016 at 7:27 PM, qiang li  wrote:
>> >
>> > > Hi Aditya,
>> > >
>> > > I tested the latest version and got this exception and the drillbit
>> fail
>> > > to startup .
>> > >
>> > > Exception in thread "main" java.lang.NoSuchMethodError:
>> > > io.netty.util.UniqueName.(Ljava/lang/String;)V
>> > > at
>> io.netty.channel.ChannelOption.(ChannelOption.java:136)
>> > > at
>> io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
>> > > at
>> io.netty.channel.ChannelOption.(ChannelOption.java:42)
>> > > at
>> > > org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
>> > > at
>> > > org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
>> > > at
>> > >
>> org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
>> > > at
>> > org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
>> > > at
>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
>> > > at
>> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
>> > > at
>> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>> > >
>> > > It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar,
>> the
>> > > drill can startup. I think there have some package dependency version
>> > > issue, do you think so ?
>> > >
>> > >
>> > >
>> > > 2016-06-15 8:14 GMT+08:00 Aditya :
>> > >
>> > >> HBase 1.x support has been merged and is available in latest
>> > >> 1.7.0-SNAPSHOT
>> > >> builds.
>> > >>
>> > >> On Wed, Jun 1, 2016 at 1:23 PM, Aditya 
>> wrote:
>> > >>
>> > >> > Thanks Jacques for promptly reviewing my long series of patches!
>> > >> >
>> > >> > I'm planning to merge the HBase 1.x support some time in next 48
>> > hours.
>> > >> >
>> > >> > If anyone else is interested and willing, please review the latest
>> > patch
>> > >> > here[1].
>> > >> >
>> > >> > aditya...
>> > >> >
>> > >> > [1] https://github.com/apache/drill/pull/443/files
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-20 Thread Aman Sinha

Hi Qiang,
were you seeing this same issue with the prior HBase version also ?  (I
would think this is not a regression).  It would be best to create a new
JIRA and attach the EXPLAIN plans for the successful and failed runs.  With
more nodes some minor fragments of the hash join may be getting empty input
batches and I am guessing that has something to do with the
SchemaChangeException.   Someone would need to debug once you create the
JIRA with relevant details.

-Aman

On Mon, Jun 20, 2016 at 5:13 AM, qiang li  wrote:

> Thanks Aditya.
>
> By the way, I found another issue.
>
> Let say I have two tables.
>
> offers_ref0 : rowkey salt(1byte)+long uid(8 byte ) , family: v,  qualifier:
> v(string)
> offers_nation_idx: rowkey salt(1byte) + string, family:v, qualifier: v(long
> 8 byte)
>
> there is the SQL:
>
> select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
> `nation` join hbase.offers_ref0 as `ref0` on
> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') =
> CONVERT_FROM(nation.`v`.`v`,'BIGINT_BE') where `nation`.row_key  > '0br'
> and `nation`.row_key  < '0bs' limit 10
>
> When I execute the query with single node or less than 5 nodes, its working
> good. But when I execute it in cluster which have about 14 nodes, its throw
> a exception:
>
> First time will throw this exception:
> *Caused by: java.sql.SQLException: SYSTEM ERROR: SchemaChangeException:
> Hash join does not support schema changes*
>
> Then if I query again, it will always throw below exception:
> *Query Failed: An Error Occurred*
> *org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> IllegalStateException: Failure while reading vector. Expected vector class
> of org.apache.drill.exec.vector.NullableIntVector but was holding vector
> class org.apache.drill.exec.vector.complex.MapVector, field=
> v(MAP:REQUIRED)[v(VARBINARY:OPTIONAL)[$bits$(UINT1:REQUIRED),
> v(VARBINARY:OPTIONAL)[$offsets$(UINT4:REQUIRED)]]] Fragment 12:4 [Error Id:
> 06c6eae4-0822-4714-b0bf-a6e04ebfec79 on xxx:31010]*
>
> Its very strange, and I do not know how to solve it.
> I tried add node to the cluster one by one, it will reproduce when I added
> 5 nodes. Can anyone help me solve this issue?
>
>
>
>
> 2016-06-17 4:39 GMT+08:00 Aditya :
>
> > https://issues.apache.org/jira/browse/DRILL-4727
> >
> > On Thu, Jun 16, 2016 at 11:39 AM, Aman Sinha 
> wrote:
> >
> >> Qiang/Aditya can you create a JIRA for this and mark it for 1.7.
> thanks.
> >>
> >> On Thu, Jun 16, 2016 at 11:25 AM, Aditya 
> wrote:
> >>
> >> > Thanks for reporting, I'm looking into it and will post a patch soon.
> >> >
> >> > On Wed, Jun 15, 2016 at 7:27 PM, qiang li 
> wrote:
> >> >
> >> > > Hi Aditya,
> >> > >
> >> > > I tested the latest version and got this exception and the drillbit
> >> fail
> >> > > to startup .
> >> > >
> >> > > Exception in thread "main" java.lang.NoSuchMethodError:
> >> > > io.netty.util.UniqueName.(Ljava/lang/String;)V
> >> > > at
> >> io.netty.channel.ChannelOption.(ChannelOption.java:136)
> >> > > at
> >> io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
> >> > > at
> >> io.netty.channel.ChannelOption.(ChannelOption.java:42)
> >> > > at
> >> > > org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
> >> > > at
> >> > > org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
> >> > > at
> >> > >
> >>
> org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
> >> > > at
> >> > org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
> >> > > at
> >> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> >> > > at
> >> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> >> > > at
> >> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> >> > >
> >> > > It will working if I remove
> jars/3rdparty/netty-all-4.0.23.Final.jar,
> >> the
> >> > > drill can startup. I think there have some package dependency
> version
> >> > > issue, do you think so ?
> >> > >
> >> > >
> >> > >
> >> > > 2016-06-15 8:14 GMT+08:00 Aditya :
> >> > >
> >> > >> HBase 1.x support has been merged and is available in latest
> >> > >> 1.7.0-SNAPSHOT
> >> > >> builds.
> >> > >>
> >> > >> On Wed, Jun 1, 2016 at 1:23 PM, Aditya 
> >> wrote:
> >> > >>
> >> > >> > Thanks Jacques for promptly reviewing my long series of patches!
> >> > >> >
> >> > >> > I'm planning to merge the HBase 1.x support some time in next 48
> >> > hours.
> >> > >> >
> >> > >> > If anyone else is interested and willing, please review the
> latest
> >> > patch
> >> > >> > here[1].
> >> > >> >
> >> > >> > aditya...
> >> > >> >
> >> > >> > [1] https://github.com/apache/drill/pull/443/files
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-16 Thread Aditya

https://issues.apache.org/jira/browse/DRILL-4727

On Thu, Jun 16, 2016 at 11:39 AM, Aman Sinha  wrote:

> Qiang/Aditya can you create a JIRA for this and mark it for 1.7.  thanks.
>
> On Thu, Jun 16, 2016 at 11:25 AM, Aditya  wrote:
>
> > Thanks for reporting, I'm looking into it and will post a patch soon.
> >
> > On Wed, Jun 15, 2016 at 7:27 PM, qiang li  wrote:
> >
> > > Hi Aditya,
> > >
> > > I tested the latest version and got this exception and the drillbit
> fail
> > > to startup .
> > >
> > > Exception in thread "main" java.lang.NoSuchMethodError:
> > > io.netty.util.UniqueName.(Ljava/lang/String;)V
> > > at
> io.netty.channel.ChannelOption.(ChannelOption.java:136)
> > > at
> io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
> > > at
> io.netty.channel.ChannelOption.(ChannelOption.java:42)
> > > at
> > > org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
> > > at
> > > org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
> > > at
> > >
> org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
> > > at
> > org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
> > > at
> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> > > at
> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> > > at
> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> > >
> > > It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar,
> the
> > > drill can startup. I think there have some package dependency version
> > > issue, do you think so ?
> > >
> > >
> > >
> > > 2016-06-15 8:14 GMT+08:00 Aditya :
> > >
> > >> HBase 1.x support has been merged and is available in latest
> > >> 1.7.0-SNAPSHOT
> > >> builds.
> > >>
> > >> On Wed, Jun 1, 2016 at 1:23 PM, Aditya 
> wrote:
> > >>
> > >> > Thanks Jacques for promptly reviewing my long series of patches!
> > >> >
> > >> > I'm planning to merge the HBase 1.x support some time in next 48
> > hours.
> > >> >
> > >> > If anyone else is interested and willing, please review the latest
> > patch
> > >> > here[1].
> > >> >
> > >> > aditya...
> > >> >
> > >> > [1] https://github.com/apache/drill/pull/443/files
> > >> >
> > >>
> > >
> > >
> >
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-16 Thread Aman Sinha

Qiang/Aditya can you create a JIRA for this and mark it for 1.7.  thanks.

On Thu, Jun 16, 2016 at 11:25 AM, Aditya  wrote:

> Thanks for reporting, I'm looking into it and will post a patch soon.
>
> On Wed, Jun 15, 2016 at 7:27 PM, qiang li  wrote:
>
> > Hi Aditya,
> >
> > I tested the latest version and got this exception and the drillbit fail
> > to startup .
> >
> > Exception in thread "main" java.lang.NoSuchMethodError:
> > io.netty.util.UniqueName.(Ljava/lang/String;)V
> > at io.netty.channel.ChannelOption.(ChannelOption.java:136)
> > at io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
> > at io.netty.channel.ChannelOption.(ChannelOption.java:42)
> > at
> > org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
> > at
> > org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
> > at
> > org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
> > at
> org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> >
> > It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar, the
> > drill can startup. I think there have some package dependency version
> > issue, do you think so ?
> >
> >
> >
> > 2016-06-15 8:14 GMT+08:00 Aditya :
> >
> >> HBase 1.x support has been merged and is available in latest
> >> 1.7.0-SNAPSHOT
> >> builds.
> >>
> >> On Wed, Jun 1, 2016 at 1:23 PM, Aditya  wrote:
> >>
> >> > Thanks Jacques for promptly reviewing my long series of patches!
> >> >
> >> > I'm planning to merge the HBase 1.x support some time in next 48
> hours.
> >> >
> >> > If anyone else is interested and willing, please review the latest
> patch
> >> > here[1].
> >> >
> >> > aditya...
> >> >
> >> > [1] https://github.com/apache/drill/pull/443/files
> >> >
> >>
> >
> >
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-16 Thread Aditya

Thanks for reporting, I'm looking into it and will post a patch soon.

On Wed, Jun 15, 2016 at 7:27 PM, qiang li  wrote:

> Hi Aditya,
>
> I tested the latest version and got this exception and the drillbit fail
> to startup .
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> io.netty.util.UniqueName.(Ljava/lang/String;)V
> at io.netty.channel.ChannelOption.(ChannelOption.java:136)
> at io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
> at io.netty.channel.ChannelOption.(ChannelOption.java:42)
> at
> org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
> at
> org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
> at
> org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>
> It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar, the
> drill can startup. I think there have some package dependency version
> issue, do you think so ?
>
>
>
> 2016-06-15 8:14 GMT+08:00 Aditya :
>
>> HBase 1.x support has been merged and is available in latest
>> 1.7.0-SNAPSHOT
>> builds.
>>
>> On Wed, Jun 1, 2016 at 1:23 PM, Aditya  wrote:
>>
>> > Thanks Jacques for promptly reviewing my long series of patches!
>> >
>> > I'm planning to merge the HBase 1.x support some time in next 48 hours.
>> >
>> > If anyone else is interested and willing, please review the latest patch
>> > here[1].
>> >
>> > aditya...
>> >
>> > [1] https://github.com/apache/drill/pull/443/files
>> >
>>
>
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-15 Thread qiang li

Also when I query hbase, its throw exception :
2016-06-16 10:29:51,788 [289df0e3-05dc-f882-8c51-20dd0f98235b:frag:0:0]
ERROR o.a.drill.exec.compile.MergeAdapter - Failure while merging classes.
java.lang.RuntimeException: Error at instruction 370: Expected an object
reference, but found . doEval(II)V
at
org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(CheckMethodAdapter.java:463)
~[asm-debug-all-5.0.3.jar:5.0.3]

But the query is success. Does this exception will affect the query
process,  for example will it affect the query speed or data integrity?

2016-06-16 10:27 GMT+08:00 qiang li :

> Hi Aditya,
>
> I tested the latest version and got this exception and the drillbit fail
> to startup .
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> io.netty.util.UniqueName.(Ljava/lang/String;)V
> at io.netty.channel.ChannelOption.(ChannelOption.java:136)
> at io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
> at io.netty.channel.ChannelOption.(ChannelOption.java:42)
> at
> org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
> at
> org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
> at
> org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
> at org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>
> It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar, the
> drill can startup. I think there have some package dependency version
> issue, do you think so ?
>
>
>
> 2016-06-15 8:14 GMT+08:00 Aditya :
>
>> HBase 1.x support has been merged and is available in latest
>> 1.7.0-SNAPSHOT
>> builds.
>>
>> On Wed, Jun 1, 2016 at 1:23 PM, Aditya  wrote:
>>
>> > Thanks Jacques for promptly reviewing my long series of patches!
>> >
>> > I'm planning to merge the HBase 1.x support some time in next 48 hours.
>> >
>> > If anyone else is interested and willing, please review the latest patch
>> > here[1].
>> >
>> > aditya...
>> >
>> > [1] https://github.com/apache/drill/pull/443/files
>> >
>>
>
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-15 Thread qiang li

Hi Aditya,

I tested the latest version and got this exception and the drillbit fail to
startup .

Exception in thread "main" java.lang.NoSuchMethodError:
io.netty.util.UniqueName.(Ljava/lang/String;)V
at io.netty.channel.ChannelOption.(ChannelOption.java:136)
at io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
at io.netty.channel.ChannelOption.(ChannelOption.java:42)
at org.apache.drill.exec.rpc.BasicServer.(BasicServer.java:63)
at
org.apache.drill.exec.rpc.user.UserServer.(UserServer.java:74)
at
org.apache.drill.exec.service.ServiceEngine.(ServiceEngine.java:78)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:108)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)

It will working if I remove jars/3rdparty/netty-all-4.0.23.Final.jar, the
drill can startup. I think there have some package dependency version
issue, do you think so ?



2016-06-15 8:14 GMT+08:00 Aditya :

> HBase 1.x support has been merged and is available in latest 1.7.0-SNAPSHOT
> builds.
>
> On Wed, Jun 1, 2016 at 1:23 PM, Aditya  wrote:
>
> > Thanks Jacques for promptly reviewing my long series of patches!
> >
> > I'm planning to merge the HBase 1.x support some time in next 48 hours.
> >
> > If anyone else is interested and willing, please review the latest patch
> > here[1].
> >
> > aditya...
> >
> > [1] https://github.com/apache/drill/pull/443/files
> >
>

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-14 Thread Aditya

HBase 1.x support has been merged and is available in latest 1.7.0-SNAPSHOT
builds.

On Wed, Jun 1, 2016 at 1:23 PM, Aditya  wrote:

> Thanks Jacques for promptly reviewing my long series of patches!
>
> I'm planning to merge the HBase 1.x support some time in next 48 hours.
>
> If anyone else is interested and willing, please review the latest patch
> here[1].
>
> aditya...
>
> [1] https://github.com/apache/drill/pull/443/files
>

DRILL-4199: Add Support for HBase 1.X - planning to merge

2016-06-01 Thread Aditya

Thanks Jacques for promptly reviewing my long series of patches!

I'm planning to merge the HBase 1.x support some time in next 48 hours.

If anyone else is interested and willing, please review the latest patch
here[1].

aditya...

[1] https://github.com/apache/drill/pull/443/files

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

Re: DRILL-4199: Add Support for HBase 1.X - planning to merge

DRILL-4199: Add Support for HBase 1.X - planning to merge

16 matches

Site Navigation

Mail list logo

Footer information