Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Jim Huang]

Good to see this patch is already in action, so we can avoid others
spending time on this once this is committed.


On Mon, Apr 7, 2014 at 7:09 PM, liushaohui  wrote:

> Please see https://issues.apache.org/jira/browse/HBASE-10692
>
> This issue is about this problem.
>
> May someone help to push it ? Thanks.
>
> - Shaohui Liu
>
>
>
> On 04/08/2014 05:48 AM, Ted Yu wrote:
>
>> You are talking about this method, right ?
>>
>>public static void initTableMapperJob(List scans,
>>Class mapper,
>>Class outputKeyClass,
>>Class outputValueClass, Job job,
>>boolean addDependencyJars) throws IOException {
>>
>> Looks like it should call initCredentials(job) as well.
>>
>>
>> On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:
>>
>>  I am looking at the 0.94 branch and I have noticed that the overloaded
>>> methods for "initTableMapperJob" behaves differently in that one of them
>>> calls the "initCredentials(job);" while the others don't.  For people who
>>> are using Kerberos security on their cluster, the initCredentials(job)
>>> sets
>>> up the HBase security tokens for all the mapper tasks only for that
>>> overloaded method.  Is there a specific reason why this could be
>>> intentional?  Otherwise, I would like to create a new JIRA to see if I
>>> can
>>> work on this as a newbie issue.
>>>
>>> Thanks for any pointer.
>>> Jim
>>>
>>>
>

[jira] [Created] (HBASE-10928) Add timerange support to thrift1

[stack (JIRA)]

stack created HBASE-10928:
-

 Summary: Add timerange support to thrift1
 Key: HBASE-10928
 URL: https://issues.apache.org/jira/browse/HBASE-10928
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Minor


thrift2 supports it.  thrift1 should too. Thrift1 hard-coding the 0/min in the 
left side of range.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-10927) Add undrain option to RollingRestart

[Rishit Shroff (JIRA)]

Rishit Shroff created HBASE-10927:
-

 Summary: Add undrain option to RollingRestart
 Key: HBASE-10927
 URL: https://issues.apache.org/jira/browse/HBASE-10927
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.89-fb
Reporter: Rishit Shroff
Priority: Minor


Currently, there is no way to undrain the region server with the same regions 
it was drained to being with. The reason being that we do not persist list of 
regions we drained.

Creating this jira to take a file with the list of regions to undrain as an 
input and drain the regions back to the regionserver.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[liushaohui]

Please see https://issues.apache.org/jira/browse/HBASE-10692

This issue is about this problem.

May someone help to push it ? Thanks.

- Shaohui Liu


On 04/08/2014 05:48 AM, Ted Yu wrote:

You are talking about this method, right ?

   public static void initTableMapperJob(List scans,
   Class mapper,
   Class outputKeyClass,
   Class outputValueClass, Job job,
   boolean addDependencyJars) throws IOException {

Looks like it should call initCredentials(job) as well.


On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:


I am looking at the 0.94 branch and I have noticed that the overloaded
methods for "initTableMapperJob" behaves differently in that one of them
calls the "initCredentials(job);" while the others don't.  For people who
are using Kerberos security on their cluster, the initCredentials(job) sets
up the HBase security tokens for all the mapper tasks only for that
overloaded method.  Is there a specific reason why this could be
intentional?  Otherwise, I would like to create a new JIRA to see if I can
work on this as a newbie issue.

Thanks for any pointer.
Jim

Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Jim Huang]

Yes, that's precisely the method that should get initCredentials(job) as
well.
All the other overloaded methods with table are covered.


On Mon, Apr 7, 2014 at 2:48 PM, Ted Yu  wrote:

> You are talking about this method, right ?
>
>   public static void initTableMapperJob(List scans,
>   Class mapper,
>   Class outputKeyClass,
>   Class outputValueClass, Job job,
>   boolean addDependencyJars) throws IOException {
>
> Looks like it should call initCredentials(job) as well.
>
>
> On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:
>
> > I am looking at the 0.94 branch and I have noticed that the overloaded
> > methods for "initTableMapperJob" behaves differently in that one of them
> > calls the "initCredentials(job);" while the others don't.  For people who
> > are using Kerberos security on their cluster, the initCredentials(job)
> sets
> > up the HBase security tokens for all the mapper tasks only for that
> > overloaded method.  Is there a specific reason why this could be
> > intentional?  Otherwise, I would like to create a new JIRA to see if I
> can
> > work on this as a newbie issue.
> >
> > Thanks for any pointer.
> > Jim
> >
>

[jira] [Created] (HBASE-10926) Use global procedure to flush table memstore cache

[Jerry He (JIRA)]

Jerry He created HBASE-10926:


 Summary: Use global procedure to flush table memstore cache
 Key: HBASE-10926
 URL: https://issues.apache.org/jira/browse/HBASE-10926
 Project: HBase
  Issue Type: Improvement
  Components: Admin
Affects Versions: 0.98.1, 0.96.2
Reporter: Jerry He
 Fix For: 0.99.0


Currently, user can trigger table flush through hbase shell or HBaseAdmin API.  
To flush the table cache, each region server hosting the regions is contacted 
and flushed sequentially, which is less inefficient.
In HBase snapshot global procedure is used to coordinate and flush the regions 
in a distributed way.
Let's provide a distributed table flush for general use.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Bringing consensus based strong consistency into HBase

[Konstantin Boudnik]

Guys,

As some of you might have noticed there is a number of new JIRAs opened
recently that are aiming on abstracting and separating ZK out of HBase guts
and making it an implementation detail, rather than a center of attention
for some parts of the HBase.

I would like to send around a short document written by Mikhail Antonov that
is trying to clarify a couple of points about this whole effort.
People are asking me about this off-line, so I decided to send this around so
we can have a lively and wider discussion about this initiative and the
development.

Here's the JIRA and the link to the PDF document.
  https://issues.apache.org/jira/browse/HBASE-10866
  https://issues.apache.org/jira/secure/attachment/12637957/HBaseConsensus.pdf

The umbrella JIRA for this effort is here
  https://issues.apache.org/jira/browse/HBASE-10909

-- 
Regardsk
  Cos

Re: Hadoop Summit EU

[Mikhail Antonov]

Well...Since that was mentioned anyway, allow me a tiny
correction/clarification.. :)

It's ConsensusNode, not ConsistencyNode, and it's not really custom Paxos
implementation, it's more like interface for coordination service atop
standard NameNode, which may be backed by any consensus library/algorithm,
be it variation of Paxos, ZooKeeper/ZAB, Raft or anything else. The
consensus API itself (ConsensusNode code) and ZooKeeper-based
implementation of consensus protocol is going to be open-sourced (we're
working on it), and once it's out, consensus libraries authors are welcome
to start integration with their libs too.

Regarding HBase - that's actually what's being developed under HBASE-10909,
HBASE-10866 and referenced jiras (everybody interested is welcome to
discuss/feedback).

-Mikhail


2014-04-07 11:36 GMT-07:00 Enis Söztutar :

> Ops sorry this was intented for internal lists. Apologies for any
> confusion.
>
> Enis
>
> On Monday, April 7, 2014, Enis Söztutar  wrote:
>
> > Me and Devaraj attended their talk on their solution for paxos based
> > namenode and HBase replication.
> >
> > They have two solutions, one for single datacenter, and the other multi
> DC
> > geo replication.
> >
> > For the namenode, there is a wrapper, called ConsistencyNode, that
> > basically gets the requests, replicate it via their consistency protocol
> to
> > other CNodes within the DC (paxos based) in the edit log. If the proposal
> > for this is accepted, the changes are made durable. However, from my
> > understanding, on the read side the client chooses only one replica to
> > read. The client decides to connect to one of the replica namenodes,
> which
> > means that it is not doing a paxos read. I think they also wrapped the
> > client, so that if it gets a FileNotFoundException or something similar,
> it
> > will retry on a different server. Also they track the last seen proposal
> id
> > as a transaction id for this as well from my understanding (so
> > read-what-you-write consistency maybe?). The full details of the
> > consistency was not clear to me from the presentation.
> > For their multi-DC replication, they are doing a similar thing, but the
> > data replication is not handled by paxos, only the namenode metadata. For
> > each datacenter, they have a target replication factor (can be set
> > differently for each DC, like 0 because of regulatory reasons). The
> > metadata of NN is replicated via a similar mechanism. The data
> replication
> > is async to the metadata replication though. When a block is finalized,
> the
> > CNode quorum on that particular DC, schedules a remote copy to one of the
> > datacenters. That copy job, copies the block with directly writing the
> > block from the datanode to a remote datanode. Then that remote DC block
> is
> > replicated to the target replication by that DC's CNode quorum. When the
> > target is reached, that DC will create another proposal about the data
> > replication being complete. So the state machine probably contains where
> > each data is replicated, but they were still mentioning the client
> getting
> > DataNotReplicatedException or something.
> >
> > Their work on HBase is still WIP. I do not remember much details on the
> > protocol, except it uses the same replication protocol (their "patented"
> > paxos based replication).
> >
> > Of course the devil is in the details. I did not get that from the
> > presentation.
> >
> > As a side note, Doug when asked, was saying that they are cooking
> > something for backups, so maybe their "secret project" also contains
> > multi-DC consistent state?
> >
> > Enis
> >
> >
> > On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu  
> > > wrote:
> >
> >> Enis:
> >> There was a talk by Konstantin Boudnik<
> http://hadoopsummit.org/amsterdam/speakers/#konstantin-boudnik>
> >> .
> >>
> >> Any interesting material from his presentation ?
> >>
> >> Cheers
> >>
> >
> >
>



-- 
Thanks,
Michael Antonov

Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Ted Yu]

Looking at references to this method, it is used by MultiTableInputFormat.

It seems the next step is to enable MultiTableInputFormat in secure cluster.

Cheers


On Mon, Apr 7, 2014 at 2:48 PM, Ted Yu  wrote:

> You are talking about this method, right ?
>
>   public static void initTableMapperJob(List scans,
>   Class mapper,
>   Class outputKeyClass,
>   Class outputValueClass, Job job,
>   boolean addDependencyJars) throws IOException {
>
> Looks like it should call initCredentials(job) as well.
>
>
> On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:
>
>> I am looking at the 0.94 branch and I have noticed that the overloaded
>> methods for "initTableMapperJob" behaves differently in that one of them
>> calls the "initCredentials(job);" while the others don't.  For people who
>> are using Kerberos security on their cluster, the initCredentials(job)
>> sets
>> up the HBase security tokens for all the mapper tasks only for that
>> overloaded method.  Is there a specific reason why this could be
>> intentional?  Otherwise, I would like to create a new JIRA to see if I can
>> work on this as a newbie issue.
>>
>> Thanks for any pointer.
>> Jim
>>
>
>

Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Nick Dimiduk]

I vaguely recall a ticket being raised and committed for this problem, but
the bug is clearly present and I'm not finding the issue. Maybe @larsh
recalls?


On Mon, Apr 7, 2014 at 2:48 PM, Ted Yu  wrote:

> You are talking about this method, right ?
>
>   public static void initTableMapperJob(List scans,
>   Class mapper,
>   Class outputKeyClass,
>   Class outputValueClass, Job job,
>   boolean addDependencyJars) throws IOException {
>
> Looks like it should call initCredentials(job) as well.
>
>
> On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:
>
> > I am looking at the 0.94 branch and I have noticed that the overloaded
> > methods for "initTableMapperJob" behaves differently in that one of them
> > calls the "initCredentials(job);" while the others don't.  For people who
> > are using Kerberos security on their cluster, the initCredentials(job)
> sets
> > up the HBase security tokens for all the mapper tasks only for that
> > overloaded method.  Is there a specific reason why this could be
> > intentional?  Otherwise, I would like to create a new JIRA to see if I
> can
> > work on this as a newbie issue.
> >
> > Thanks for any pointer.
> > Jim
> >
>

Re: HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Ted Yu]

You are talking about this method, right ?

  public static void initTableMapperJob(List scans,
  Class mapper,
  Class outputKeyClass,
  Class outputValueClass, Job job,
  boolean addDependencyJars) throws IOException {

Looks like it should call initCredentials(job) as well.


On Mon, Apr 7, 2014 at 2:37 PM, Jim Huang  wrote:

> I am looking at the 0.94 branch and I have noticed that the overloaded
> methods for "initTableMapperJob" behaves differently in that one of them
> calls the "initCredentials(job);" while the others don't.  For people who
> are using Kerberos security on their cluster, the initCredentials(job) sets
> up the HBase security tokens for all the mapper tasks only for that
> overloaded method.  Is there a specific reason why this could be
> intentional?  Otherwise, I would like to create a new JIRA to see if I can
> work on this as a newbie issue.
>
> Thanks for any pointer.
> Jim
>

[jira] [Created] (HBASE-10925) Do not OOME, throw RowTooBigException instead

[stack (JIRA)]

stack created HBASE-10925:
-

 Summary: Do not OOME, throw RowTooBigException instead
 Key: HBASE-10925
 URL: https://issues.apache.org/jira/browse/HBASE-10925
 Project: HBase
  Issue Type: Bug
Reporter: stack


If 10M columns in a row, throw a RowTooBigException rather than OOME when 
Get'ing or Scanning w/o in-row scan flag set.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

HBase 0.94 - TableMapReduceUtil class, overloaded method "initTableMapperJob" behaves differently

[Jim Huang]

I am looking at the 0.94 branch and I have noticed that the overloaded
methods for "initTableMapperJob" behaves differently in that one of them
calls the "initCredentials(job);" while the others don't.  For people who
are using Kerberos security on their cluster, the initCredentials(job) sets
up the HBase security tokens for all the mapper tasks only for that
overloaded method.  Is there a specific reason why this could be
intentional?  Otherwise, I would like to create a new JIRA to see if I can
work on this as a newbie issue.

Thanks for any pointer.
Jim

[jira] [Resolved] (HBASE-10816) CatalogTracker abortable usage

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang resolved HBASE-10816.
-

Resolution: Won't Fix

I looked into it, and prefer not to do anything for now.

> CatalogTracker abortable usage
> --
>
> Key: HBASE-10816
> URL: https://issues.apache.org/jira/browse/HBASE-10816
> Project: HBase
>  Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
>
> In reviewing patch for HBASE-10569, Stack pointed out some existing issue 
> with CatalogTracker. I looked into it and I think the abortable usage can be 
> improved.
> * If ZK is null, when a new one is created, the abortable could be null. We 
> need consider this.
> * The throwableAborter is to abort the process in case some ZK exception in 
> MetaRegionTracker. In case the tracker is in a server, we don't need to do 
> this, we can use the server as the abortable. In case the tracker is in a 
> client, we can just abort the connection. Right?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Hadoop Summit EU

[Enis Söztutar]

Ops sorry this was intented for internal lists. Apologies for any
confusion.

Enis

On Monday, April 7, 2014, Enis Söztutar  wrote:

> Me and Devaraj attended their talk on their solution for paxos based
> namenode and HBase replication.
>
> They have two solutions, one for single datacenter, and the other multi DC
> geo replication.
>
> For the namenode, there is a wrapper, called ConsistencyNode, that
> basically gets the requests, replicate it via their consistency protocol to
> other CNodes within the DC (paxos based) in the edit log. If the proposal
> for this is accepted, the changes are made durable. However, from my
> understanding, on the read side the client chooses only one replica to
> read. The client decides to connect to one of the replica namenodes, which
> means that it is not doing a paxos read. I think they also wrapped the
> client, so that if it gets a FileNotFoundException or something similar, it
> will retry on a different server. Also they track the last seen proposal id
> as a transaction id for this as well from my understanding (so
> read-what-you-write consistency maybe?). The full details of the
> consistency was not clear to me from the presentation.
> For their multi-DC replication, they are doing a similar thing, but the
> data replication is not handled by paxos, only the namenode metadata. For
> each datacenter, they have a target replication factor (can be set
> differently for each DC, like 0 because of regulatory reasons). The
> metadata of NN is replicated via a similar mechanism. The data replication
> is async to the metadata replication though. When a block is finalized, the
> CNode quorum on that particular DC, schedules a remote copy to one of the
> datacenters. That copy job, copies the block with directly writing the
> block from the datanode to a remote datanode. Then that remote DC block is
> replicated to the target replication by that DC's CNode quorum. When the
> target is reached, that DC will create another proposal about the data
> replication being complete. So the state machine probably contains where
> each data is replicated, but they were still mentioning the client getting
> DataNotReplicatedException or something.
>
> Their work on HBase is still WIP. I do not remember much details on the
> protocol, except it uses the same replication protocol (their "patented"
> paxos based replication).
>
> Of course the devil is in the details. I did not get that from the
> presentation.
>
> As a side note, Doug when asked, was saying that they are cooking
> something for backups, so maybe their "secret project" also contains
> multi-DC consistent state?
>
> Enis
>
>
> On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu 
> 
> > wrote:
>
>> Enis:
>> There was a talk by Konstantin 
>> Boudnik
>> .
>>
>> Any interesting material from his presentation ?
>>
>> Cheers
>>
>
>

[jira] [Created] (HBASE-10924) [region_mover]: Adjust region_mover script to retry unloading a server a configurable number of times in case of region splits/merges

[Aleksandr Shulman (JIRA)]

Aleksandr Shulman created HBASE-10924:
-

 Summary: [region_mover]: Adjust region_mover script to retry 
unloading a server a configurable number of times in case of region 
splits/merges
 Key: HBASE-10924
 URL: https://issues.apache.org/jira/browse/HBASE-10924
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 0.94.15
Reporter: Aleksandr Shulman
Assignee: Aleksandr Shulman
 Fix For: 0.94.19


Observed behavior:
In about 5% of cases, my rolling upgrade tests fail because of stuck regions 
during a region server unload. My theory is that this occurs when region 
assignment information changes between the time the region list is generated, 
and the time when the region is to be moved.

An example of such a region information change is a split or merge.

Example:
Regionserver A has 100 regions (#0-#99). The balancer is turned off and the 
regionmover script is called to unload this regionserver. The regionmover 
script will generate the list of 100 regions to be moved and then proceed down 
that list, moving the regions off in series. However, there is a region, #84, 
that has split into two daughter regions while regions 0-83 were moved. The 
script will be stuck trying to move #84, timeout, and then the failure will 
bubble up (attempt 1 failed).

Proposed solution:
This specific failure mode should be caught and the region_mover script should 
now attempt to move off all the regions. Now, it will have 16+1 (due to split) 
regions to move. There is a good chance that it will be able to move all 17 off 
without issues. However, should it encounter this same issue (attempt 2 
failed), it will retry again. This process will continue until the maximum 
number of unload retry attempts has been reached.

This is not foolproof, but let's say for the sake of argument that 5% of unload 
attempts hit this issue, then with a retry count of 3, it will reduce the 
unload failure probability from 0.05 to 0.000125 (0.05^3).

Next steps:
I am looking for feedback on this approach. If it seems like a sensible 
approach, I will create a strawman patch and test it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-10923) Control where to put meta region

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HBASE-10923:
---

 Summary: Control where to put meta region
 Key: HBASE-10923
 URL: https://issues.apache.org/jira/browse/HBASE-10923
 Project: HBase
  Issue Type: Improvement
Reporter: Jimmy Xiang


There is a concern on placing meta regions on the master, as in the comments of 
HBASE-10569. I was thinking we should have a configuration for a load balancer 
to decide where to put it.  Adjusting this configuration we can control whether 
to put the meta on master, or other region server.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-10922) Log splitting seems to hang

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HBASE-10922:
---

 Summary: Log splitting seems to hang
 Key: HBASE-10922
 URL: https://issues.apache.org/jira/browse/HBASE-10922
 Project: HBase
  Issue Type: Bug
  Components: wal
Reporter: Jimmy Xiang


With distributed log replay enabled by default, I ran into an issue that log 
splitting hasn't completed after 13 hours. It seems to hang somewhere.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-10921) Port HBASE-10323 'Auto detect data block encoding in HFileOutputFormat' to 0.94 / 0.96

[Ted Yu (JIRA)]

Ted Yu created HBASE-10921:
--

 Summary: Port HBASE-10323 'Auto detect data block encoding in 
HFileOutputFormat' to 0.94 / 0.96
 Key: HBASE-10921
 URL: https://issues.apache.org/jira/browse/HBASE-10921
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu


This issue is to backport auto detection of data block encoding in 
HFileOutputFormat to 0.94 and 0.96 branches.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HBASE-10817) Add some tests on a real cluster for replica: multi master, replication

[Nicolas Liochon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon resolved HBASE-10817.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

> Add some tests on a real cluster for replica: multi master, replication
> ---
>
> Key: HBASE-10817
> URL: https://issues.apache.org/jira/browse/HBASE-10817
> Project: HBase
>  Issue Type: Sub-task
>  Components: master, regionserver, Replication
>Affects Versions: hbase-10070
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: hbase-10070
>
> Attachments: 10817.v1.patch, 10817.v2.patch, 10817.v3.patch, 
> 10817.v4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HBASE-10920) HBase HConnectionManager.createConnection throws java.io.IOException: java.lang.reflect.InvocationTargetException

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar resolved HBASE-10920.
---

Resolution: Not a Problem

TableMapReduceUtil methods should be setting the correct classpath including 
htrace. Resolving this. 

> HBase HConnectionManager.createConnection throws  java.io.IOException: 
> java.lang.reflect.InvocationTargetException 
> ---
>
> Key: HBASE-10920
> URL: https://issues.apache.org/jira/browse/HBASE-10920
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 0.99.0
>Reporter: Sean McCully
>
>   Configuration conf = HBaseConfiguration.create();
>   conf.set("hbase.zookeeper.quorum","znode-1");
>   conf.set("hbase.zookeeper.property.clientPort","2181");
>   conf.set("hbase.master",master:16000");
>   hconnection = HConnectionManager.createConnection(conf); 
> //(Parser.java:40)
> =
> Results In
> ==
> Error: java.io.IOException: java.lang.reflect.InvocationTargetException at 
> org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:341)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager.createConnectionInternal(ConnectionManager.java:234)
>  at 
> org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:215)
>  at com.example.job.Mapper.setup(Parser.java:40) at 
> org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at 
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at 
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:415) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597)
>  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) Caused by: 
> java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at 
> org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:339)
>  ... 11 more Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace at 
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:195)
>  at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:480) at 
> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>  at 
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:84)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:779)
>  at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:588)
>  ... 16 more Caused by: java.lang.ClassNotFoundException: org.htrace.Trace at 
> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at 
> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at 
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at 
> java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 22 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

RE: One question regarding bulk load

[ashish singhi]

Yes. Thanks Kashif for pointing it out. There was an empty line at the end of 
the file.

Regards
Ashish
-Original Message-
From: Kashif Jawed Siddiqui [mailto:kashi...@huawei.com] 
Sent: 07 April 2014 15:28
To: u...@hbase.apache.org
Cc: dev@hbase.apache.org
Subject: RE: One question regarding bulk load

Hi,

Please check if your file contains empty lines(maybe in the beginning 
or the end).

Since -Dimporttsv.skip.bad.lines=false is set, any empty lines will 
cause this error.

Regards
KASHIF

-Original Message-
From: ashish singhi [mailto:ashish.sin...@huawei.com] 
Sent: 07 April 2014 13:56
To: u...@hbase.apache.org
Cc: dev@hbase.apache.org
Subject: One question regarding bulk load

Hi all.

I have one question regarding bulk load.
How to load data with table empty column values in few rows using bulk load 
tool ?

I tried the following simple example in HBase 0.94.11 and Hadoop-2, with table 
having three columns and second column value is empty in few rows using bulk 
load tool.


Ø  Data in file is in below format

row0,value1,value0

row1,,value1

row2,value3,value2

row3,,value3

row4,value5,value4

row5,,value5

row6,value7,value6

row7,,value7

row8,value9,value8



Ø  When I execute the command

hadoop jar /hbase-0.94.11-security.jar importtsv 
-Dimporttsv.skip.bad.lines=false -Dimporttsv.separator=, 
-Dimporttsv.columns=HBASE_ROW_KEY,cf1:c1,cf1:c2 -Dimporttsv.bulk.output= 
/bulkdata/comma_separated _3columns comma_separated_3columns /comma_separated_ 
3columns.txt



I get the below Exception.



2014-04-07 11:15:01,870 INFO  [main] mapreduce.Job 
(Job.java:printTaskEvents(1424)) - Task Id : 
attempt_1396526639698_0028_m_00_2, Status : FAILED

Error: java.io.IOException: 
org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser$BadTsvLineException: No 
delimiter

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:135)

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:33)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

Regards,
Ashish Singhi

Re: Hadoop Summit EU

[Enis Söztutar]

Me and Devaraj attended their talk on their solution for paxos based
namenode and HBase replication.

They have two solutions, one for single datacenter, and the other multi DC
geo replication.

For the namenode, there is a wrapper, called ConsistencyNode, that
basically gets the requests, replicate it via their consistency protocol to
other CNodes within the DC (paxos based) in the edit log. If the proposal
for this is accepted, the changes are made durable. However, from my
understanding, on the read side the client chooses only one replica to
read. The client decides to connect to one of the replica namenodes, which
means that it is not doing a paxos read. I think they also wrapped the
client, so that if it gets a FileNotFoundException or something similar, it
will retry on a different server. Also they track the last seen proposal id
as a transaction id for this as well from my understanding (so
read-what-you-write consistency maybe?). The full details of the
consistency was not clear to me from the presentation.
For their multi-DC replication, they are doing a similar thing, but the
data replication is not handled by paxos, only the namenode metadata. For
each datacenter, they have a target replication factor (can be set
differently for each DC, like 0 because of regulatory reasons). The
metadata of NN is replicated via a similar mechanism. The data replication
is async to the metadata replication though. When a block is finalized, the
CNode quorum on that particular DC, schedules a remote copy to one of the
datacenters. That copy job, copies the block with directly writing the
block from the datanode to a remote datanode. Then that remote DC block is
replicated to the target replication by that DC's CNode quorum. When the
target is reached, that DC will create another proposal about the data
replication being complete. So the state machine probably contains where
each data is replicated, but they were still mentioning the client getting
DataNotReplicatedException or something.

Their work on HBase is still WIP. I do not remember much details on the
protocol, except it uses the same replication protocol (their "patented"
paxos based replication).

Of course the devil is in the details. I did not get that from the
presentation.

As a side note, Doug when asked, was saying that they are cooking something
for backups, so maybe their "secret project" also contains multi-DC
consistent state?

Enis


On Sat, Apr 5, 2014 at 1:55 AM, Ted Yu  wrote:

> Enis:
> There was a talk by Konstantin 
> Boudnik
> .
>
> Any interesting material from his presentation ?
>
> Cheers
>

RE: One question regarding bulk load

[Kashif Jawed Siddiqui]

Hi,

Please check if your file contains empty lines(maybe in the beginning 
or the end).

Since -Dimporttsv.skip.bad.lines=false is set, any empty lines will 
cause this error.

Regards
KASHIF

-Original Message-
From: ashish singhi [mailto:ashish.sin...@huawei.com] 
Sent: 07 April 2014 13:56
To: u...@hbase.apache.org
Cc: dev@hbase.apache.org
Subject: One question regarding bulk load

Hi all.

I have one question regarding bulk load.
How to load data with table empty column values in few rows using bulk load 
tool ?

I tried the following simple example in HBase 0.94.11 and Hadoop-2, with table 
having three columns and second column value is empty in few rows using bulk 
load tool.


Ø  Data in file is in below format

row0,value1,value0

row1,,value1

row2,value3,value2

row3,,value3

row4,value5,value4

row5,,value5

row6,value7,value6

row7,,value7

row8,value9,value8



Ø  When I execute the command

hadoop jar /hbase-0.94.11-security.jar importtsv 
-Dimporttsv.skip.bad.lines=false -Dimporttsv.separator=, 
-Dimporttsv.columns=HBASE_ROW_KEY,cf1:c1,cf1:c2 -Dimporttsv.bulk.output= 
/bulkdata/comma_separated _3columns comma_separated_3columns /comma_separated_ 
3columns.txt



I get the below Exception.



2014-04-07 11:15:01,870 INFO  [main] mapreduce.Job 
(Job.java:printTaskEvents(1424)) - Task Id : 
attempt_1396526639698_0028_m_00_2, Status : FAILED

Error: java.io.IOException: 
org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser$BadTsvLineException: No 
delimiter

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:135)

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:33)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

Regards,
Ashish Singhi

One question regarding bulk load

[ashish singhi]

Hi all.

I have one question regarding bulk load.
How to load data with table empty column values in few rows using bulk load 
tool ?

I tried the following simple example in HBase 0.94.11 and Hadoop-2, with table 
having three columns and second column value is empty in few rows using bulk 
load tool.


Ø  Data in file is in below format

row0,value1,value0

row1,,value1

row2,value3,value2

row3,,value3

row4,value5,value4

row5,,value5

row6,value7,value6

row7,,value7

row8,value9,value8



Ø  When I execute the command

hadoop jar /hbase-0.94.11-security.jar importtsv 
-Dimporttsv.skip.bad.lines=false -Dimporttsv.separator=, 
-Dimporttsv.columns=HBASE_ROW_KEY,cf1:c1,cf1:c2 -Dimporttsv.bulk.output= 
/bulkdata/comma_separated _3columns comma_separated_3columns /comma_separated_ 
3columns.txt



I get the below Exception.



2014-04-07 11:15:01,870 INFO  [main] mapreduce.Job 
(Job.java:printTaskEvents(1424)) - Task Id : 
attempt_1396526639698_0028_m_00_2, Status : FAILED

Error: java.io.IOException: 
org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser$BadTsvLineException: No 
delimiter

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:135)

at 
org.apache.hadoop.hbase.mapreduce.TsvImporterTextMapper.map(TsvImporterTextMapper.java:33)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

Regards,
Ashish Singhi