[jira] [Commented] (HBASE-21195) Support Log storage similar to FB LogDevice
[
https://issues.apache.org/jira/browse/HBASE-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613024#comment-16613024
]
jagan commented on HBASE-21195:
---
{noformat}
LogsDB is a layer on top of RocksDB, an ordered durable key-value data store
based on LSM trees. LogsDB is a time-ordered collection of RocksDB column
families, which are full-fledged RocksDB instances sharing a common write-ahead
log. Each RocksDB instance is called a LogsDB partition. All new writes for all
logs, be it one log or a million, go into the most recent partition, which
orders them by (log id, LSN), and saves on disk in a sequence of large sorted
immutable files, called SST files. This makes the write IO workload on the
drive mostly sequential, but creates the need to merge data from multiple files
(up to the maximum allowed number of files in a LogsDB partition, typically
about 10) when reading records. Reading from multiple files may lead to read
amplification, or wasting some read IO.
LogsDB controls read amplification in a way uniquely suited for the log data
model with its immutable records identified by immutable LSNs monotonically
increasing with time. Instead of controlling the number of sorted files by
compacting (merge-sorting) them into a bigger sorted run LogsDB simply leaves
the partition alone once it reaches its maximum number of SST files, and
creates a new most recent partition. Because partitions are read sequentially,
at no time the number of files to read concurrently will exceed the maximum
number of files in a single partition, even if the total number of SST files in
all partitions reaches tens of thousands. Space reclamation is performed
efficiently by deleting (or in some cases infrequently compacting) the oldest
partition.
{noformat}
Source
https://code.fb.com/core-data/logdevice-a-distributed-data-store-for-logs/
Was wondering why not have this (no compaction and range instead of index and
bloomfilter for hfile) as an option for log storage kind of usecase in HBase.
> Support Log storage similar to FB LogDevice
> ---
>
> Key: HBASE-21195
> URL: https://issues.apache.org/jira/browse/HBASE-21195
> Project: HBase
> Issue Type: New Feature
>Reporter: jagan
>Priority: Major
>
> Log storage, which is write once and sequential data, can be optimized in the
> following ways,
> 1. Key generated should be incremental.
> 2. HFile key index can be range and need not use BloomFilter
> 3. Instead of compaction, periodic delete of old files based on TTL can be
> supported
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21195) Support Log storage similar to FB LogDevice
[ https://issues.apache.org/jira/browse/HBASE-21195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613017#comment-16613017 ] Allan Yang commented on HBASE-21195: Could you explain more? Interesting. > Support Log storage similar to FB LogDevice > --- > > Key: HBASE-21195 > URL: https://issues.apache.org/jira/browse/HBASE-21195 > Project: HBase > Issue Type: New Feature >Reporter: jagan >Priority: Major > > Log storage, which is write once and sequential data, can be optimized in the > following ways, > 1. Key generated should be incremental. > 2. HFile key index can be range and need not use BloomFilter > 3. Instead of compaction, periodic delete of old files based on TTL can be > supported -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613018#comment-16613018 ] stack commented on HBASE-21191: --- .004 Addresses [~allan163] review. Removes just-in-case verifying Scan. Also address checkstyle issues. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch, HBASE-21191.branch-2.1.003.patch, > HBASE-21191.branch-2.1.004.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613016#comment-16613016
]
Allan Yang commented on HBASE-21190:
{quote}
I figured the later line with count of entries at end of processing was
replacement
{quote}
I see, no problem, then.
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Attachment: HBASE-21191.branch-2.1.004.patch > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch, HBASE-21191.branch-2.1.003.patch, > HBASE-21191.branch-2.1.004.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[
https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613013#comment-16613013
]
Allan Yang commented on HBASE-21185:
{quote}
Would you think that should be ok, to use that, instead of changing access
modifier from getSumOfCellElementLengths?
{quote}
I think it is OK, or you can just use other public method to calculate the heap
size(which including the structure size of cell not only the data size). It
depends what size do you need to show.
> WALPrettyPrinter: Additional useful info to be printed by wal printer tool,
> for debugability purposes
> -
>
> Key: HBASE-21185
> URL: https://issues.apache.org/jira/browse/HBASE-21185
> Project: HBase
> Issue Type: Improvement
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Trivial
> Attachments: HBASE-21185.master.001.patch
>
>
> *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as
> faulty replication sinks. An useful information one might want to track is
> the size of a single WAL entry edit, as well as size for each edit cell. Am
> proposing a patch that adds calculations for these two, as well an option to
> seek straight to a given position on the WAL file being analysed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Attachment: HBASE-21191.branch-2.1.003.patch > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch, HBASE-21191.branch-2.1.003.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613009#comment-16613009 ] Allan Yang commented on HBASE-21035: Since [~stack] is running into this problem too, I still want to mention that I still think scheduling a SCP for a splitting server is doable(as in my patch)... I have already commit this patch in our internal version. We want to make HBase2.0 in to production, we can't afford to resolve this dilemma using external tools like HBCK2(which is not available yet...) It won't cause any trouble if multiple SCP was submitted for a single server(If you are concern about safe fence [~Apache9] ). And I still can't think of any data loss case or meta inconsistency case resulting of this. And actually, when master starting, [~Apache9] also use a similar way to check if there is already a InitMetaProcedure, scheduling one if not. > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613006#comment-16613006 ] stack commented on HBASE-21191: --- bq. It is a bit of hack I think... Smile. Thanks for taking a look. bq. I think we don't have to scan the meta table to make sure it is online. Sorry. Being paranoid. I suppose it would be hard for a RS to be in the online set and in the RegionState and meta is not deployed there. Let me undo the scan bit. bq. schedule a initMetaProc (don't if there is already one) I don't want to schedule an assign in here. I want the operator to do it. See our conversation over in HBASE-20786. bq. Another opinion is that we don't have to wait namespace region. Unfortunately, initClusterSchemaService is deceptive. It implements guava Service and does async start BUT the TableNamespaceManager it starts is a blocking call. I would like to undo the blocking call and undo namespace as a distinct table but that is a battle for another day. Thanks for review. Let me put up a patch that drops the verifying Scan. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613002#comment-16613002 ] Hudson commented on HBASE-21189: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1163 (See [https://builds.apache.org/job/HBase-1.2-IT/1163/]) HBASE-21189 flaky job should gather machine stats (busbey: rev 6679643d6ea5ab1765693dd8fa38cc10f082a63a) * (edit) dev-support/flaky-tests/run-flaky-tests.Jenkinsfile > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612996#comment-16612996 ] Hudson commented on HBASE-21189: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #478 (See [https://builds.apache.org/job/HBase-1.3-IT/478/]) HBASE-21189 flaky job should gather machine stats (busbey: rev 2dec02022429703f7c89e9a0e89542c1fbe7803d) * (edit) dev-support/flaky-tests/run-flaky-tests.Jenkinsfile > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612990#comment-16612990 ] Allan Yang commented on HBASE-21191: Just reviewed the patch. It is a bit of hack I think... I think we don't have to scan the meta table to make sure it is online. We can just check the RegionState. The logic should like this: if RegionSate of meta shows offline, then schedule a initMetaProc (don't if there is already one) if RegionSate of meta shows online, then check if the server is alive if the server is not alive , then Loop wait here to check the RegionState, until meta is online on a alive server. Meatime, we can log a message to tell the operator to do something as in the patch. Another opinion is that we don't have to wait namespace region. Since we are already doing it async in initClusterSchemaService(). > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21189: Resolution: Fixed Fix Version/s: 2.0.3 2.1.1 1.4.8 2.2.0 1.2.8 1.3.3 1.5.0 3.0.0 Status: Resolved (was: Patch Available) > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612979#comment-16612979
]
Reid Chan commented on HBASE-20993:
---
How's going sir [~jackbearden].
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.2.001.patch,
> HBASE-20993.branch-1.wip.002.patch, HBASE-20993.branch-1.wip.patch,
> yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
> org.playground.hbase.KerberizedClientFallback.main(KerberizedClientFallback.java:55)
> Caused by:
[jira] [Commented] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612977#comment-16612977 ] stack commented on HBASE-21189: --- +1 Nice addition. > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612976#comment-16612976
]
Reid Chan commented on HBASE-20734:
---
bq. Would you prefer I submit a separate patch for style changes? This doesn't
follow Java camel case conventions.
Got your mind, but i meant the indents, new patch for style changes is fine by
me.
bq. but it should be 53... In the real world this should not be an issue as
you mention.
Good.
*{color:green}+1{color}* for master v12.
> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
> Issue Type: Improvement
> Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch,
> HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch,
> HBASE-20734.branch-1.004.patch, HBASE-20734.master.001.patch,
> HBASE-20734.master.002.patch, HBASE-20734.master.003.patch,
> HBASE-20734.master.004.patch, HBASE-20734.master.005.patch,
> HBASE-20734.master.006.patch, HBASE-20734.master.007.patch,
> HBASE-20734.master.008.patch, HBASE-20734.master.009.patch,
> HBASE-20734.master.010.patch, HBASE-20734.master.011.patch,
> HBASE-20734.master.012.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best
> performance when hbase.wal.dir is configured to be on different (fast) media
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server
> failover.
> This issue is to find proper (hopefully backward compatible) way in
> colocating recovered edits directory with hbase.wal.dir .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[ https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612967#comment-16612967 ] stack commented on HBASE-21190: --- Hey [~allan163]. I figured the later line with count of entries at end of processing was replacement (thought it overkill doing before and after). You think different sir? I can push an amendment. > Log files and count of entries in each as we load from the MasterProcWAL store > -- > > Key: HBASE-21190 > URL: https://issues.apache.org/jira/browse/HBASE-21190 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21190.branch-2.1.001.patch > > > Sometimes this can take a while especially if loads of files. Emit counts of > entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612960#comment-16612960
]
Allan Yang edited comment on HBASE-21190 at 9/13/18 3:05 AM:
-
Maybe it is too late to join the discussion, but I have a question, why
deleting this line? Isn't useful we can see which procedure wal is loading?
{code}
-LOG.debug("Loading WAL id={}", log.getLogId());
{code}
was (Author: allan163):
Maybe it is too late to join the discussion, but I have a question, why
deleting this line? Isn't useful we can see which procedure wal is loading?
{quote}
-LOG.debug("Loading WAL id={}", log.getLogId());
{quote}
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612960#comment-16612960
]
Allan Yang commented on HBASE-21190:
Maybe it is too late to join the discussion, but I have a question, why
deleting this line? Isn't useful we can see which procedure wal is loading?
{quote}
-LOG.debug("Loading WAL id={}", log.getLogId());
{quote}
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18829) Consider reverting HBASE-14893 Race between mutation on region and region closing operation
[
https://issues.apache.org/jira/browse/HBASE-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612959#comment-16612959
]
Hadoop QA commented on HBASE-18829:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m
57s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
23s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 23s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
24s{color} | {color:red} hbase-server: The patch generated 7 new + 213
unchanged - 0 fixed = 220 total (was 213) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
24s{color} | {color:red} patch has 16 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
49s{color} | {color:red} The patch causes 17 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
55s{color} | {color:red} The patch causes 17 errors with Hadoop v3.0.0. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 51s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18829 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12887586/HBASE-18829.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 71e74fc45b27 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / dc1dedb073 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/14405/artifact/patchprocess/patch-mvninstall-root.txt
|
| compile |
[jira] [Commented] (HBASE-18829) Consider reverting HBASE-14893 Race between mutation on region and region closing operation
[ https://issues.apache.org/jira/browse/HBASE-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612941#comment-16612941 ] Allan Yang commented on HBASE-18829: Any progress on this? +1 to revert this. Taking the Region's lock (write lock) first and set closing to true later will cause the new coming request waiting on the region's lock(read lock) in startRegionOperation. This is no good, since write lock has priority for Java's ReadWriteLock, the region closing operation will finish first, and those new coming request will still throw a NotServingRegionException to client, the waiting is useless. > Consider reverting HBASE-14893 Race between mutation on region and region > closing operation > --- > > Key: HBASE-18829 > URL: https://issues.apache.org/jira/browse/HBASE-18829 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 18829.v1.txt, HBASE-18829.patch > > > HBASE-14893 was brought to attention by [~rajeshbabu] over PHOENIX-3111. > This issue is to consider reverting the fix from HBASE-14893 based on the > following observations: > * The closing boolean was intended to be acquired before taking the lock > ([~enis]) > * Phoenix local index has evolved over the years, the situation leading to > NotServingRegionException may not exist from Phoenix side > * Even if the situation still exists, downstream project (Phoenix) should > properly handle NotServingRegionException without change in locking scheme in > hbase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21195) Support Log storage similar to FB LogDevice
jagan created HBASE-21195: - Summary: Support Log storage similar to FB LogDevice Key: HBASE-21195 URL: https://issues.apache.org/jira/browse/HBASE-21195 Project: HBase Issue Type: New Feature Reporter: jagan Log storage, which is write once and sequential data, can be optimized in the following ways, 1. Key generated should be incremental. 2. HFile key index can be range and need not use BloomFilter 3. Instead of compaction, periodic delete of old files based on TTL can be supported -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21188) Print heap and gc informations in our junit ResourceChecker
[ https://issues.apache.org/jira/browse/HBASE-21188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612934#comment-16612934 ] Duo Zhang commented on HBASE-21188: --- Pushed to master. > Print heap and gc informations in our junit ResourceChecker > --- > > Key: HBASE-21188 > URL: https://issues.apache.org/jira/browse/HBASE-21188 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21188.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21188) Print heap and gc informations in our junit ResourceChecker
[ https://issues.apache.org/jira/browse/HBASE-21188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612930#comment-16612930 ] Duo Zhang commented on HBASE-21188: --- Anyway let me commit to master to see if it could help debugging the failed UTs. Can revert or push addendum later. > Print heap and gc informations in our junit ResourceChecker > --- > > Key: HBASE-21188 > URL: https://issues.apache.org/jira/browse/HBASE-21188 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21188.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21097) Flush pressure assertion may fail in testFlushThroughputTuning
[
https://issues.apache.org/jira/browse/HBASE-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612921#comment-16612921
]
Duo Zhang commented on HBASE-21097:
---
The new code is fine. But please still add the comment to say why here we can
not use the old assertion.
> Flush pressure assertion may fail in testFlushThroughputTuning
> ---
>
> Key: HBASE-21097
> URL: https://issues.apache.org/jira/browse/HBASE-21097
> Project: HBase
> Issue Type: Test
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21097.v1.txt, 21097.v2.txt, 21097.v4.txt,
> HBASE-21097.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14137/artifact/patchprocess/patch-unit-hbase-server.txt
> :
> {code}
> [ERROR]
> testFlushThroughputTuning(org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController)
> Time elapsed: 17.446 s <<< FAILURE!
> java.lang.AssertionError: expected:<0.0> but was:<1.2906294173808417E-6>
> at
> org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController.testFlushThroughputTuning(TestFlushWithThroughputController.java:185)
> {code}
> Here is the related assertion:
> {code}
> assertEquals(0.0, regionServer.getFlushPressure(), EPSILON);
> {code}
> where EPSILON = 1E-6
> In the above case, due to margin of 2.9E-7, the assertion didn't pass.
> It seems the epsilon can be adjusted to accommodate different workload /
> hardware combination.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20941) Create and implement HbckService in master
[ https://issues.apache.org/jira/browse/HBASE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612920#comment-16612920 ] stack commented on HBASE-20941: --- I just pushed this to branch-2.1. I'd missed doing so in above commits. > Create and implement HbckService in master > -- > > Key: HBASE-20941 > URL: https://issues.apache.org/jira/browse/HBASE-20941 > Project: HBase > Issue Type: Sub-task >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.2 > > Attachments: hbase-20941.master.001.patch, > hbase-20941.master.002.patch, hbase-20941.master.003.patch, > hbase-20941.master.004.patch, hbase-20941.master.004.patch, > hbase-20941.master.004.patch > > > Create HbckService in master and implement following methods: > # setTableState(): If table state are inconsistent with action/ procedures > working on them, sometimes manipulating their states in meta fix things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612895#comment-16612895
]
Hadoop QA commented on HBASE-21191:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 8 new + 366
unchanged - 0 fixed = 374 total (was 366) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
24s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}156m 59s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}204m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21191 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939469/HBASE-21191.branch-2.1.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0d8e4e1883ab 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[
https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612871#comment-16612871
]
Hadoop QA commented on HBASE-19418:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} hbase-server: The patch generated 1 new + 80 unchanged
- 0 fixed = 81 total (was 80) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 13s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.coprocessor.TestRegionObserverForAddingMutationsFromCoprocessors |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-19418 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939471/HBASE-19418.master.000.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f627a681bf81 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 76199a0a29 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14404/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612841#comment-16612841
]
Hudson commented on HBASE-21190:
Results for branch branch-2.0
[build #806 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/806/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/806//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/806//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/806//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21192) Add HOW-TO repair damaged AMv2.
[
https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612821#comment-16612821
]
stack edited comment on HBASE-21192 at 9/12/18 10:55 PM:
-
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
h3. Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
h3. Why you need to cleanup STUCK regions.
Will cause buildup of master proc WALs IF an associated, outstanding procedure
(not always the case). If too many WALs, Master will be slow to start as it
runs through all outstanding WALs.
was (Author: stack):
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
h3. Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
h3. Why you need to cleanup STUCK regions.
Will cause buildup of master proc WALs. If too many, Master will be slow to
start as it runs through all outstanding WALs.
> Add HOW-TO repair damaged AMv2.
> ---
>
> Key: HBASE-21192
> URL: https://issues.apache.org/jira/browse/HBASE-21192
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to
> identify particular circumstance, how to run a repair, as well as caveats
> (e.g. if no log recovery, then region may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for
> operator intervention; e.g. Master#inMeta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[
https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612835#comment-16612835
]
Hudson commented on HBASE-21168:
Results for branch branch-1.3
[build #465 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/465/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/465//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/465//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/465//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> BloomFilterUtil uses hardcoded randomness
> -
>
> Key: HBASE-21168
> URL: https://issues.apache.org/jira/browse/HBASE-21168
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-21168.branch-1.002.patch,
> HBASE-21168.master.001.patch, HBASE-21168.master.002.patch
>
>
> This was flagged by a Fortify scan and while it doesn't appear to be a real
> issue, it's pretty easy to take care of anyway.
> The hard coded rand can be moved to the test class that actually needs it to
> make the static analysis happy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[
https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612833#comment-16612833
]
Hudson commented on HBASE-21168:
Results for branch branch-2.1
[build #316 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BloomFilterUtil uses hardcoded randomness
> -
>
> Key: HBASE-21168
> URL: https://issues.apache.org/jira/browse/HBASE-21168
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-21168.branch-1.002.patch,
> HBASE-21168.master.001.patch, HBASE-21168.master.002.patch
>
>
> This was flagged by a Fortify scan and while it doesn't appear to be a real
> issue, it's pretty easy to take care of anyway.
> The hard coded rand can be moved to the test class that actually needs it to
> make the static analysis happy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612834#comment-16612834
]
Hudson commented on HBASE-21190:
Results for branch branch-2.1
[build #316 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/316//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21192) Add HOW-TO repair damaged AMv2.
[
https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612821#comment-16612821
]
stack edited comment on HBASE-21192 at 9/12/18 10:38 PM:
-
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
h3. Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
h3. Why you need to cleanup STUCK regions.
Will cause buildup of master proc WALs. If too many, Master will be slow to
start as it runs through all outstanding WALs.
was (Author: stack):
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
.h3 Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
> Add HOW-TO repair damaged AMv2.
> ---
>
> Key: HBASE-21192
> URL: https://issues.apache.org/jira/browse/HBASE-21192
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to
> identify particular circumstance, how to run a repair, as well as caveats
> (e.g. if no log recovery, then region may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for
> operator intervention; e.g. Master#inMeta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21192) Add HOW-TO repair damaged AMv2.
[
https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612821#comment-16612821
]
stack edited comment on HBASE-21192 at 9/12/18 10:37 PM:
-
For example
h2. STUCK Region
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... _as long as no lock on the region (See below)_, you should be able to just
do an assign of region to unSTUCK it. If many, make a file of them all and cat
into hbase shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has
stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
.h3 Locked STUCK Region
If locked, it will complain when you try to assign... TODO .. .how to check if
locked. TODO: ain't it bad if you add an assign on a locked region? Explain.
was (Author: stack):
For example:
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... as long as no lock on the region, you should be able to just do an assign
of region to unSTUCK it. If many, make a file of them all and cat into hbase
shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
> Add HOW-TO repair damaged AMv2.
> ---
>
> Key: HBASE-21192
> URL: https://issues.apache.org/jira/browse/HBASE-21192
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to
> identify particular circumstance, how to run a repair, as well as caveats
> (e.g. if no log recovery, then region may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for
> operator intervention; e.g. Master#inMeta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21192) Add HOW-TO repair damaged AMv2.
[
https://issues.apache.org/jira/browse/HBASE-21192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612821#comment-16612821
]
stack commented on HBASE-21192:
---
For example:
If this in log:
{code}
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=va1001.halxg.cloudera.com,22101,1536173230599,
table=IntegrationTestBigLinkedList_20180626110336,
region=dbdb56242f17610c46ea044f7a42895b
2018-09-12 15:29:06,558 WARN
org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
Region-In-Transition rit=OPENING,
location=ve1229.halxg.cloudera.com,22101,1536173229844,
table=IntegrationTestBigLinkedList_20180803113809,
region=4d1618634dae662acb06f5e3b55223c9
{code}
... as long as no lock on the region, you should be able to just do an assign
of region to unSTUCK it. If many, make a file of them all and cat into hbase
shell as in: {{$ cat /tmp/a.txt |hbase shell}} where /tmp/a.txt has stuff like:
{code}
assign 'fb9e0a6e864e36894c48da74074de65d'
assign '494e64585e49a22dad2f35383e7b9bb9'
assign 'e1fa1d4c3dcd59d6a0a61a5c63f4fda5'
assign '4b1fa4fd3bc52d1a6a94db1c4c13ab2b'
assign '86c5348e84e200fdf2f8633c9ac188b5'
assign 'ab60573f41a978de566a8a7097cf8ccc'
assign '085e05caefffcfb17356d4326e99c523'
assign '6ab89f20867d6a97fdb2a61fa82be4cc'
assign '4feb719da3cb53374d7b9162c0849c90'
assign '38d66170d5004c22ed61b184b8209f74'
assign 'c9807aef53ef14f14c9fc1de6ad942c5'
assign 'dbdb56242f17610c46ea044f7a42895b'
assign '4d1618634dae662acb06f5e3b55223c9'
assign '95035cf88e92179c5673c49d3eceaf7d'
{code}
> Add HOW-TO repair damaged AMv2.
> ---
>
> Key: HBASE-21192
> URL: https://issues.apache.org/jira/browse/HBASE-21192
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> Need a page or two on how to do various fixups. Will include doc on how to
> identify particular circumstance, how to run a repair, as well as caveats
> (e.g. if no log recovery, then region may be missing edits).
> Add pointer to log messages, especially those that explicitly ask for
> operator intervention; e.g. Master#inMeta.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612814#comment-16612814
]
Hadoop QA commented on HBASE-21191:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
21s{color} | {color:red} hbase-server: The patch generated 8 new + 366
unchanged - 0 fixed = 374 total (was 366) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 13s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21191 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939454/HBASE-21191.branch-2.1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5b9a113f1725 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612776#comment-16612776
]
Mingliang Liu commented on HBASE-21190:
---
Thanks [~apurtell], I was going to say the {{LOG}} is not supporting
placeholders in {{branch-1}} only to find you have taken care of it.
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[ https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramie Raufdeen updated HBASE-19418: --- Status: In Progress (was: Patch Available) > RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable. > - > > Key: HBASE-19418 > URL: https://issues.apache.org/jira/browse/HBASE-19418 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Ramie Raufdeen >Priority: Major > Attachments: HBASE-19418.master.000.patch > > > When RSs have a LOT of regions and CFs, flushing everything within 5 minutes > is not always doable. It might be interesting to be able to increase the > RANGE_OF_DELAY. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[ https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612740#comment-16612740 ] Hudson commented on HBASE-21190: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #477 (See [https://builds.apache.org/job/HBase-1.3-IT/477/]) HBASE-21190 Log files and count of entries in each as we load from the (apurtell: rev 31c7863383c32add6eab965e1d3e0d0c22db3576) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java > Log files and count of entries in each as we load from the MasterProcWAL store > -- > > Key: HBASE-21190 > URL: https://issues.apache.org/jira/browse/HBASE-21190 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21190.branch-2.1.001.patch > > > Sometimes this can take a while especially if loads of files. Emit counts of > entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[ https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramie Raufdeen updated HBASE-19418: --- Status: In Progress (was: Patch Available) > RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable. > - > > Key: HBASE-19418 > URL: https://issues.apache.org/jira/browse/HBASE-19418 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Ramie Raufdeen >Priority: Major > Attachments: HBASE-19418.master.000.patch > > > When RSs have a LOT of regions and CFs, flushing everything within 5 minutes > is not always doable. It might be interesting to be able to increase the > RANGE_OF_DELAY. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[ https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramie Raufdeen updated HBASE-19418: --- Attachment: HBASE-19418.master.000.patch Status: Patch Available (was: In Progress) > RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable. > - > > Key: HBASE-19418 > URL: https://issues.apache.org/jira/browse/HBASE-19418 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Ramie Raufdeen >Priority: Major > Attachments: HBASE-19418.master.000.patch > > > When RSs have a LOT of regions and CFs, flushing everything within 5 minutes > is not always doable. It might be interesting to be able to increase the > RANGE_OF_DELAY. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[ https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramie Raufdeen updated HBASE-19418: --- Attachment: (was: HBASE-19418.master.000.patch) > RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable. > - > > Key: HBASE-19418 > URL: https://issues.apache.org/jira/browse/HBASE-19418 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4 >Reporter: Jean-Marc Spaggiari >Assignee: Ramie Raufdeen >Priority: Major > Attachments: HBASE-19418.master.000.patch > > > When RSs have a LOT of regions and CFs, flushing everything within 5 minutes > is not always doable. It might be interesting to be able to increase the > RANGE_OF_DELAY. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21174) [REST] Failed to parse empty qualifier in TableResource#getScanResource
[
https://issues.apache.org/jira/browse/HBASE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612726#comment-16612726
]
Hudson commented on HBASE-21174:
Results for branch branch-1.4
[build #460 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/460/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/460//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/460//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/460//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [REST] Failed to parse empty qualifier in TableResource#getScanResource
> ---
>
> Key: HBASE-21174
> URL: https://issues.apache.org/jira/browse/HBASE-21174
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21174.branch-1.001.patch,
> HBASE-21174.master.001.patch, HBASE-21174.master.002.patch
>
>
> {code:xml}
> GET /t1/*?column=f:c1=f:
> {code}
> If I want to get the values of 'f:'(empty qualifier) for all rows in the
> table by rest server, I will send the above request. However, this request
> will return all column values.
> {code:java|title=TableResource#getScanResource|borderStyle=solid}
> for (String csplit : column) {
> String[] familysplit = csplit.trim().split(":");
> if (familysplit.length == 2) {
> if (familysplit[1].length() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family and column : " + familysplit[0] + " " +
> familysplit[1]);
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]),
> Bytes.toBytes(familysplit[1]));
> } else {
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0] + " and empty
> qualifier.");
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]), null);
> }
> } else if (StringUtils.isNotEmpty(familysplit[0])) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0]);
> }
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> }
> }
> {code}
> Through the above code, when the column has an empty qualifier, the empty
> qualifier cannot be parsed correctly.In other words, 'f:'(empty qualifier)
> and 'f' (column family) are considered to have the same meaning, which is
> wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612721#comment-16612721
]
Sean Busbey commented on HBASE-21182:
-
sorry for the basic question. but when you say:
{quote}
Built master branch like below:
{code}
mvn clean install -DskipTests
{code}
Then tried to execute {{start-hbase.sh}} failed with NoClassDefFoundError
{code}
./bin/start-hbase.sh
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.
...
{code}
{quote}
Do you mean you ran ./bin/start-hbase.sh from the source checkout directory and
not from a generated tarball that was the output of the build step?
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Subrat Mishra
>Priority: Major
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612723#comment-16612723 ] Hudson commented on HBASE-21168: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1162 (See [https://builds.apache.org/job/HBase-1.2-IT/1162/]) HBASE-21168 BloomFilterUtil uses hardcoded randomness (Mike Drob) (apurtell: rev 2c8060e1e8be34b92f17645c80b8985354485281) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBloomFilter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-21168.branch-1.002.patch, > HBASE-21168.master.001.patch, HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[ https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21190: --- Fix Version/s: 1.4.8 1.3.3 1.5.0 No problem [~liuml07], picked back, fix versions updated > Log files and count of entries in each as we load from the MasterProcWAL store > -- > > Key: HBASE-21190 > URL: https://issues.apache.org/jira/browse/HBASE-21190 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21190.branch-2.1.001.patch > > > Sometimes this can take a while especially if loads of files. Emit counts of > entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Attachment: HBASE-21191.branch-2.1.002.patch > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612718#comment-16612718 ] Hudson commented on HBASE-21168: FAILURE: Integrated in Jenkins build HBase-1.3-IT #476 (See [https://builds.apache.org/job/HBase-1.3-IT/476/]) HBASE-21168 BloomFilterUtil uses hardcoded randomness (Mike Drob) (apurtell: rev bfe373d1882c9b89aaa0fa092846d6a77d7b0602) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBloomFilter.java > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-21168.branch-1.002.patch, > HBASE-21168.master.001.patch, HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21194) Add TestCopyTable which exercises MOB feature
Ted Yu created HBASE-21194: -- Summary: Add TestCopyTable which exercises MOB feature Key: HBASE-21194 URL: https://issues.apache.org/jira/browse/HBASE-21194 Project: HBase Issue Type: Test Reporter: Ted Yu Currently TestCopyTable doesn't cover table(s) with MOB feature enabled. We should add variant that enables MOB on the table being copied and verify that MOB content is copied correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612714#comment-16612714
]
stack commented on HBASE-21191:
---
Playing w/ this patch... I removed master wal procs content. Had to remove WALs
for any old servers too else the meta assign was getting scheduled; i.e. its
hard to manufacture case where there is no assign for hbase:meta.
I tried to use straight shell assign to do the hbase:meta assign but it came
back with:
{code}
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at
org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2966)
at
org.apache.hadoop.hbase.master.MasterRpcServices.assignRegion(MasterRpcServices.java:558)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
So, ok, this should be an hbck2 task. Working on it.
> Add a holding-pattern if no assign for meta or namespace (Can happen if
> masterprocwals have been cleared).
> --
>
> Key: HBASE-21191
> URL: https://issues.apache.org/jira/browse/HBASE-21191
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21191.branch-2.1.001.patch
>
>
> If the masterprocwals have been removed -- operator error, hdfs dataloss, or
> because we have gotten ourselves into a pathological state where we have
> hundreds of masterprocwals too process and it is taking too long so we just
> want to startover -- then master startup will have a dilemma. Master startup
> needs hbase:meta to be online. If the masterprocwals have been removed, there
> may be no outstanding assign or a servercrashprocedure with coverage for
> hbase:meta (I ran into this issue repeatedly in internal testing purging
> masterprocwals on a large test cluster). Worse, when master startup cannot
> find an online hbase:meta, it exits after exhausting the RPC retries.
> So, we need a holding-pattern for master startup if hbase:meta is not online
> if only so an operator can schedule an assign for meta or so they can assign
> fixup procedures (HBASE-20786 has discussion on why we cannot just
> auto-schedule an assign of meta).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21168: --- Fix Version/s: 1.4.8 1.2.8 1.3.3 1.5.0 Thanks for the branch-1 patch [~liuml07], I've applied it. Updated fix versions accordingly. > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-21168.branch-1.002.patch, > HBASE-21168.master.001.patch, HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612645#comment-16612645 ] stack commented on HBASE-21035: --- I put up a patch on HBASE-21191 that steals [~allan163]'s patch from here. It has master go into a holding pattern complaining that hbase:meta is not online asking for operator intervention. Ditto for namespace table. Will work on doc on what operators need to do to effect repair in a while after I've backfilled more of the hbck2 stuff. A test over in HBASE-21191 shows that scheduling an hbase:meta assign seems to do the job getting the cluster up again (missing is how to do this from the outside, from a client... working on that next). > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[
https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612646#comment-16612646
]
Mingliang Liu commented on HBASE-21168:
---
I think it is helpful to have this in {{branch-1}} and attach a patch with
minor conflict resolved. Thanks,
> BloomFilterUtil uses hardcoded randomness
> -
>
> Key: HBASE-21168
> URL: https://issues.apache.org/jira/browse/HBASE-21168
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21168.branch-1.002.patch,
> HBASE-21168.master.001.patch, HBASE-21168.master.002.patch
>
>
> This was flagged by a Fortify scan and while it doesn't appear to be a real
> issue, it's pretty easy to take care of anyway.
> The hard coded rand can be moved to the test class that actually needs it to
> make the static analysis happy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HBASE-21168: -- Attachment: HBASE-21168.branch-1.002.patch > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-21168.branch-1.002.patch, > HBASE-21168.master.001.patch, HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612638#comment-16612638 ] stack commented on HBASE-21191: --- One note on how the 'holding pattern' was implemented. It was suggested that we just do longer timesout on the RPCs when we are trying to read meta. I tried it. What I found was that the Retrying Callable we use to make the RPC takes the # of retries from default rather than those of the current Configuration so extending RPC retries and operation timeout has a bug or is need of a bit of work (HBASE-21193). The other awkward thing about using a long RPC timeout/retry rather than what we have here is that it is easier flagging operator they need to intercede if we have the explicit hold as we have in this patch. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21193) Retrying Callable doesn't take max retries from current context; uses defaults instead
stack created HBASE-21193: - Summary: Retrying Callable doesn't take max retries from current context; uses defaults instead Key: HBASE-21193 URL: https://issues.apache.org/jira/browse/HBASE-21193 Project: HBase Issue Type: Bug Reporter: stack This makes it hard to change retry count on a read of meta for instance. I noticed this when trying to change the defaults for a meta read. I made a customer Connection inside in the master with a new Configuration that had rpc retries and timings upped radically. My reads nonetheless were finishing at the usual retry point (31 tries after 60 seconds or so) because it looked like the Retrying Callable that does the read was taking max retries from defaults rather than reading the passed in Configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21192) Add HOW-TO repair damaged AMv2.
stack created HBASE-21192: - Summary: Add HOW-TO repair damaged AMv2. Key: HBASE-21192 URL: https://issues.apache.org/jira/browse/HBASE-21192 Project: HBase Issue Type: Sub-task Components: amv2 Reporter: stack Assignee: stack Need a page or two on how to do various fixups. Will include doc on how to identify particular circumstance, how to run a repair, as well as caveats (e.g. if no log recovery, then region may be missing edits). Add pointer to log messages, especially those that explicitly ask for operator intervention; e.g. Master#inMeta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Release Note: Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21192). (was: Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in ).) > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Fix Version/s: 2.1.1 Release Note: Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in ). Status: Patch Available (was: Open) > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612628#comment-16612628 ] stack commented on HBASE-21191: --- .001 adds holding-pattern if meta is not online. Ditto if namespace is not allocated. Test (mostly stolen from HBASE-20786) injects meta assigns after we've achieved holding pattern and shows that we again make progress. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-21191.branch-2.1.001.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Attachment: HBASE-21191.branch-2.1.001.patch > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-21191.branch-2.1.001.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[
https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612616#comment-16612616
]
Mingliang Liu commented on HBASE-21190:
---
Seems helpful to {{branch-1}} as well. Ping [~apurtell]
> Log files and count of entries in each as we load from the MasterProcWAL store
> --
>
> Key: HBASE-21190
> URL: https://issues.apache.org/jira/browse/HBASE-21190
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21190.branch-2.1.001.patch
>
>
> Sometimes this can take a while especially if loads of files. Emit counts of
> entries so operator gets sense of scale of procedures being processed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20306) LoadTestTool does not print summary at end of run
[ https://issues.apache.org/jira/browse/HBASE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612569#comment-16612569 ] Colin Garcia commented on HBASE-20306: -- [~mdrob] I was wondering if you've had a chance to look this over? > LoadTestTool does not print summary at end of run > - > > Key: HBASE-20306 > URL: https://issues.apache.org/jira/browse/HBASE-20306 > Project: HBase > Issue Type: Bug > Components: tooling >Reporter: Mike Drob >Assignee: Colin Garcia >Priority: Major > Labels: beginner > Attachments: HBASE-20306.000.patch, HBASE-20306.001.patch > > > ltt currently prints status as it goes, but doesn't give a nice summary of > what happened so users have to infer it from the last status line printed. > Would be nice to print a real summary with statistics about what was run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21097) Flush pressure assertion may fail in testFlushThroughputTuning
[
https://issues.apache.org/jira/browse/HBASE-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612564#comment-16612564
]
Hadoop QA commented on HBASE-21097:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}154m
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}200m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21097 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939421/21097.v4.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7b392c3d1413 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0075093d21 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14401/testReport/ |
| Max. process+thread count | 4802 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612540#comment-16612540
]
Sean Busbey commented on HBASE-21182:
-
can you give me specific steps to reproduce? I've walked through the same
things the nightly test does and nothing shows a problem.
none of the shaded client facing artifacts should be in *any* of the cluster
process classpaths, so if you're seeing that something is off.
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Subrat Mishra
>Priority: Major
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
stack created HBASE-21191: - Summary: Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared). Key: HBASE-21191 URL: https://issues.apache.org/jira/browse/HBASE-21191 Project: HBase Issue Type: Sub-task Components: amv2 Reporter: stack Assignee: stack If the masterprocwals have been removed -- operator error, hdfs dataloss, or because we have gotten ourselves into a pathological state where we have hundreds of masterprocwals too process and it is taking too long so we just want to startover -- then master startup will have a dilemma. Master startup needs hbase:meta to be online. If the masterprocwals have been removed, there may be no outstanding assign or a servercrashprocedure with coverage for hbase:meta (I ran into this issue repeatedly in internal testing purging masterprocwals on a large test cluster). Worse, when master startup cannot find an online hbase:meta, it exits after exhausting the RPC retries. So, we need a holding-pattern for master startup if hbase:meta is not online if only so an operator can schedule an assign for meta or so they can assign fixup procedures (HBASE-20786 has discussion on why we cannot just auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21172) Reimplement the retry backoff logic for ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612525#comment-16612525
]
Hudson commented on HBASE-21172:
Results for branch branch-2.1
[build #315 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/315/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/315//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/315//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/315//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reimplement the retry backoff logic for ReopenTableRegionsProcedure
> ---
>
> Key: HBASE-21172
> URL: https://issues.apache.org/jira/browse/HBASE-21172
> Project: HBase
> Issue Type: Sub-task
> Components: amv2, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21172-branch-2.1-v1.patch,
> HBASE-21172-branch-2.1-v1.patch, HBASE-21172-branch-2.1.patch,
> HBASE-21172-v1.patch, HBASE-21172-v2.patch, HBASE-21172-v3.patch,
> HBASE-21172-v4.patch, HBASE-21172.patch
>
>
> Now we just do a blocking sleep in the execute method, and there is no
> exponential backoff.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20828) Finish-up AMv2 Design/List of Tenets/Specification of operation
[ https://issues.apache.org/jira/browse/HBASE-20828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612522#comment-16612522 ] stack commented on HBASE-20828: --- How to address the issue where a STUCK procedure holds up free-up of master proc WALs? If a Master crash, replaying all edits though most may belong to finished procedures, can keep the Master occupied for a good amount of time reconstructing in-memory AMv2 state. Chatting w/ Duo, could we do a Procedure Store that was region based? What would it take? What would the model look like? It would at a minimum purge one of the three methods of writing WALs in HDFS that we currently have (MasterProcWAL has its own way of doing appends). TODO. > Finish-up AMv2 Design/List of Tenets/Specification of operation > --- > > Key: HBASE-20828 > URL: https://issues.apache.org/jira/browse/HBASE-20828 > Project: HBase > Issue Type: Umbrella > Components: amv2 >Reporter: stack >Priority: Major > > AMv2 is missing specification. There are too many grey-areas still. Also > missing are a concise listing of the tenets of AMv2 operation. Here are some > examples: > * HBASE-19529 "Handle null states in AM": Asks how we should treat null > state in hbase:meta. What does it 'mean'. We seem to treat it differently > dependent on context. Needs clarification. [~Apache9] recently asked similar > about the meaning of OFFLINE. > * Logging needs to have a particular form to help trace Procedure progress; > needs a write-up. > Lets fill in items to address in this umbrella issue. Can address in > subissues and produce specification doc too. We have the below but these are > mostly (incomplete) description for devs on pv2 and amv2; the specification > is missing: > http://hbase.apache.org/book.html#pv2 > http://hbase.apache.org/book.html#amv2 > (Other areas include addressing what is up w/ rollback -- when, how much, and > when it is not appropriate -- as well as recommendation on Procedures > coarseness, locking -- is it ok to lock table in alter table procedure for > the life of the procedure? -- and so on). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612503#comment-16612503 ] Wellington Chevreuil commented on HBASE-21185: -- Thanks [~allan163]. Indeed, had not counted the TS and wasn't aware of CellUtils. Just realised there's also *PrivateCellUtil.estimatedSizeOfCell* that uses *getSumOfCellElementLengths.* Would you think that should be ok, to use that, instead of changing access modifier from getSumOfCellElementLengths? > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Attachments: HBASE-21185.master.001.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612494#comment-16612494
]
Toshihiro Suzuki commented on HBASE-21182:
--
Looks like when running ./bin/start-hbase.sh, hbase-shaded-mapreduce.jar is in
the classpath, and then some classes are loaded from
hbase-shaded-mapreduce.jar. I think that's related to the issue. Actually after
removing hbase-shaded-mapreduce.jar and running start-hbase.sh, the error
didn't occur.
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Subrat Mishra
>Priority: Major
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[ https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21190. --- Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 I pushed this two line change to branch-2.0+ > Log files and count of entries in each as we load from the MasterProcWAL store > -- > > Key: HBASE-21190 > URL: https://issues.apache.org/jira/browse/HBASE-21190 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21190.branch-2.1.001.patch > > > Sometimes this can take a while especially if loads of files. Emit counts of > entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
[ https://issues.apache.org/jira/browse/HBASE-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21190: -- Attachment: HBASE-21190.branch-2.1.001.patch > Log files and count of entries in each as we load from the MasterProcWAL store > -- > > Key: HBASE-21190 > URL: https://issues.apache.org/jira/browse/HBASE-21190 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21190.branch-2.1.001.patch > > > Sometimes this can take a while especially if loads of files. Emit counts of > entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21190) Log files and count of entries in each as we load from the MasterProcWAL store
stack created HBASE-21190: - Summary: Log files and count of entries in each as we load from the MasterProcWAL store Key: HBASE-21190 URL: https://issues.apache.org/jira/browse/HBASE-21190 Project: HBase Issue Type: Sub-task Components: amv2 Reporter: stack Assignee: stack Fix For: 2.1.1, 2.0.3 Sometimes this can take a while especially if loads of files. Emit counts of entries so operator gets sense of scale of procedures being processed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21165) During ProcedureStore load, there is no listing of progress...
[
https://issues.apache.org/jira/browse/HBASE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612489#comment-16612489
]
stack commented on HBASE-21165:
---
.001 is a simple log line that gives some sense of the amount of entries in
WALs. Needs more logging for later stages of processing. In fact, let me commit
this patch in a sub-issue and leave this open as an umbrella.
> During ProcedureStore load, there is no listing of progress...
> --
>
> Key: HBASE-21165
> URL: https://issues.apache.org/jira/browse/HBASE-21165
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-21165.branch-2.1.001.patch
>
>
> I have a Master that crashed on a large cluster with hundreds of outstanding
> Procedure WALs and (probably --TBD) a few million Procedures to load. It is
> taking a long time (two hours)...
> There were STUCK procedures that were preventing clean-up of the old WALs.
> I can tell we are making progress by enabling TRACE on the Procedure Store.
> Better would be an emission as we made progress through the files with an
> output after every so many procedures loaded.
> Then, post-load, there is a long time spent sorting-out the Procedure
> image... We are in finish for ages doing stuff like:
> {code}
> "master/vc0207:22001:becomeActiveMaster" #98 daemon prio=5 os_prio=0
> tid=0x00d31800 nid=0x1efc0 runnable [0x7f0a3c17d000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$WalProcedureMap.removeFromMap(ProcedureWALFormatReader.java:837)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$WalProcedureMap.fetchReady(ProcedureWALFormatReader.java:614)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.finish(ProcedureWALFormatReader.java:201)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:94)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:426)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:382)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:663)
> at
> org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1335)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:878)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2119)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:567)
> at
> org.apache.hadoop.hbase.master.HMaster$$Lambda$42/1930759883.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> and
> {code}
> "master/vc0207:22001:becomeActiveMaster" #98 daemon prio=5 os_prio=0
> tid=0x00d31800 nid=0x1efc0 runnable [0x7f0a3c17d000]
>java.lang.Thread.State: RUNNABLE
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.security.Provider$Service.newInstance(Provider.java:1595)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.MessageDigest.getInstance(MessageDigest.java:167)
> at org.apache.hadoop.hbase.util.MD5Hash.getMD5AsHex(MD5Hash.java:59)
> at
> org.apache.hadoop.hbase.client.RegionInfo.createRegionName(RegionInfo.java:560)
> at
> org.apache.hadoop.hbase.client.RegionInfo.createRegionName(RegionInfo.java:490)
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.(RegionInfoBuilder.java:243)
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder.build(RegionInfoBuilder.java:120)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3132)
> at
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.deserializeStateData(ServerCrashProcedure.java:335)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProcedure(ProcedureUtil.java:283)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$Entry.convert(ProcedureWALFormatReader.java:359)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$EntryIterator.next(ProcedureWALFormatReader.java:410)
> at
[jira] [Updated] (HBASE-21165) During ProcedureStore load, there is no listing of progress...
[
https://issues.apache.org/jira/browse/HBASE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21165:
--
Attachment: HBASE-21165.branch-2.1.001.patch
> During ProcedureStore load, there is no listing of progress...
> --
>
> Key: HBASE-21165
> URL: https://issues.apache.org/jira/browse/HBASE-21165
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-21165.branch-2.1.001.patch
>
>
> I have a Master that crashed on a large cluster with hundreds of outstanding
> Procedure WALs and (probably --TBD) a few million Procedures to load. It is
> taking a long time (two hours)...
> There were STUCK procedures that were preventing clean-up of the old WALs.
> I can tell we are making progress by enabling TRACE on the Procedure Store.
> Better would be an emission as we made progress through the files with an
> output after every so many procedures loaded.
> Then, post-load, there is a long time spent sorting-out the Procedure
> image... We are in finish for ages doing stuff like:
> {code}
> "master/vc0207:22001:becomeActiveMaster" #98 daemon prio=5 os_prio=0
> tid=0x00d31800 nid=0x1efc0 runnable [0x7f0a3c17d000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$WalProcedureMap.removeFromMap(ProcedureWALFormatReader.java:837)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$WalProcedureMap.fetchReady(ProcedureWALFormatReader.java:614)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.finish(ProcedureWALFormatReader.java:201)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:94)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:426)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:382)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:663)
> at
> org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1335)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:878)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2119)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:567)
> at
> org.apache.hadoop.hbase.master.HMaster$$Lambda$42/1930759883.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> and
> {code}
> "master/vc0207:22001:becomeActiveMaster" #98 daemon prio=5 os_prio=0
> tid=0x00d31800 nid=0x1efc0 runnable [0x7f0a3c17d000]
>java.lang.Thread.State: RUNNABLE
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at java.security.Provider$Service.newInstance(Provider.java:1595)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.MessageDigest.getInstance(MessageDigest.java:167)
> at org.apache.hadoop.hbase.util.MD5Hash.getMD5AsHex(MD5Hash.java:59)
> at
> org.apache.hadoop.hbase.client.RegionInfo.createRegionName(RegionInfo.java:560)
> at
> org.apache.hadoop.hbase.client.RegionInfo.createRegionName(RegionInfo.java:490)
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder$MutableRegionInfo.(RegionInfoBuilder.java:243)
> at
> org.apache.hadoop.hbase.client.RegionInfoBuilder.build(RegionInfoBuilder.java:120)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toRegionInfo(ProtobufUtil.java:3132)
> at
> org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.deserializeStateData(ServerCrashProcedure.java:335)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProcedure(ProcedureUtil.java:283)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$Entry.convert(ProcedureWALFormatReader.java:359)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader$EntryIterator.next(ProcedureWALFormatReader.java:410)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.loadProcedures(ProcedureExecutor.java:460)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:76)
>
[jira] [Commented] (HBASE-21172) Reimplement the retry backoff logic for ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612484#comment-16612484
]
Hudson commented on HBASE-21172:
Results for branch branch-2.0
[build #805 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/805/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/805//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/805//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/805//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Reimplement the retry backoff logic for ReopenTableRegionsProcedure
> ---
>
> Key: HBASE-21172
> URL: https://issues.apache.org/jira/browse/HBASE-21172
> Project: HBase
> Issue Type: Sub-task
> Components: amv2, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21172-branch-2.1-v1.patch,
> HBASE-21172-branch-2.1-v1.patch, HBASE-21172-branch-2.1.patch,
> HBASE-21172-v1.patch, HBASE-21172-v2.patch, HBASE-21172-v3.patch,
> HBASE-21172-v4.patch, HBASE-21172.patch
>
>
> Now we just do a blocking sleep in the execute method, and there is no
> exponential backoff.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21179) Fix the number of actions in responseTooSlow log
[
https://issues.apache.org/jira/browse/HBASE-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612446#comment-16612446
]
Andrew Purtell commented on HBASE-21179:
+1 for branch-1 and all derived release branches from me
> Fix the number of actions in responseTooSlow log
>
>
> Key: HBASE-21179
> URL: https://issues.apache.org/jira/browse/HBASE-21179
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-21179.branch-1.001.patch,
> HBASE-21179.master.001.patch, HBASE-21179.master.002.patch
>
>
> {panel:title=responseTooSlow|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#CE}
> 2018-09-10 16:13:53,022 WARN
> [B.DefaultRpcServer.handler=209,queue=29,port=60020] ipc.RpcServer:
> (responseTooSlow):
> {"processingtimems":321262,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"127.0.0.1:56149","param":"region=
>
> tsdb,\\x00\\x00.[\\x89\\x1F\\xB0\\x00\\x00\\x01\\x00\\x01Y\\x00\\x00\\x02\\x00\\x00\\x04,1536133210446.7c752de470bd5558a001117b123a5db5.,
> {color:red}for 1 actions and 1st row{color}
> key=\\x00\\x00.[\\x96\\x16p","starttimems":1536566911759,"queuetimems":0,"class":"HRegionServer","responsesize":2,"method":"Multi"}
> {panel}
> The responseTooSlow log is printed when the processing time of a request
> exceeds the specified threshold. The number of actions and the contents of
> the first rowkey in the request will be included in the log.
> However, the number of actions is inaccurate, and it is actually the number
> of regions that the request needs to visit.
> Just like the logs above, users may be mistaken for using 321262ms to process
> an action, which is incredible, so we need to fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612413#comment-16612413
]
Toshihiro Suzuki commented on HBASE-21182:
--
It seems like the issue is fixed when commenting out maven-shade-plugin in
hbase-shaded/hbase-shaded-mapreduce/pom.xml. But I'm not sure why.
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Subrat Mishra
>Priority: Major
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612410#comment-16612410 ] stack commented on HBASE-20786: --- Sorry. Focus elsewhere. Let me get numbers here before and after. > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Priority: Major > Attachments: HBASE-20786.master.001.patch > > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21174) [REST] Failed to parse empty qualifier in TableResource#getScanResource
[
https://issues.apache.org/jira/browse/HBASE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-21174:
--
Resolution: Fixed
Fix Version/s: 2.0.3
2.1.1
1.4.8
2.2.0
1.5.0
3.0.0
Status: Resolved (was: Patch Available)
> [REST] Failed to parse empty qualifier in TableResource#getScanResource
> ---
>
> Key: HBASE-21174
> URL: https://issues.apache.org/jira/browse/HBASE-21174
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21174.branch-1.001.patch,
> HBASE-21174.master.001.patch, HBASE-21174.master.002.patch
>
>
> {code:xml}
> GET /t1/*?column=f:c1=f:
> {code}
> If I want to get the values of 'f:'(empty qualifier) for all rows in the
> table by rest server, I will send the above request. However, this request
> will return all column values.
> {code:java|title=TableResource#getScanResource|borderStyle=solid}
> for (String csplit : column) {
> String[] familysplit = csplit.trim().split(":");
> if (familysplit.length == 2) {
> if (familysplit[1].length() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family and column : " + familysplit[0] + " " +
> familysplit[1]);
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]),
> Bytes.toBytes(familysplit[1]));
> } else {
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0] + " and empty
> qualifier.");
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]), null);
> }
> } else if (StringUtils.isNotEmpty(familysplit[0])) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0]);
> }
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> }
> }
> {code}
> Through the above code, when the column has an empty qualifier, the empty
> qualifier cannot be parsed correctly.In other words, 'f:'(empty qualifier)
> and 'f' (column family) are considered to have the same meaning, which is
> wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612357#comment-16612357 ] Mike Drob commented on HBASE-20786: --- Any improvements seen, mighty [~stack]? > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Priority: Major > Attachments: HBASE-20786.master.001.patch > > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21174) [REST] Failed to parse empty qualifier in TableResource#getScanResource
[
https://issues.apache.org/jira/browse/HBASE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612334#comment-16612334
]
Hadoop QA commented on HBASE-21174:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} hbase-rest: The patch generated 0 new + 16 unchanged
- 10 fixed = 16 total (was 26) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
38s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
29s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21174 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939419/HBASE-21174.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9afb645f6987 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64
[jira] [Commented] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612297#comment-16612297 ] Mike Drob commented on HBASE-21168: --- I didn't backport this to branch-1 because there was a cherry-pick conflict, but it's probably pretty trivial to resolve if somebody is interested. > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-21168.master.001.patch, > HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21097) Flush pressure assertion may fail in testFlushThroughputTuning
[
https://issues.apache.org/jira/browse/HBASE-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21097:
---
Attachment: (was: 21097.v3.txt)
> Flush pressure assertion may fail in testFlushThroughputTuning
> ---
>
> Key: HBASE-21097
> URL: https://issues.apache.org/jira/browse/HBASE-21097
> Project: HBase
> Issue Type: Test
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21097.v1.txt, 21097.v2.txt, 21097.v4.txt,
> HBASE-21097.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14137/artifact/patchprocess/patch-unit-hbase-server.txt
> :
> {code}
> [ERROR]
> testFlushThroughputTuning(org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController)
> Time elapsed: 17.446 s <<< FAILURE!
> java.lang.AssertionError: expected:<0.0> but was:<1.2906294173808417E-6>
> at
> org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController.testFlushThroughputTuning(TestFlushWithThroughputController.java:185)
> {code}
> Here is the related assertion:
> {code}
> assertEquals(0.0, regionServer.getFlushPressure(), EPSILON);
> {code}
> where EPSILON = 1E-6
> In the above case, due to margin of 2.9E-7, the assertion didn't pass.
> It seems the epsilon can be adjusted to accommodate different workload /
> hardware combination.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21097) Flush pressure assertion may fail in testFlushThroughputTuning
[
https://issues.apache.org/jira/browse/HBASE-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21097:
---
Attachment: 21097.v4.txt
> Flush pressure assertion may fail in testFlushThroughputTuning
> ---
>
> Key: HBASE-21097
> URL: https://issues.apache.org/jira/browse/HBASE-21097
> Project: HBase
> Issue Type: Test
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: 21097.v1.txt, 21097.v2.txt, 21097.v4.txt,
> HBASE-21097.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14137/artifact/patchprocess/patch-unit-hbase-server.txt
> :
> {code}
> [ERROR]
> testFlushThroughputTuning(org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController)
> Time elapsed: 17.446 s <<< FAILURE!
> java.lang.AssertionError: expected:<0.0> but was:<1.2906294173808417E-6>
> at
> org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController.testFlushThroughputTuning(TestFlushWithThroughputController.java:185)
> {code}
> Here is the related assertion:
> {code}
> assertEquals(0.0, regionServer.getFlushPressure(), EPSILON);
> {code}
> where EPSILON = 1E-6
> In the above case, due to margin of 2.9E-7, the assertion didn't pass.
> It seems the epsilon can be adjusted to accommodate different workload /
> hardware combination.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21188) Print heap and gc informations in our junit ResourceChecker
[
https://issues.apache.org/jira/browse/HBASE-21188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612290#comment-16612290
]
Hadoop QA commented on HBASE-21188:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939412/HBASE-21188.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux eb4b3a08e03c 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3810ba2c6e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14398/testReport/ |
| Max. process+thread count | 361 (vs. ulimit of 1) |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14398/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Print heap and gc informations in our
[jira] [Updated] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-21168: -- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Thanks for reviews [~apurtell], [~liuml07]! > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-21168.master.001.patch, > HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21174) [REST] Failed to parse empty qualifier in TableResource#getScanResource
[
https://issues.apache.org/jira/browse/HBASE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-21174:
--
Attachment: HBASE-21174.branch-1.001.patch
> [REST] Failed to parse empty qualifier in TableResource#getScanResource
> ---
>
> Key: HBASE-21174
> URL: https://issues.apache.org/jira/browse/HBASE-21174
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-21174.branch-1.001.patch,
> HBASE-21174.master.001.patch, HBASE-21174.master.002.patch
>
>
> {code:xml}
> GET /t1/*?column=f:c1=f:
> {code}
> If I want to get the values of 'f:'(empty qualifier) for all rows in the
> table by rest server, I will send the above request. However, this request
> will return all column values.
> {code:java|title=TableResource#getScanResource|borderStyle=solid}
> for (String csplit : column) {
> String[] familysplit = csplit.trim().split(":");
> if (familysplit.length == 2) {
> if (familysplit[1].length() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family and column : " + familysplit[0] + " " +
> familysplit[1]);
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]),
> Bytes.toBytes(familysplit[1]));
> } else {
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0] + " and empty
> qualifier.");
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]), null);
> }
> } else if (StringUtils.isNotEmpty(familysplit[0])) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0]);
> }
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> }
> }
> {code}
> Through the above code, when the column has an empty qualifier, the empty
> qualifier cannot be parsed correctly.In other words, 'f:'(empty qualifier)
> and 'f' (column family) are considered to have the same meaning, which is
> wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612261#comment-16612261 ] stack commented on HBASE-21035: --- bq. Shouldn't we just trust the meta location in the ZK node? Maybe I could look there first. If I find something, go with it. Otherwise, hunt all WAL dirs for .meta WALs. That'd be best. I'll have to take the lease on any WALs I find if only to read them. Maybe I should delete them when done though I think there should be no damage done if they are replayed. I need to figure how to split inline rather than ask our splitting service to do it for us. bq. So maybe the problem here is that, we should make master retrying for a longer time before exiting, and add a new method in the rpc service, which is for hbck2 to schedule some recovery procedures? Was thinking on this. Yeah, let me change the isMeta method here so that rather than schedule the assign of meta procedure, instead I drop logs asking the operator to schedule a meta assign and perhaps a procedure that does the recover of meta WALs. Then do the same for namespace (Namespace needs to go away). Let me try it. > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21189) flaky job should gather machine stats
[
https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612223#comment-16612223
]
Hadoop QA commented on HBASE-21189:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21189 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939413/HBASE-21189.0.patch |
| Optional Tests | asflicense |
| uname | Linux ff1ed442ce35 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3810ba2c6e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Max. process+thread count | 43 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14399/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> flaky job should gather machine stats
> -
>
> Key: HBASE-21189
> URL: https://issues.apache.org/jira/browse/HBASE-21189
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Attachments: HBASE-21189.0.patch
>
>
> flaky test should gather all the same environment information as our normal
> nightly tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21189: Status: Patch Available (was: In Progress) > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21189) flaky job should gather machine stats
[ https://issues.apache.org/jira/browse/HBASE-21189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21189: Attachment: HBASE-21189.0.patch > flaky job should gather machine stats > - > > Key: HBASE-21189 > URL: https://issues.apache.org/jira/browse/HBASE-21189 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Attachments: HBASE-21189.0.patch > > > flaky test should gather all the same environment information as our normal > nightly tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21174) [REST] Failed to parse empty qualifier in TableResource#getScanResource
[
https://issues.apache.org/jira/browse/HBASE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612212#comment-16612212
]
Hudson commented on HBASE-21174:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [REST] Failed to parse empty qualifier in TableResource#getScanResource
> ---
>
> Key: HBASE-21174
> URL: https://issues.apache.org/jira/browse/HBASE-21174
> Project: HBase
> Issue Type: Bug
> Components: REST
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-21174.master.001.patch,
> HBASE-21174.master.002.patch
>
>
> {code:xml}
> GET /t1/*?column=f:c1=f:
> {code}
> If I want to get the values of 'f:'(empty qualifier) for all rows in the
> table by rest server, I will send the above request. However, this request
> will return all column values.
> {code:java|title=TableResource#getScanResource|borderStyle=solid}
> for (String csplit : column) {
> String[] familysplit = csplit.trim().split(":");
> if (familysplit.length == 2) {
> if (familysplit[1].length() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family and column : " + familysplit[0] + " " +
> familysplit[1]);
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]),
> Bytes.toBytes(familysplit[1]));
> } else {
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0] + " and empty
> qualifier.");
> }
> tableScan.addColumn(Bytes.toBytes(familysplit[0]), null);
> }
> } else if (StringUtils.isNotEmpty(familysplit[0])) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Scan family : " + familysplit[0]);
> }
> tableScan.addFamily(Bytes.toBytes(familysplit[0]));
> }
> }
> {code}
> Through the above code, when the column has an empty qualifier, the empty
> qualifier cannot be parsed correctly.In other words, 'f:'(empty qualifier)
> and 'f' (column family) are considered to have the same meaning, which is
> wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21021) Result returned by Append operation should be ordered
[
https://issues.apache.org/jira/browse/HBASE-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612209#comment-16612209
]
Hudson commented on HBASE-21021:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Result returned by Append operation should be ordered
> -
>
> Key: HBASE-21021
> URL: https://issues.apache.org/jira/browse/HBASE-21021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.5.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21021.branch-1.001.patch,
> HBASE-21021.master.001.patch
>
>
> *Problem:*
> The result returned by the append operation should be ordered. Currently, it
> returns an unordered list, which may cause problems like if the user tries to
> perform Result.getValue(byte[] family, byte[] qualifier), even if the
> returned result has a value corresponding to (family, qualifier), the method
> may return null as it performs a binary search over the unsorted result
> (which should have been sorted actually).
>
> The result is enumerated by iterating over each entry of tempMemstore hashmap
> (which will never be ordered) and adding the values (see
> [HRegion.java#L7882|https://github.com/apache/hbase/blob/1b50fe53724aa62a242b7f64adf7845048df/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L7882]).
>
> *Actual:* The returned result is unordered
> *Expected:* Similar to increment op, the returned result should be ordered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21179) Fix the number of actions in responseTooSlow log
[
https://issues.apache.org/jira/browse/HBASE-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612213#comment-16612213
]
Hudson commented on HBASE-21179:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix the number of actions in responseTooSlow log
>
>
> Key: HBASE-21179
> URL: https://issues.apache.org/jira/browse/HBASE-21179
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-21179.branch-1.001.patch,
> HBASE-21179.master.001.patch, HBASE-21179.master.002.patch
>
>
> {panel:title=responseTooSlow|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#CE}
> 2018-09-10 16:13:53,022 WARN
> [B.DefaultRpcServer.handler=209,queue=29,port=60020] ipc.RpcServer:
> (responseTooSlow):
> {"processingtimems":321262,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"127.0.0.1:56149","param":"region=
>
> tsdb,\\x00\\x00.[\\x89\\x1F\\xB0\\x00\\x00\\x01\\x00\\x01Y\\x00\\x00\\x02\\x00\\x00\\x04,1536133210446.7c752de470bd5558a001117b123a5db5.,
> {color:red}for 1 actions and 1st row{color}
> key=\\x00\\x00.[\\x96\\x16p","starttimems":1536566911759,"queuetimems":0,"class":"HRegionServer","responsesize":2,"method":"Multi"}
> {panel}
> The responseTooSlow log is printed when the processing time of a request
> exceeds the specified threshold. The number of actions and the contents of
> the first rowkey in the request will be included in the log.
> However, the number of actions is inaccurate, and it is actually the number
> of regions that the request needs to visit.
> Just like the logs above, users may be mistaken for using 321262ms to process
> an action, which is incredible, so we need to fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21098) Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
[
https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612211#comment-16612211
]
Hudson commented on HBASE-21098:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir
> on S3
> -
>
> Key: HBASE-21098
> URL: https://issues.apache.org/jira/browse/HBASE-21098
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 1.4.8, 2.1.1
>Reporter: Tyler Mi
>Priority: Major
> Attachments: HBASE-21098.master.001.patch,
> HBASE-21098.master.002.patch, HBASE-21098.master.003.patch,
> HBASE-21098.master.004.patch, HBASE-21098.master.005.patch,
> HBASE-21098.master.006.patch, HBASE-21098.master.007.patch,
> HBASE-21098.master.008.patch, HBASE-21098.master.009.patch,
> HBASE-21098.master.010.patch, HBASE-21098.master.011.patch,
> HBASE-21098.master.012.patch, HBASE-21098.master.013.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in
> time recovery. To do this, HBase creates a manifest of all the files in all
> of the Regions so that those files can be referenced again when a user
> restores a snapshot. With HBase's S3 storage mode, developers can store their
> data off-cluster on Amazon S3. However, utilizing S3 as a file system is
> inefficient in some operations, namely renames. Most Hadoop ecosystem
> applications use an atomic rename as a method of committing data. However,
> with S3, a rename is a separate copy and then a delete of every file which is
> no longer atomic and, in fact, quite costly. In addition, puts and deletes on
> S3 have latency issues that traditional filesystems do not encounter when
> manipulating the region snapshots to consolidate into a single manifest. When
> HBase on S3 users have a significant amount of regions, puts, deletes, and
> renames (the final commit stage of the snapshot) become the bottleneck
> causing snapshots to take many minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots
> while utilizing HBase on S3 through the use of a temporary directory for the
> snapshots that exists on a traditional filesystem like HDFS to circumvent the
> bottlenecks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21172) Reimplement the retry backoff logic for ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612208#comment-16612208
]
Hudson commented on HBASE-21172:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reimplement the retry backoff logic for ReopenTableRegionsProcedure
> ---
>
> Key: HBASE-21172
> URL: https://issues.apache.org/jira/browse/HBASE-21172
> Project: HBase
> Issue Type: Sub-task
> Components: amv2, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21172-branch-2.1-v1.patch,
> HBASE-21172-branch-2.1-v1.patch, HBASE-21172-branch-2.1.patch,
> HBASE-21172-v1.patch, HBASE-21172-v2.patch, HBASE-21172-v3.patch,
> HBASE-21172-v4.patch, HBASE-21172.patch
>
>
> Now we just do a blocking sleep in the execute method, and there is no
> exponential backoff.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21181) Use the same filesystem for wal archive directory and wal directory
[
https://issues.apache.org/jira/browse/HBASE-21181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612210#comment-16612210
]
Hudson commented on HBASE-21181:
Results for branch master
[build #487 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/487/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/487//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use the same filesystem for wal archive directory and wal directory
> ---
>
> Key: HBASE-21181
> URL: https://issues.apache.org/jira/browse/HBASE-21181
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21181.master.001.patch
>
>
> when {{hbase.wal.dir}} is set to any filesystem other than the same
> filesystem used by rootDir e.g. {{hbase.wal.dir}} set to
> {{hdfs://something/wal}} and {{hbase.rootdir}} set to {{s3://something}},
> before this change, WAL archive directory ({{walArchiveDir}}) cannot be
> created and failed the WALProcedureStore on HMaster.
> The issue is that WAL archive directory was considered to be collocated with
> {{hbase.rootdir}} and creates a subdirectory under it. this logic needs to be
> updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21188) Print heap and gc informations in our junit ResourceChecker
[ https://issues.apache.org/jira/browse/HBASE-21188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612207#comment-16612207 ] Duo Zhang commented on HBASE-21188: --- Maybe we should not define GC as a resource... > Print heap and gc informations in our junit ResourceChecker > --- > > Key: HBASE-21188 > URL: https://issues.apache.org/jira/browse/HBASE-21188 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21188.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
