[jira] [Updated] (HBASE-16832) Reduce the default number of versions in Meta table for branch-1
[
https://issues.apache.org/jira/browse/HBASE-16832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16832:
---
Attachment: 16832.branch-1.patch
Same patch.
The subsequent png upload confuses the bot:
{code}
HBASE-16832 patch is being downloaded at Fri Oct 14 16:20:38 UTC 2016 from
https://issues.apache.org/jira/secure/attachment/12833282/rpc_queueSize.png
-> Downloaded
ERROR: Unsure how to process HBASE-16832.
{code}
> Reduce the default number of versions in Meta table for branch-1
>
>
> Key: HBASE-16832
> URL: https://issues.apache.org/jira/browse/HBASE-16832
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.3.0
>Reporter: binlijin
>Assignee: binlijin
> Attachments: 16832.branch-1.patch, HBASE-16832.branch-1.patch,
> rpc-handler.png, rpc_processingCallTime.png, rpc_processingCallTimeV2.png,
> rpc_qps.png, rpc_queueLength.png, rpc_queueSize.png, rpc_scan_latency.png,
> rpc_scan_latencyV2.png, rpc_totalcalltime.png, rpc_totalcalltimeV2.png
>
>
> I find the DEFAULT_HBASE_META_VERSIONS is still 10 in branch-1, and in master
> version DEFAULT_HBASE_META_VERSIONS is 3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16840) Reuse cell's timestamp and type in ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16840: - Attachment: HBASE-16840_master_V2.patch > Reuse cell's timestamp and type in ScanQueryMatcher > --- > > Key: HBASE-16840 > URL: https://issues.apache.org/jira/browse/HBASE-16840 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16840_master.patch, HBASE-16840_master_V2.patch > > > Reuse cell's timestamp and type in ScanQueryMatcher, this is useful for > KeyValue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16842) Chaos policies can terminate all masters for extended periods of time
Andrew Purtell created HBASE-16842: -- Summary: Chaos policies can terminate all masters for extended periods of time Key: HBASE-16842 URL: https://issues.apache.org/jira/browse/HBASE-16842 Project: HBase Issue Type: Bug Components: integration tests Reporter: Andrew Purtell Running ITBLL with the slowDeterministic monkey I observe our primary and backup masters in the test cluster can be both shut down by signals followed by no attempt to start replacements for an extended period of time. Meanwhile other actions continue to run that churn the regionserver fleet. Other monkeys may enter a similar state, but I haven't observed it. The outcome of the convergence of these behaviors is the eventual time out and termination of the running integration test, which is obvious and expected and unhelpful, so I believe unintentional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16840) Reuse cell's timestamp and type in ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575824#comment-15575824 ] binlijin commented on HBASE-16840: -- I do not test the performance, and i think this will not have much gain in performance. but it will better than before, because this is odd, so upload HBASE-16840_master_V2.patch, what do you think of the new patch? > Reuse cell's timestamp and type in ScanQueryMatcher > --- > > Key: HBASE-16840 > URL: https://issues.apache.org/jira/browse/HBASE-16840 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16840_master.patch, HBASE-16840_master_V2.patch > > > Reuse cell's timestamp and type in ScanQueryMatcher, this is useful for > KeyValue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16799) CP exposed Store should not expose unwanted APIs
[ https://issues.apache.org/jira/browse/HBASE-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16799: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Release Note: Below APIs from CP exposed Store interface are removed upsert(Iterable cells, long readpoint) add(Cell cell) add(Iterable cells) replayCompactionMarker(CompactionDescriptor compaction, boolean pickCompactionFiles, boolean removeFiles) assertBulkLoadHFileOk(Path srcPath) bulkLoadHFile(String srcPathStr, long sequenceId) bulkLoadHFile(StoreFileInfo fileInfo) Status: Resolved (was: Patch Available) Pushed to master. Thanks for the reviews Stack and Ram. > CP exposed Store should not expose unwanted APIs > - > > Key: HBASE-16799 > URL: https://issues.apache.org/jira/browse/HBASE-16799 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16799.patch, HBASE-16799.patch, > HBASE-16799_V2.patch, HBASE-16799_V3.patch, HBASE-16799_V4.patch > > > Store is exposed to CPs. The main use cases I can think of are getting store > scanner and other getters which return different states like memstore size, > max seqId etc. Those make sense. > But we added many other APIs which changes the state of the memstore, bulk > load files etc into this interface. Even an API which expose the memstore > itself!. This allow adding mutations into memstore bypassing all steps in > region. We track the memstore size per region level as well as globally. > These only allow us to flush region at sizes and/or flush selected regions > because of global heap pressure. Now if a CP get hold of store and/or > memstore, it can add mutations with out knowledge of these size accounting > and possibly OOME the RS. Similar way the bulk load related APIs. At HRegion > level, there are steps done (WAL write etc) after the bulk load HFile on > store. So bypassing these wont be correct. > In this jira, plan is to remove all such leaked APIs from Store. They are > called from HRegion and we can type cast to HStore to call them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16792) Reuse KeyValue.KeyOnlyKeyValue in BufferedDataBlockEncoder.SeekerState
[ https://issues.apache.org/jira/browse/HBASE-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575849#comment-15575849 ] Ted Yu commented on HBASE-16792: Please correct the commit message - JIRA number: iHBASE-16792 Reuse KeyValue.KeyOnlyKeyValue in BufferedDataBlockEncoder.SeekerState (Binlijin) > Reuse KeyValue.KeyOnlyKeyValue in BufferedDataBlockEncoder.SeekerState > -- > > Key: HBASE-16792 > URL: https://issues.apache.org/jira/browse/HBASE-16792 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16792_master.patch, HBASE-16792_master_v2.patch, > HBASE-16792_master_v3.patch > > > When every SeekerState#invalidate will new a fresh KeyValue.KeyOnlyKeyValue, > we should reuse it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16840) Reuse cell's timestamp and type in ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575852#comment-15575852 ] Anoop Sam John commented on HBASE-16840: Looks better. Am +1 > Reuse cell's timestamp and type in ScanQueryMatcher > --- > > Key: HBASE-16840 > URL: https://issues.apache.org/jira/browse/HBASE-16840 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16840_master.patch, HBASE-16840_master_V2.patch > > > Reuse cell's timestamp and type in ScanQueryMatcher, this is useful for > KeyValue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
[ https://issues.apache.org/jira/browse/HBASE-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16783: --- Attachment: (was: HBASE-16783_2.patch) > Use ByteBufferPool for the header and message during Rpc response > - > > Key: HBASE-16783 > URL: https://issues.apache.org/jira/browse/HBASE-16783 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Attachments: HBASE-16783.patch, HBASE-16783_1.patch > > > With ByteBufferPool in place we could avoid the byte[] creation in > RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool > rather than creating byte[] every time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16840) Reuse cell's timestamp and type in ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-16840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575107#comment-15575107 ] Anoop Sam John commented on HBASE-16840: Ya if it is really giving some boost we can do.. I agree it looks bit odd.. Specially this stuff in DeleteTracker add(Cell, long ts, byte type) > Reuse cell's timestamp and type in ScanQueryMatcher > --- > > Key: HBASE-16840 > URL: https://issues.apache.org/jira/browse/HBASE-16840 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16840_master.patch > > > Reuse cell's timestamp and type in ScanQueryMatcher, this is useful for > KeyValue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16729) Define the behavior of (default) empty FilterList
[
https://issues.apache.org/jira/browse/HBASE-16729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575190#comment-15575190
]
ramkrishna.s.vasudevan commented on HBASE-16729:
QA looks green. Findbugs is not related to the patch.
> Define the behavior of (default) empty FilterList
> -
>
> Key: HBASE-16729
> URL: https://issues.apache.org/jira/browse/HBASE-16729
> Project: HBase
> Issue Type: Wish
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-16729.v0.patch, HBASE-16729.v1.patch,
> HBASE-16729.v2.patch
>
>
> Current empty FilterList filters all data, because the
> FilterList#isFamilyEssential always returns false which causes the null cell
> retrieved by RegionScannerImpl.storeHeap.
> It seems to me that empty FilterList should do nothing, because the following
> code is common.
> {noformat}
> private static Filter makeFilter() {
> FilterList filterList = new FilterList ();
> for (some conditions) {
> // add some filters. Or nothing to add.
> }
> return filterList;
> }
> {noformat}
> If we keep the current logic which filters all data, we should add enough
> comments to explain it. Or add the FilterList#size() or FilterList#empty()
> for preventing filtering all data.
> Any comments? Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[
https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575250#comment-15575250
]
Hudson commented on HBASE-16807:
FAILURE: Integrated in Jenkins build HBase-0.98-matrix #410 (See
[https://builds.apache.org/job/HBase-0.98-matrix/410/])
HBASE-16807, RegionServer will fail to report new active Hmaster until
(chenheng: rev 09b89eedb585517d734bf3af3875e0e1ff5359e0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
> RegionServer will fail to report new active Hmaster until
> HMaster/RegionServer failover
> ---
>
> Key: HBASE-16807
> URL: https://issues.apache.org/jira/browse/HBASE-16807
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 0.98.24, 1.1.8
>
> Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch,
> HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch,
> HBASE-16807-branch-1.patch, HBASE-16807.patch
>
>
> It's little weird, but it happened in the product environment that few
> RegionServer missed master znode create notification on master failover. In
> that case ZooKeeperNodeTracker will not refresh the cached data and
> MasterAddressTracker will always return old active HM detail to Region server
> on ServiceException.
> Though We create region server stub on failure but without refreshing the
> MasterAddressTracker data.
> In HRegionServer.createRegionServerStatusStub()
> {code}
> boolean refresh = false; // for the first time, use cached data
> RegionServerStatusService.BlockingInterface intf = null;
> boolean interrupted = false;
> try {
> while (keepLooping()) {
> sn = this.masterAddressTracker.getMasterAddress(refresh);
> if (sn == null) {
> if (!keepLooping()) {
> // give up with no connection.
> LOG.debug("No master found and cluster is stopped; bailing out");
> return null;
> }
> if (System.currentTimeMillis() > (previousLogTime + 1000)) {
> LOG.debug("No master found; retry");
> previousLogTime = System.currentTimeMillis();
> }
> refresh = true; // let's try pull it from ZK directly
> if (sleep(200)) {
> interrupted = true;
> }
> continue;
> }
> {code}
> Here we refresh node only when 'sn' is NULL otherwise it will use same cached
> data.
> So in above case RegionServer will never report active HMaster successfully
> until HMaster failover or RegionServer restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16839) Procedure v2 - Move all protobuf handling to ProcedureUtil
[ https://issues.apache.org/jira/browse/HBASE-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16839: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Move all protobuf handling to ProcedureUtil > -- > > Key: HBASE-16839 > URL: https://issues.apache.org/jira/browse/HBASE-16839 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16839-v0.patch > > > At the moment we have some of the protobuf conversion in Procedure and some > in ProcedureUtil, let's move all the serialization to ProcedureUtil and try > to keep the Procedure object "clean". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16845) Run tests under hbase-spark module in test phase
Ted Yu created HBASE-16845: -- Summary: Run tests under hbase-spark module in test phase Key: HBASE-16845 URL: https://issues.apache.org/jira/browse/HBASE-16845 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu [~appy] reported over in the discussion thread titled 'Testing and CI' that tests under hbase-spark module are not run by QA bot. This issue is to run the unit tests in test phase so that we detect build breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Status: Patch Available (was: Open) > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16845.v1.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Attachment: 16845.v1.txt > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Fix Version/s: 2.0.0 > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14167) hbase-spark integration tests do not respect -DskipIntegrationTests
[
https://issues.apache.org/jira/browse/HBASE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14167:
Assignee: (was: Sean Busbey)
> hbase-spark integration tests do not respect -DskipIntegrationTests
> ---
>
> Key: HBASE-14167
> URL: https://issues.apache.org/jira/browse/HBASE-14167
> Project: HBase
> Issue Type: Improvement
> Components: pom, spark
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Minor
> Attachments: HBASE-14167-master-v2.patch, HBASE-14167.11.patch
>
>
> When running a build with {{mvn ... -DskipIntegrationTests}}, the hbase-spark
> module's integration tests do not respect the flag and run anyway. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16698:
--
Affects Version/s: (was: 1.1.6)
Ok, let me explain.
One of the main difference between our customized 1.1.2 and current branch-1.1
is that I've backported HBASE-14465 (Allow rowlock to be read/write in
branch-1) to improve the non-conditional put performance. HBASE-14465 only goes
in branch-1.2+, and this is what cause the confusion. Please check the latest
branch-1 code and you'll find out what I'm trying to fix in this JIRA
[~allan163]
Removing 1.1.6 from "Affects Versions" to avoid the confusion, and thanks for
pointing this out [~allan163].
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch, HBASE-16698.patch,
> HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576217#comment-15576217 ] Ted Yu commented on HBASE-16845: I logged this JIRA before seeing Appy's reply w.r.t. HBASE-14167 IMHO running hbase-spark tests through QA bot is different from what HBASE-14167 stated. > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16823) Add examples in HBase Spark module
[ https://issues.apache.org/jira/browse/HBASE-16823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576062#comment-15576062 ] Weiqing Yang commented on HBASE-16823: -- [~tedyu] Thanks for the review. > Add examples in HBase Spark module > -- > > Key: HBASE-16823 > URL: https://issues.apache.org/jira/browse/HBASE-16823 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-16823_v0.patch, HBASE-16823_v1.patch > > > This patch adds examples that show how to use Spark Dataframe to read and > write Hbase tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576097#comment-15576097
]
stack commented on HBASE-16698:
---
Pardon me. Was looking in master branch.
Let me revert this patch from master branch. This discussion is not done.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.1.6, 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch, HBASE-16698.patch,
> HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16844) Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback
Stephen Yuan Jiang created HBASE-16844: -- Summary: Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback Key: HBASE-16844 URL: https://issues.apache.org/jira/browse/HBASE-16844 Project: HBase Issue Type: Improvement Components: master, proc-v2 Affects Versions: 2.0.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Priority: Trivial HBASE-16507 missed the DispatchMergingRegionsProcedure to use base class for procedure abort and rollback. This ticket fixes DispatchMergingRegionsProcedure to use base class's abort() and setNextState(); and overrides isRollbackSupported(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-16698:
---
I reverted the commit against master. Discussion revived with some questions to
answer.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.1.6, 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch, HBASE-16698.patch,
> HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16844) Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback
[ https://issues.apache.org/jira/browse/HBASE-16844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16844: --- Status: Patch Available (was: Open) > Procedure V2: DispatchMergingRegionsProcedure to use base class > StateMachineProcedure for abort and rollback > - > > Key: HBASE-16844 > URL: https://issues.apache.org/jira/browse/HBASE-16844 > Project: HBase > Issue Type: Improvement > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-16844.v1-master.patch > > > HBASE-16507 missed the DispatchMergingRegionsProcedure to use base class for > procedure abort and rollback. This ticket fixes > DispatchMergingRegionsProcedure to use base class's abort() and > setNextState(); and overrides isRollbackSupported(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16844) Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback
[ https://issues.apache.org/jira/browse/HBASE-16844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16844: --- Attachment: HBASE-16844.v1-master.patch > Procedure V2: DispatchMergingRegionsProcedure to use base class > StateMachineProcedure for abort and rollback > - > > Key: HBASE-16844 > URL: https://issues.apache.org/jira/browse/HBASE-16844 > Project: HBase > Issue Type: Improvement > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-16844.v1-master.patch > > > HBASE-16507 missed the DispatchMergingRegionsProcedure to use base class for > procedure abort and rollback. This ticket fixes > DispatchMergingRegionsProcedure to use base class's abort() and > setNextState(); and overrides isRollbackSupported(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16818) Avoid multiple copies of binary data during the conversion from Result to Row
[ https://issues.apache.org/jira/browse/HBASE-16818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576065#comment-15576065 ] Weiqing Yang commented on HBASE-16818: -- [~tedyu] Thanks for the review. > Avoid multiple copies of binary data during the conversion from Result to Row > - > > Key: HBASE-16818 > URL: https://issues.apache.org/jira/browse/HBASE-16818 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-16818_v0.patch > > > In the buildRow() of HBaseRelation, CellUtil.cloneValue will already create a > copy of the data. If the data type is BinaryType, another copy is being made > within Utils.hbaseFieldToScalaType in Utils.scala. Generally, binary data can > be fairly large, so copying may be an expensive operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16844) Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback
[ https://issues.apache.org/jira/browse/HBASE-16844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16844: --- Fix Version/s: 2.0.0 > Procedure V2: DispatchMergingRegionsProcedure to use base class > StateMachineProcedure for abort and rollback > - > > Key: HBASE-16844 > URL: https://issues.apache.org/jira/browse/HBASE-16844 > Project: HBase > Issue Type: Improvement > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16844.v1-master.patch > > > HBASE-16507 missed the DispatchMergingRegionsProcedure to use base class for > procedure abort and rollback. This ticket fixes > DispatchMergingRegionsProcedure to use base class's abort() and > setNextState(); and overrides isRollbackSupported(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16844) Procedure V2: DispatchMergingRegionsProcedure to use base class StateMachineProcedure for abort and rollback
[ https://issues.apache.org/jira/browse/HBASE-16844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576117#comment-15576117 ] Matteo Bertozzi commented on HBASE-16844: - +1 > Procedure V2: DispatchMergingRegionsProcedure to use base class > StateMachineProcedure for abort and rollback > - > > Key: HBASE-16844 > URL: https://issues.apache.org/jira/browse/HBASE-16844 > Project: HBase > Issue Type: Improvement > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16844.v1-master.patch > > > HBASE-16507 missed the DispatchMergingRegionsProcedure to use base class for > procedure abort and rollback. This ticket fixes > DispatchMergingRegionsProcedure to use base class's abort() and > setNextState(); and overrides isRollbackSupported(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576234#comment-15576234
]
Stephen Yuan Jiang commented on HBASE-16014:
LGTM.
[~brandboat], are you planning to do more tests? If not, we can commit it.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014_v0.patch, HBASE-16014_v1.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576234#comment-15576234
]
Stephen Yuan Jiang edited comment on HBASE-16014 at 10/14/16 7:19 PM:
--
LGTM.
[~brandboat], are you planning to do more tests? If not, we can commit it.
FYI. [~ndimiduk]
was (Author: syuanjiang):
LGTM.
[~brandboat], are you planning to do more tests? If not, we can commit it.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014_v0.patch, HBASE-16014_v1.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Attachment: 16845.v2.txt Patch v2 renames skipSparkTests profile as skipIntegrationTests > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16014) Get and Put constructor argument lists are divergent
[
https://issues.apache.org/jira/browse/HBASE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576258#comment-15576258
]
brandboat commented on HBASE-16014:
---
I am not going to do more tests. Thanks for reviewing it.
> Get and Put constructor argument lists are divergent
>
>
> Key: HBASE-16014
> URL: https://issues.apache.org/jira/browse/HBASE-16014
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: brandboat
> Fix For: 2.0.0
>
> Attachments: HBASE-16014_v0.patch, HBASE-16014_v1.patch
>
>
> API for construing Get and Put objects for a specific rowkey is quite
> different.
> [Put|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html#constructor_summary]
> supports many more variations for specifying the target rowkey and timestamp
> compared to
> [Get|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#constructor_summary].
> Notably lacking are {{Get(byte[], int, int)}} and {{Get(ByteBuffer)}}
> variations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16617) Procedure V2 - Improvements
[ https://issues.apache.org/jira/browse/HBASE-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16617: --- Summary: Procedure V2 - Improvements (was: Procedure v2 - Improvements) > Procedure V2 - Improvements > --- > > Key: HBASE-16617 > URL: https://issues.apache.org/jira/browse/HBASE-16617 > Project: HBase > Issue Type: Umbrella > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > > Umbrella jira for all the jiras that are adding helpers, utilities and > improvement to the proc framework or to the master procs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16617) Procedure V2 - Improvements
[ https://issues.apache.org/jira/browse/HBASE-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16617: --- Component/s: master > Procedure V2 - Improvements > --- > > Key: HBASE-16617 > URL: https://issues.apache.org/jira/browse/HBASE-16617 > Project: HBase > Issue Type: Umbrella > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > > Umbrella jira for all the jiras that are adding helpers, utilities and > improvement to the proc framework or to the master procs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16617) Procedure V2 - Improvements
[ https://issues.apache.org/jira/browse/HBASE-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16617: --- Affects Version/s: 2.0.0 > Procedure V2 - Improvements > --- > > Key: HBASE-16617 > URL: https://issues.apache.org/jira/browse/HBASE-16617 > Project: HBase > Issue Type: Umbrella > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > > Umbrella jira for all the jiras that are adding helpers, utilities and > improvement to the proc framework or to the master procs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16642) Procedure
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16642: --- Summary: Procedure (was: Use DelayQueue instead of TimeoutBlockingQueue) > Procedure > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642-v3.patch, > HBASE-16642-v4.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16642) Procedure V2:
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16642: --- Summary: Procedure V2: (was: Procedure ) > Procedure V2: > -- > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642-v3.patch, > HBASE-16642-v4.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16642) Procedure V2: Use DelayQueue instead of TimeoutBlockingQueue
[ https://issues.apache.org/jira/browse/HBASE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16642: --- Summary: Procedure V2: Use DelayQueue instead of TimeoutBlockingQueue (was: Procedure V2: ) > Procedure V2: Use DelayQueue instead of TimeoutBlockingQueue > > > Key: HBASE-16642 > URL: https://issues.apache.org/jira/browse/HBASE-16642 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Hiroshi Ikeda >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16642-v2.patch, HBASE-16642-v3.patch, > HBASE-16642-v4.patch, HBASE-16642.master.V1.patch > > > Enqueue poisons in order to wake up and end the internal threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Attachment: 16845.v2.txt > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16845) Run tests under hbase-spark module in test phase
[ https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16845: --- Attachment: (was: 16845.v2.txt) > Run tests under hbase-spark module in test phase > > > Key: HBASE-16845 > URL: https://issues.apache.org/jira/browse/HBASE-16845 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16845.v1.txt, 16845.v2.txt > > > [~appy] reported over in the discussion thread titled 'Testing and CI' that > tests under hbase-spark module are not run by QA bot. > This issue is to run the unit tests in test phase so that we detect build > breakage early. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation
[
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576287#comment-15576287
]
huaxiang sun commented on HBASE-16578:
--
Hi [~jingcheng...@intel.com], thanks for the reply. I did not do enough
thinking yesterday.
The case I described is invalid as you mentioned that the compacted new
reference file will get a bigger seqId.
You patch looks good to me so + 1 from me.
Looking through the code, I found that it is possible for the following
sequence which could cause an issue.
1. put mob cell r1, flush, it will create ref1 and mobFile1.
2. put mob cell r2, flush, it will create ref2 and mobFile2.
3. put normal cell r3, do not flush.
4. mob compact, it will flush r3 to hfile1 and create a new reference file.
In this case, the maxSeqId in hfile1 is same as the seqId in the new
reference file, let's say it is 10
5. Since in step 4, flush happens before bulkload hfile. After flush,
compaction may kick in and compacts ref1, ref2, hfile1 into hfile2 (with
maxSeqId to be 10).
6. bulkloaded hfile finishes and it creates *_seqId_10_.
7. In this case, references in hfile2 and *_seqId_10 may mess up.
I think we need to change the following line:
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L432,
it needs to be applied to mob bulkloaded file as well to avoid the case.
> Mob data loss after mob compaction and normal compcation
>
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Jingcheng Du
> Attachments: HBASE-16578-V2.patch, HBASE-16578.patch,
> TestMobCompaction.java, TestMobCompaction.java
>
>
> StoreFileScanners on MOB cells rely on the scannerOrder to find the latest
> cells after mob compaction. The value of scannerOrder is assigned by the
> order of maxSeqId of StoreFile, and this maxSeqId is valued only after the
> reader of the StoreFile is created.
> In {{Compactor.compact}}, the compacted store files are cloned and their
> readers are not created. And in {{StoreFileScanner.getScannersForStoreFiles}}
> the StoreFiles are sorted before the readers are created and at that time the
> maxSeqId for each file is -1 (the default value). This will lead to a chaos
> in scanners in the following normal compaction. Some older cells might be
> chosen during the normal compaction.
> We need to create readers either before the sorting in the method
> {{StoreFileScanner.getScannersForStoreFiles}}, or create readers just after
> the store files are cloned in {{Compactor.compact}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16845) Run tests under hbase-spark module in test phase
[
https://issues.apache.org/jira/browse/HBASE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576303#comment-15576303
]
Hadoop QA commented on HBASE-16845:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 34s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833433/16845.v1.txt |
| JIRA Issue | HBASE-16845 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux db3eceb017a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 62c8411 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4024/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4024/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Run tests under hbase-spark module in test phase
>
>
> Key: HBASE-16845
> URL: https://issues.apache.org/jira/browse/HBASE-16845
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16845.v1.txt, 16845.v2.txt
>
>
>
