I've bisected twice and it lands on this commit:
commit 6bc46bb10920c1c335b784b01d2a326db1a3d587 (HEAD, refs/bisect/bad)
HBASE-21959 CompactionTool should close the store it uses for
compacting files, in order to properly archive compacted files.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
| 2 ++
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionTool.java
| 100
At first glance it's hard to see how this change is relevant, but it does
introduce a new unit test.
On Tue, Apr 16, 2019 at 7:48 PM Andrew Purtell <andrew.purt...@gmail.com>
wrote:
> I’ve been able to reproduce it sometimes too and am bisecting. It may be
> an interaction between test cases, not a failure per se, but does seem have
> a recent cause, as you pointed out. I’ll be looking at it.
>
> Thank you for your kind consideration and for revoking your veto.
>
> A coprocessor API fix was just committed to branch-1 so I want to roll a
> new RC soon to include it. There is also an issue open to improve the
> behavior of the UI when the profiler link is clicked but system support is
> not available.
>
> > On Apr 16, 2019, at 7:40 PM, Yu Li <car...@gmail.com> wrote:
> >
> > After more investigation, the ConnectionRefused exception could be
> > reproduced with "mvn -Dtest=<case_name> test" after a complete run of all
> > cases through "mvn -PrunAllTests clean test", but cannot by a clean
> > standalone run (with "mvn *clean* test"). So now I'm more convinced it's
> > some kind of environment chaos caused by parallel execution of test
> cases,
> > and not a blocker issue.
> >
> > @Andrew It seems to me that kerby jar is not included in our binary
> > package, so I'm not sure whether a new RC is required by HBASE-22219.
> > Anyway I'd like to revoke my -1 vote now. Thanks.
> >
> > Best Regards,
> > Yu
> >
> >
> >> On Tue, 16 Apr 2019 at 10:19, Yu Li <car...@gmail.com> wrote:
> >>
> >> Sorry for the late response due to job priority.
> >>
> >> This ConnectionRefused issue cannot be reproduced on my laptop (MacOS
> >> 10.14.4) but could on the linux env. And I've checked and confirmed it
> >> could pass with 1.4.7/1.4.9 source package but stably failed with 1.5.0,
> >> performing a git bisect now, will report back later.
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Sat, 13 Apr 2019 at 00:38, Andrew Purtell <andrew.purt...@gmail.com>
> >> wrote:
> >>
> >>> I also see the occasional ConnectionRefused errors. They don’t
> reproduce
> >>> if you run the test standalone. I also only see them on a Linux dev
> host.
> >>> That may be enough to find by bisect the commit that introduced this
> >>> behavior. Working on it. There is a JIRA filed for this one. Search for
> >>> “TestBlocksRead” and label “branch-1”.
> >>>
> >>> Thanks for the investigations.
> >>>
> >>>> On Apr 12, 2019, at 6:36 AM, Yu Li <car...@gmail.com> wrote:
> >>>>
> >>>> Quick updates:
> >>>>
> >>>> W/ patch of HBASE-22219 or say upgrading kerby version to 1.0.1, the
> >>>> failures listed above in the 1st part of hbase-server disappeared.
> >>>>
> >>>> However, in the 2nd part of hbase-server UT there're still many
> >>>> ConnectionRefused exceptions (17 errors in total) as shown below,
> which
> >>>> could be reproduced easily with -Dtest=xxx command on my environments,
> >>>> still checking the root cause.
> >>>>
> >>>> [INFO] Running org.apache.hadoop.hbase.regionserver.TestBlocksRead
> >>>> [ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time
> elapsed:
> >>>> 0.853 s <<< FAILURE! - in
> >>>> org.apache.hadoop.hbase.regionserver.TestBlocksRead
> >>>> [ERROR]
> >>>>
> >>>
> testBlocksStoredWhenCachingDisabled(org.apache.hadoop.hbase.regionserver.TestBlocksRead)
> >>>> Time elapsed: 0.17 s <<< ERROR!
> >>>> java.net.ConnectException: Call From
> >>>> z05f06378.sqa.zth.tbsite.net/11.163.183.195 to localhost:35669 failed
> >>> on
> >>>> connection exception: java.net.ConnectException: Connection refused;
> For
> >>>> more details see:
> >>>> http://wiki.apache.org/hadoop/ConnectionRefused
> >>>> at
> >>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112)
> >>>> at
> >>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389)
> >>>> Caused by: java.net.ConnectException: Connection refused
> >>>> at
> >>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112)
> >>>> at
> >>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389)
> >>>>
> >>>> Best Regards,
> >>>> Yu
> >>>>
> >>>>
> >>>>> On Fri, 12 Apr 2019 at 13:11, Yu Li <car...@gmail.com> wrote:
> >>>>>
> >>>>> I have no doubt that you've run the tests locally before announcing a
> >>>>> release as you're always a great RM boss. And this shows one value of
> >>>>> verifying release, that different voter has different environments.
> >>>>>
> >>>>> Now I think the failures may be kerberos related, since I possibly
> has
> >>>>> changed some system configuration when doing Flink testing on this
> env
> >>>>> weeks ago. Located one issue (HBASE-22219) which also observed in
> >>> 1.4.7,
> >>>>> will further investigate.
> >>>>>
> >>>>> Best Regards,
> >>>>> Yu
> >>>>>
> >>>>>
> >>>>> On Fri, 12 Apr 2019 at 12:38, Andrew Purtell <
> andrew.purt...@gmail.com
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>>> “However it's good to find the issue earlier if there
> >>>>>> really is any, before release announced.”
> >>>>>>
> >>>>>> I run the complete unit test suite before announcing a release
> >>> candidate.
> >>>>>> Just to be clear.
> >>>>>>
> >>>>>> Totally agree we should get these problems sorted before an actual
> >>>>>> release. My policy is to cancel a RC if anyone vetoes for this
> >>> reason...
> >>>>>> want as much coverage and varying environments as we can manage.
> >>>>>>
> >>>>>> Thank you for your help so far and I hope the failures you see
> result
> >>> in
> >>>>>> analysis and fixes that lead to better test stability.
> >>>>>>
> >>>>>>> On Apr 11, 2019, at 9:32 PM, Yu Li <car...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Confirmed in 1.4.7 source the listed out cases passed (all in the
> 1st
> >>>>>> part
> >>>>>>> of hbase-server so the result comes out quickly.)... Also confirmed
> >>> the
> >>>>>>> test ran order are the same...
> >>>>>>>
> >>>>>>> Will try 1.5.0 again to prevent the environment difference caused
> by
> >>>>>> time.
> >>>>>>> If 1.5.0 still fails, will start to do the git bisect to locate the
> >>>>>> first
> >>>>>>> bad commit.
> >>>>>>>
> >>>>>>> Was also expecting an easy pass and +1 as always to save time and
> >>>>>> efforts,
> >>>>>>> but obvious no luck. However it's good to find the issue earlier if
> >>>>>> there
> >>>>>>> really is any, before release announced.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Yu
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Fri, 12 Apr 2019 at 12:16, Yu Li <car...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Fine, let's focus on verifying whether it's a real problem rather
> >>> than
> >>>>>>>> arguing about wording, after all that's not my intention...
> >>>>>>>>
> >>>>>>>> As mentioned, I participated in the 1.4.7 release vote[1] and
> IIRC I
> >>>>>> was
> >>>>>>>> using the same env and all tests passed w/o issue, that's where my
> >>>>>> concern
> >>>>>>>> lies and the main reason I gave a -1 vote. I'm running against
> 1.4.7
> >>>>>> source
> >>>>>>>> on the same now and let's see the result.
> >>>>>>>>
> >>>>>>>> [1]
> https://www.mail-archive.com/dev@hbase.apache.org/msg51380.html
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Yu
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, 12 Apr 2019 at 12:05, Andrew Purtell <
> >>> andrew.purt...@gmail.com
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I believe the test execution order matters. We run some tests in
> >>>>>>>>> parallel. The ordering of tests is determined by readdir()
> results
> >>>>>> and this
> >>>>>>>>> differs from host to host and checkout to checkout. So when you
> >>> see a
> >>>>>>>>> repeatable group of failures, that’s great. And when someone else
> >>>>>> doesn’t
> >>>>>>>>> see those same tests fail, or they cannot be reproduced when
> >>> running
> >>>>>> by
> >>>>>>>>> themselves, the commonly accepted term of art for this is
> “flaky”.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Apr 11, 2019, at 8:52 PM, Yu Li <car...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Sorry but I'd call it "possible environment related problem" or
> >>> "some
> >>>>>>>>>> feature may not work well in specific environment", rather than
> a
> >>>>>> flaky.
> >>>>>>>>>>
> >>>>>>>>>> Will check against 1.4.7 released source package before opening
> >>> any
> >>>>>>>>> JIRA.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Yu
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 12 Apr 2019 at 11:37, Andrew Purtell <
> >>>>>> andrew.purt...@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> And if they pass in my environment , then what should we call
> it
> >>>>>> then.
> >>>>>>>>> I
> >>>>>>>>>>> have no doubt you are seeing failures. Therefore can you please
> >>> file
> >>>>>>>>> JIRAs
> >>>>>>>>>>> and attach information that can help identify a fix. Thanks.
> >>>>>>>>>>>
> >>>>>>>>>>>> On Apr 11, 2019, at 8:35 PM, Yu Li <car...@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I ran the test suite with the
> >>> -Dsurefire.rerunFailingTestsCount=2
> >>>>>>>>> option
> >>>>>>>>>>>> and on two different env separately, so it sums up to 6 times
> >>>>>> stable
> >>>>>>>>>>>> failure for each case, and from my perspective this is not
> >>> flaky.
> >>>>>>>>>>>>
> >>>>>>>>>>>> IIRC last time when verifying 1.4.7 on the same env no such
> >>> issue
> >>>>>>>>>>> observed,
> >>>>>>>>>>>> will double check.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Yu
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, 12 Apr 2019 at 00:07, Andrew Purtell <
> >>>>>>>>> andrew.purt...@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> There are two failure cases it looks like. And this looks
> like
> >>>>>>>>> flakes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The wrong FS assertions are not something I see when I run
> >>> these
> >>>>>>>>> tests
> >>>>>>>>>>>>> myself. I am not able to investigate something I can’t
> >>> reproduce.
> >>>>>>>>> What I
> >>>>>>>>>>>>> suggest is since you can reproduce do a git bisect to find
> the
> >>>>>> commit
> >>>>>>>>>>> that
> >>>>>>>>>>>>> introduced the problem. Then we can revert it. As an
> >>> alternative
> >>>>>> we
> >>>>>>>>> can
> >>>>>>>>>>>>> open a JIRA, report the problem, temporarily @ignore the
> test,
> >>> and
> >>>>>>>>>>>>> continue. This latter option only should be done if we are
> >>> fairly
> >>>>>>>>>>> confident
> >>>>>>>>>>>>> it is a test only problem.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The connect exceptions are interesting. I see these sometimes
> >>> when
> >>>>>>>>> the
> >>>>>>>>>>>>> suite is executed, not this particular case, but when the
> >>> failed
> >>>>>>>>> test is
> >>>>>>>>>>>>> executed by itself it always passes. It is possible some
> >>> change to
> >>>>>>>>>>> classes
> >>>>>>>>>>>>> related to the minicluster or startup or shutdown timing are
> >>> the
> >>>>>>>>> cause,
> >>>>>>>>>>> but
> >>>>>>>>>>>>> it is test time flaky behavior. I’m not happy about this but
> it
> >>>>>>>>> doesn’t
> >>>>>>>>>>>>> actually fail the release because the failure is never
> >>> repeatable
> >>>>>>>>> when
> >>>>>>>>>>> the
> >>>>>>>>>>>>> test is run standalone.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In general it would be great if some attention was paid to
> test
> >>>>>>>>>>>>> cleanliness on branch-1. As RM I’m not in a position to
> insist
> >>>>>> that
> >>>>>>>>>>>>> everything is perfect or there will never be another 1.x
> >>> release,
> >>>>>>>>>>> certainly
> >>>>>>>>>>>>> not from branch-1. So, tests which fail repeatedly block a
> >>> release
> >>>>>>>>> IMHO
> >>>>>>>>>>> but
> >>>>>>>>>>>>> flakes do not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Apr 10, 2019, at 11:20 PM, Yu Li <car...@gmail.com>
> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -1
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Observed many UT failures when checking the source package
> >>> (tried
> >>>>>>>>>>>>> multiple
> >>>>>>>>>>>>>> rounds on two different environments, MacOs and Linux, got
> the
> >>>>>> same
> >>>>>>>>>>>>>> result), including (but not limited to):
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> TestBulkload:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> shouldBulkLoadSingleFamilyHLog(org.apache.hadoop.hbase.regionserver.TestBulkLoad)
> >>>>>>>>>>>>>> Time elapsed: 0.083 s <<< ERROR!
> >>>>>>>>>>>>>> java.lang.IllegalArgumentException: Wrong FS:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> file:/var/folders/t6/vch4nh357f98y1wlq09lbm7h0000gn/T/junit1805329913454564189/junit8020757893576011944/data/default/shouldBulkLoadSingleFamilyHLog/8f4a6b584533de2fd1bf3c398dfaac29,
> >>>>>>>>>>>>>> expected: hdfs://localhost:55938
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamiliesAndSpecifiedTableName(TestBulkLoad.java:246)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamilies(TestBulkLoad.java:256)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBulkLoad.shouldBulkLoadSingleFamilyHLog(TestBulkLoad.java:150)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> TestStoreFile:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile)
> >>>>>>>>>>>>>> Time elapsed: 0.083 s <<< ERROR!
> >>>>>>>>>>>>>> java.net.ConnectException: Call From localhost/127.0.0.1 to
> >>>>>>>>>>>>> localhost:55938
> >>>>>>>>>>>>>> failed on connection exception: java.net.ConnectException:
> >>>>>>>>> Connection
> >>>>>>>>>>>>>> refused; For more details see:
> >>>>>>>>>>>>>> http://wiki.apache.org/hadoop/ConnectionRefused
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestStoreFile.writeStoreFile(TestStoreFile.java:1047)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestStoreFile.testCacheOnWriteEvictOnClose(TestStoreFile.java:908)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> TestHFile:
> >>>>>>>>>>>>>> testEmptyHFile(org.apache.hadoop.hbase.io.hfile.TestHFile)
> >>> Time
> >>>>>>>>>>> elapsed:
> >>>>>>>>>>>>>> 0.08 s <<< ERROR!
> >>>>>>>>>>>>>> java.net.ConnectException: Call From
> >>>>>>>>>>>>>> z05f06378.sqa.zth.tbsite.net/11.163.183.195 to
> >>> localhost:35529
> >>>>>>>>> failed
> >>>>>>>>>>> on
> >>>>>>>>>>>>>> connection exception: java.net.ConnectException: Connection
> >>>>>> refused;
> >>>>>>>>>>> For
> >>>>>>>>>>>>>> more details see:
> >>>>>> http://wiki.apache.org/hadoop/ConnectionRefused
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.io
> >>>>>>>>>>>>> .hfile.TestHFile.testEmptyHFile(TestHFile.java:90)
> >>>>>>>>>>>>>> Caused by: java.net.ConnectException: Connection refused
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.io
> >>>>>>>>>>>>> .hfile.TestHFile.testEmptyHFile(TestHFile.java:90)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> TestBlocksScanned:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned)
> >>>>>>>>>>>>>> Time elapsed: 0.069 s <<< ERROR!
> >>>>>>>>>>>>>> java.lang.IllegalArgumentException: Wrong FS:
> >>>>>>>>>>> hdfs://localhost:35529/tmp/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> hbase-jueding.ly/hbase/data/default/TestBlocksScannedWithEncoding/a4a416cc3060d9820a621c294af0aa08
> >>>>>>>>>>>>> ,
> >>>>>>>>>>>>>> expected: file:///
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksScanned._testBlocksScanned(TestBlocksScanned.java:90)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> org.apache.hadoop.hbase.regionserver.TestBlocksScanned.testBlocksScannedWithEncoding(TestBlocksScanned.java:86)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And please let me know if any known issue I'm not aware of.
> >>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Yu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, 8 Apr 2019 at 11:38, Yu Li <car...@gmail.com>
> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The performance report LGTM, thanks! (and sorry for the lag
> >>> due
> >>>>>> to
> >>>>>>>>>>>>>>> Qingming Festival Holiday here in China)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Still verifying the release, just some quick feedback:
> >>> observed
> >>>>>>>>> some
> >>>>>>>>>>>>>>> incompatible changes in compatibility report including
> >>>>>>>>>>>>>>> HBASE-21492/HBASE-21684 and worth a reminder in
> ReleaseNote.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Irrelative but noticeable: the 1.4.9 release note URL is
> >>>>>> invalid on
> >>>>>>>>>>>>>>> https://hbase.apache.org/downloads.html
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Yu
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, 5 Apr 2019 at 08:45, Andrew Purtell <
> >>>>>> apurt...@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The difference is basically noise per the usual YCSB
> >>>>>> evaluation.
> >>>>>>>>>>> Small
> >>>>>>>>>>>>>>>> differences in workloads D and F (slightly worse) and
> >>> workload
> >>>>>> E
> >>>>>>>>>>>>> (slightly
> >>>>>>>>>>>>>>>> better) that do not indicate serious regression.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Linux version 4.14.55-62.37.amzn1.x86_64
> >>>>>>>>>>>>>>>> c3.8xlarge x 5
> >>>>>>>>>>>>>>>> OpenJDK Runtime Environment (build
> 1.8.0_181-shenandoah-b13)
> >>>>>>>>>>>>>>>> -Xms20g -Xmx20g -XX:+UseG1GC -XX:+AlwaysPreTouch
> >>> -XX:+UseNUMA
> >>>>>>>>>>>>>>>> -XX:-UseBiasedLocking -XX:+ParallelRefProcEnabled
> >>>>>>>>>>>>>>>> Hadoop 2.9.2
> >>>>>>>>>>>>>>>> Init: Load 100 M rows and snapshot
> >>>>>>>>>>>>>>>> Run: Delete table, clone and redeploy from snapshot, run
> 10
> >>> M
> >>>>>>>>>>>>> operations
> >>>>>>>>>>>>>>>> Args: -threads 100 -target 50000
> >>>>>>>>>>>>>>>> Test table: {NAME => 'u', BLOOMFILTER => 'ROW', VERSIONS
> =>
> >>>>>> '1',
> >>>>>>>>>>>>> IN_MEMORY
> >>>>>>>>>>>>>>>> => 'false', KEEP_DELETED_CELLS => 'FALSE',
> >>> DATA_BLOCK_ENCODING
> >>>>>> =>
> >>>>>>>>>>>>>>>> 'ROW_INDEX_V1', TTL => 'FOREVER', COMPRESSION => 'SNAPPY',
> >>>>>>>>>>>>> MIN_VERSIONS =>
> >>>>>>>>>>>>>>>> '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536',
> >>>>>>>>> REPLICATION_SCOPE =>
> >>>>>>>>>>>>>>>> '0'}
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload A
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200592 200583
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49852 49855
> >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 544 559
> >>>>>>>>>>>>>>>> [READ], MinLatency(us) 267 292
> >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 165631 185087
> >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 738 742
> >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us), 1877 1961
> >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1370 1181
> >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 702 646
> >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 180735 177279
> >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 1943 1652
> >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 3257 3085
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload B
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200599 200581
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49850 49855
> >>>>>>>>>>>>>>>> [READ], AverageLatency(us), 454 471
> >>>>>>>>>>>>>>>> [READ], MinLatency(us) 203 213
> >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 183423 174207
> >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 563 599
> >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 1360 1172
> >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1064 1029
> >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 746 726
> >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 163455 101631
> >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 1327 1157
> >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 2241 1898
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload C
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200541 200538
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49865 49865
> >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 332 327
> >>>>>>>>>>>>>>>> [READ], MinLatency(us) 175 179
> >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 210559 170367
> >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 410 396
> >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 871 892
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload D
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200579 200562
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49855 49859
> >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 487 547
> >>>>>>>>>>>>>>>> [READ], MinLatency(us) 210 214
> >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 192255 177535
> >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 973 1529
> >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 1836 2683
> >>>>>>>>>>>>>>>> [INSERT], AverageLatency(us) 1239 1152
> >>>>>>>>>>>>>>>> [INSERT], MinLatency(us) 807 788
> >>>>>>>>>>>>>>>> [INSERT], MaxLatency(us) 184575 148735
> >>>>>>>>>>>>>>>> [INSERT], 95thPercentileLatency(us) 1496 1243
> >>>>>>>>>>>>>>>> [INSERT], 99thPercentileLatency(us) 2965 2495
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload E
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 10k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 100605 100568
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 9939 9943
> >>>>>>>>>>>>>>>> [SCAN], AverageLatency(us) 3548 2687
> >>>>>>>>>>>>>>>> [SCAN], MinLatency(us) 696 678
> >>>>>>>>>>>>>>>> [SCAN], MaxLatency(us) 1059839 238463
> >>>>>>>>>>>>>>>> [SCAN], 95thPercentileLatency(us) 8327 6791
> >>>>>>>>>>>>>>>> [SCAN], 99thPercentileLatency(us) 17647 14415
> >>>>>>>>>>>>>>>> [INSERT], AverageLatency(us) 2688 1555
> >>>>>>>>>>>>>>>> [INSERT], MinLatency(us) 887 815
> >>>>>>>>>>>>>>>> [INSERT], MaxLatency(us) 173311 154623
> >>>>>>>>>>>>>>>> [INSERT], 95thPercentileLatency(us) 4455 2571
> >>>>>>>>>>>>>>>> [INSERT], 99thPercentileLatency(us) 9303 5375
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> YCSB Workload F
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200562 204178
> >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49859 48976
> >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 856 1137
> >>>>>>>>>>>>>>>> [READ], MinLatency(us) 262 257
> >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 205567 222335
> >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 2365 3475
> >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 3099 4143
> >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], AverageLatency(us) 2559 2917
> >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], MinLatency(us) 1100 1034
> >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], MaxLatency(us) 208767 204799
> >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], 95thPercentileLatency(us) 5747 7627
> >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], 99thPercentileLatency(us) 7203 8919
> >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1700 1777
> >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 737 687
> >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 97983 94271
> >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 3377 4147
> >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 4147 4831
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Apr 4, 2019 at 1:14 AM Yu Li <car...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the efforts boss.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Since it's a new minor release, do we have performance
> >>>>>> comparison
> >>>>>>>>>>>>> report
> >>>>>>>>>>>>>>>>> with 1.4.9 as we did when releasing 1.4.0? If so, any
> >>>>>> reference?
> >>>>>>>>>>> Many
> >>>>>>>>>>>>>>>>> thanks!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Yu
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, 4 Apr 2019 at 07:44, Andrew Purtell <
> >>>>>> apurt...@apache.org
> >>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The fourth HBase 1.5.0 release candidate (RC3) is
> >>> available
> >>>>>> for
> >>>>>>>>>>>>>>>> download
> >>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>
> >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/
> >>>>>>>>> and
> >>>>>>>>>>>>>>>> Maven
> >>>>>>>>>>>>>>>>>> artifacts are available in the temporary repository
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> https://repository.apache.org/content/repositories/orgapachehbase-1292/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The git tag corresponding to the candidate is '1.5.0RC3’
> >>>>>>>>>>>>> (b0bc7225c5).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> A detailed source and binary compatibility report for
> this
> >>>>>>>>> release
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> available for your review at
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/compat-check-report.html
> >>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> A list of the 115 issues resolved in this release can be
> >>>>>> found
> >>>>>>>>> at
> >>>>>>>>>>>>>>>>>> https://s.apache.org/K4Wk . The 1.5.0 changelog is
> >>> derived
> >>>>>> from
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> changelog of the last branch-1.4 release, 1.4.9.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Please try out the candidate and vote +1/0/-1.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The vote will be open for at least 72 hours. Unless
> >>>>>> objection I
> >>>>>>>>>>> will
> >>>>>>>>>>>>>>>> try
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> close it Friday April 12, 2019 if we have sufficient
> >>> votes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Prior to making this announcement I made the following
> >>>>>> preflight
> >>>>>>>>>>>>>>>> checks:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> RAT check passes (7u80)
> >>>>>>>>>>>>>>>>>> Unit test suite passes (7u80, 8u181)*
> >>>>>>>>>>>>>>>>>> Opened the UI in a browser, poked around
> >>>>>>>>>>>>>>>>>> LTT load 100M rows with 100% verification and 20%
> updates
> >>>>>>>>> (8u181)
> >>>>>>>>>>>>>>>>>> ITBLL 1B rows with slowDeterministic monkey (8u181)
> >>>>>>>>>>>>>>>>>> ITBLL 1B rows with serverKilling monkey (8u181)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> There are known flaky tests. See HBASE-21904 and
> >>> HBASE-21905.
> >>>>>>>>> These
> >>>>>>>>>>>>>>>> flaky
> >>>>>>>>>>>>>>>>>> tests do not represent serious test failures that would
> >>>>>> prevent
> >>>>>>>>> a
> >>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>> Andrew
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Words like orphans lost among the crosstalk, meaning torn
> >>> from
> >>>>>>>>>>> truth's
> >>>>>>>>>>>>>>>> decrepit hands
> >>>>>>>>>>>>>>>> - A23, Crosstalk
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
