[jira] [Created] (HBASE-26752) Fix flappy test TestSimpleRegionNormalizerOnCluster.java
Aman Poonia created HBASE-26752: --- Summary: Fix flappy test TestSimpleRegionNormalizerOnCluster.java Key: HBASE-26752 URL: https://issues.apache.org/jira/browse/HBASE-26752 Project: HBase Issue Type: Bug Components: Normalizer Affects Versions: 1.7.1 Reporter: Aman Poonia Assignee: Aman Poonia TestSimpleRegionNormalizerOnCluster.java can hang after HBASE-26744 The assumption that order of HTable list is sorted is wrong so depending on that order can cause the test to hang or be inaccurate -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: New dedicated Jenkins Conrtoller
Most of the work is done. There is still an arm jenkins job on ci-hadoop, as we do not have arm nodes on ci-hbase. Another problem is that it seems the github web hook will not notify ci-hbase, so we need to wait for the pre-commit job to scan the github repo for triggering a build. Will keep an eye on it and work together with infra to solve it if possible. Thanks. 张铎(Duo Zhang) 于2022年2月10日周四 11:16写道: > Some updates. > > For nightly build we are still suffering from the python problem... > Anyway, using python2 directly can solve the problem. > > I'm still working on it. > > Thanks. > > 张铎(Duo Zhang) 于2022年2月9日周三 09:34写道: > >> The new Jenkins jobs are still unavailable due to lack of python support. >> >> If this still can not be resolved tonight, I will rollback to ci-hadoop >> and try migration next time. >> >> Thanks. >> >> 张铎(Duo Zhang) 于2022年2月8日周二 21:35写道: >> >>> Hi Gavin, the ssh transfer plugin is fine, but now we miss python on >>> hbase11-20. >>> >>> I've filed INFRA-22855, PTAL. >>> >>> Thanks. >>> >>> 张铎(Duo Zhang) 于2022年2月8日周二 20:19写道: >>> Thank you Gavin. WIll take a look soon! Gavin McDonald 于2022年2月8日周二 18:31写道: > Give it a try now please > > On Sat, Feb 5, 2022 at 8:55 AM 张铎(Duo Zhang) > wrote: > > > Hi, Gavin, any luck on the ssh transfer plugin? > > > > Thanks. > > > > 张铎(Duo Zhang) 于2022年1月29日周六 14:53写道: > > > >> So any updates here Gavin? > >> > >> Is the ssh transfer plugin available now? > >> > >> Thanks. > >> > >> Sean Busbey 于2022年1月23日周日 09:04写道: > >> > >>> It looks like I currently do have access to manage the ci-hbase > >>> controller. > >>> > >>> > >>> Could you check that the git-websites label handling where a common > >>> machine gets leased to the ci-hbase controller is set up properly? > or give > >>> me a pointer on how I would check this myself? > >>> > >>> I started trying to migrate our website generation job again and > it's > >>> been waiting for a node with the git-websites label to show up for > ~8 hours > >>> at this point. > >>> > >>> https://ci-hbase.apache.org/job/hbase_generate_website/3/ > >>> > >>> > >>> > >>> On Sat, Jan 22, 2022 at 2:42 AM Gavin McDonald < > gmcdon...@apache.org> > >>> wrote: > >>> > Hi, > Timestamper plugin installed. > > By the way I thought I had given the PMC members permissions to > manage > Jenkins > on ci-hbase including installing plugins etc, can someone check? > > > On Sat, Jan 22, 2022 at 7:34 AM 张铎(Duo Zhang) < > palomino...@gmail.com> > wrote: > > > We faced a strange issue.. > > > > rg.codehaus.groovy.control.MultipleCompilationErrorsException: > startup failed: > > WorkflowScript: 29: Invalid option type "timestamps". Valid > option > types: [buildDiscarder, catchError, checkoutToSubdirectory, > disableConcurrentBuilds, disableResume, durabilityHint, > githubProjectProperty, newContainerPerStage, > overrideIndexTriggers, > parallelsAlwaysFailFast, preserveStashes, quietPeriod, > rateLimitBuilds, > retry, script, skipDefaultCheckout, skipStagesAfterUnstable, > timeout, > waitUntil, warnError, withChecks, withContext, withCredentials, > withEnv, > wrap, ws] @ line 29, column 5. > >timestamps() > >^ > > > > 1 error > > > > at > > org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) > > at > > org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) > > at > > org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) > > at > > org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) > > at > > org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) > > at > > groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) > > at > > groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) > > at > groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) > > at groovy.lang.GroovyShell.parse(GroovyShell.java:700) > > at > > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) > > at > > org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) > > at > >
[jira] [Resolved] (HBASE-26749) Migrate HBase main pre commit job to ci-hbase
[ https://issues.apache.org/jira/browse/HBASE-26749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26749. --- Fix Version/s: 2.5.0 1.7.2 3.0.0-alpha-3 2.4.10 Hadoop Flags: Reviewed Resolution: Fixed Merged to all ative branches. Thanks [~busbey] for reviewing! > Migrate HBase main pre commit job to ci-hbase > - > > Key: HBASE-26749 > URL: https://issues.apache.org/jira/browse/HBASE-26749 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 1.7.2, 3.0.0-alpha-3, 2.4.10 > > > We used to run this on the hadoop label so first we need to change the label > in jenkins file. > And before migration, we need to move the 10 hbase nodes from ci-hadoop to > ci-hbase as there are no jobs on them any more. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26697) Migrate HBase Nightly HBase-Flaky-Tests and HBase-Find-Flaky-Tests to ci-hbase
[ https://issues.apache.org/jira/browse/HBASE-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26697. --- Fix Version/s: 2.5.0 1.7.2 3.0.0-alpha-3 2.4.10 Hadoop Flags: Reviewed Resolution: Fixed The work is done. Thanks all for helping! > Migrate HBase Nightly HBase-Flaky-Tests and HBase-Find-Flaky-Tests to ci-hbase > -- > > Key: HBASE-26697 > URL: https://issues.apache.org/jira/browse/HBASE-26697 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 1.7.2, 3.0.0-alpha-3, 2.4.10 > > > They are tied together so let's do them at once. > We also need to modify the url for our pre commit job. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26644) Spurious compaction failures with file tracker
[ https://issues.apache.org/jira/browse/HBASE-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-26644. Resolution: Not A Problem Yep, all good. I believe you fixed this in HBASE-26675 > Spurious compaction failures with file tracker > -- > > Key: HBASE-26644 > URL: https://issues.apache.org/jira/browse/HBASE-26644 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > Noticed when running a basic {{{}hbase pe randomWrite{}}}, we'll see > compactions failing at various points. > One example: > {noformat} > 2022-01-03 17:41:18,319 ERROR > [regionserver/localhost:16020-shortCompactions-0] > regionserver.CompactSplit(670): Compaction failed > region=TestTable,0004054490,1641249249856.2dc7251c6eceb660b9c7bb0b587db913., > storeName=2dc7251c6eceb660b9c7bb0b587db913/info0, priority=6, > startTime=1641249666161 > java.io.IOException: Root-level entries already added in single-level mode > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeSingleLevelIndex(HFileBlockIndex.java:1136) > at > org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterWriter$MetaWriter.write(CompoundBloomFilterWriter.java:279) > at > org.apache.hadoop.hbase.io.hfile.HFileWriterImpl$1.writeToBlock(HFileWriterImpl.java:713) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeBlock(HFileBlock.java:1205) > at > org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:660) > at > org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:377) > at > org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.commitWriter(DefaultCompactor.java:70) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:386) > at > org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:62) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:125) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1141) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2388) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:654) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:697) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) {noformat} > This isn't a super-critical issue because compactions will be retried > automatically and they appear to eventually succeed. However, when the max > storefiles limit is reaching, this does cause ingest to hang (as I was doing > with my modest configuration). > We had seen a similar kind of problem in our testing when backporting to > HBase 2.4 (not upstream as the decision was to not do this) which we > eventually tracked down to a bad merge-conflict resolution to the new HFile > Cleaner. However, initial investigations don't have the same exact problem. > It seems that we have some kind of generic race condition. Would be good to > add more logging to catch this in the future (since we have two separate > instances of this category of bug already). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26750) The branch-1 nightly jenkins job fails with "No such DSL method 'jiraComment' found"
[ https://issues.apache.org/jira/browse/HBASE-26750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26750. --- Assignee: Duo Zhang Resolution: Fixed I installed jira plugin on ci-hbase and also configured the jira site. Now it works, see HBASE-26747. Resolved. > The branch-1 nightly jenkins job fails with "No such DSL method 'jiraComment' > found" > > > Key: HBASE-26750 > URL: https://issues.apache.org/jira/browse/HBASE-26750 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-24146) Run a perf comparison of 2.3.x with 2.2.x
[ https://issues.apache.org/jira/browse/HBASE-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24146. -- Resolution: Won't Do > Run a perf comparison of 2.3.x with 2.2.x > - > > Key: HBASE-24146 > URL: https://issues.apache.org/jira/browse/HBASE-24146 > Project: HBase > Issue Type: Task > Components: community, Performance >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Priority: Major > > Kick the tires on a recent build. Compare results with a run on the latest > 2.2.x. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26741) Incorrect exception handling in shell
[ https://issues.apache.org/jira/browse/HBASE-26741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26741. --- Fix Version/s: 2.5.0 3.0.0-alpha-3 2.4.10 Resolution: Fixed Merged to branch-2.4+. Thanks for the review [~elserj]. Filed HBASE-26751 to cover the shell exit behavior with tests. > Incorrect exception handling in shell > - > > Key: HBASE-26741 > URL: https://issues.apache.org/jira/browse/HBASE-26741 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.4.10 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10 > > > The exception handling changed in the shell compared to 2.2. > {noformat} > ➜ hbase-upstream git:(branch-2.4) ✗ cat commands.txt > scan 'foo' > exit > ➜ hbase-upstream git:(branch-2.4) ✗ bin/hbase shell -n commands.txt > 2022-02-07 16:21:33,654 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > hbase:001:0> scan 'foo' > ROW COLUMN+CELL > Took 0.3890 seconds > Traceback (most > recent call last): > RuntimeError (Unknown table foo!) > hbase:002:0> exit > ➜ hbase-upstream git:(branch-2.4) ✗ echo $? > 0 {noformat} > The execution continues even after an exception is thrown. In 2.2.7 the > execution stops when an exception is thrown and the shell exits with error > code. > {noformat} > ➜ hbase-2.2.7 bin/hbase shell -n commands.txt > 2022-02-07 16:33:54,930 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > ROW COLUMN+CELL > Took 0.3082 seconds > RuntimeError: > Unknown table foo! > translate_hbase_exceptions at > /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:130 > command_safe at > /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell/commands.rb:49 > internal_command at > /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:148 > command at > /Users/petersomogyi/tmp/hbase-2.2.7/lib/ruby/shell.rb:140 > scan at (eval):2 > at commands.txt:1 > load at org/jruby/RubyKernel.java:973 > at > /Users/petersomogyi/tmp/hbase-2.2.7/bin/../bin/hirb.rb:186 > ➜ hbase-2.2.7 echo $? > 1 {noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26751) Tests for shell exit behavior
Peter Somogyi created HBASE-26751: - Summary: Tests for shell exit behavior Key: HBASE-26751 URL: https://issues.apache.org/jira/browse/HBASE-26751 Project: HBase Issue Type: Test Components: shell, test Reporter: Peter Somogyi HBase shell exit behavior and exception handling is not covered with tests. The tests should cover the clean exits and also when a command throws exceptions in interactive and non-interactive modes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26744) Normalizer exits without normalizing all the tables
[ https://issues.apache.org/jira/browse/HBASE-26744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha resolved HBASE-26744. Fix Version/s: 1.8.0 Resolution: Fixed Merged to branch-1, thanks [~mnpoonia] for contributing. > Normalizer exits without normalizing all the tables > --- > > Key: HBASE-26744 > URL: https://issues.apache.org/jira/browse/HBASE-26744 > Project: HBase > Issue Type: Bug > Components: Normalizer >Affects Versions: 1.7.1 >Reporter: Aman Poonia >Assignee: Aman Poonia >Priority: Minor > Fix For: 1.8.0 > > > Currently if there are multiple tables to normalize, normalizer is exiting > even before iterating all the table if there is a table that doesnot require > normalization > here is the offending code > [https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1736] > > {code:java} > List plans = this.normalizer.computePlansForTable(table); > > if (plans == null || plans.isEmpty()) { >return true; > } {code} > this is running inside loop over tables > -- This message was sent by Atlassian Jira (v8.20.1#820001)