[jira] [Created] (HBASE-21549) Add shell command for serial replication peer
Guanghao Zhang created HBASE-21549: -- Summary: Add shell command for serial replication peer Key: HBASE-21549 URL: https://issues.apache.org/jira/browse/HBASE-21549 Project: HBase Issue Type: Improvement Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] JDK8 for >=1.5.x?
That sounds like a fine plan for Phoenix to do. Please no changing minimum jdk requirements for HBase in minor releases. It's bad enough that we had to give up on Hadoop compatibility for minor releases. I'm sure it's only a matter of time before that puts us in a bad spot wrt JDK support. On Tue, Dec 4, 2018, 19:05 Thomas D'Silva I think Phoenix might end up moving the connector modules (spark, kakfa > etc) and the query server into a separate repos > so that they can be compiled with Java 8. Phoenix core would continue to > maintain the same java version compatibility as HBase. > > > On Tue, Dec 4, 2018 at 4:37 PM Andrew Purtell wrote: > > > I'm not taking a position, just pointing out that dropping Phoenix > > coprocessors compiled with Java 8 into a Java 7 runtime won't work. It is > > unclear at least to me if mixing HBase compiled with Java 7 and > > coprocessors compiled with Java 8 in a Java 8 runtime would be ok. > > > > Based on that I would say Phoenix, at least the coprocessor code, is > > constrained by the compatibility choices that HBase makes. > > > > That said I don't know of an obstacle to a build process on the Phoenix > > side that uses Java 7 to build coprocessors and Java 8 to build the PQS. > > Maybe someone on the Phoenix project has tried it? Or can try it? > > > > On Tue, Dec 4, 2018 at 4:33 PM Zach York > > wrote: > > > > > Isn't Phoenix's compatibility guarantees separate from HBase's? If > > Phoenix > > > only works with a Java8 environment, then couldn't Phoenix only support > > > Java8 environments in releases after the Java 8 Avatica issue without > > > requiring HBase's compatibility to drop support for Java7? > > > > > > While it's nice to have Phoenix and HBase compatibility guarantees > remain > > > in sync, I don't think that's a requirement for either project. > > > I do feel like dropping support for a java version in a minor release > is > > > very questionable as a Java upgrade isn't always trivial. > Unfortunately, > > > this is a by product of our long lived release lines. > > > > > > On Tue, Dec 4, 2018 at 4:21 PM Andrew Purtell > > wrote: > > > > > > > We have the current compatibility promise that we will build with > Java > > 7 > > > so > > > > folks using a Java 7 JRE can consume and deploy them. > > > > > > > > If we will be assuming Java 8 runtimes predominate, then we can build > > > with > > > > Java 8. > > > > > > > > > > > > On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey > wrote: > > > > > > > > > What about installing Phoenix coprocessors built with Java 8 into a > > > Java > > > > 8 > > > > > runtime that's running the currently built for jdk7 HBase? > Shouldn't > > > that > > > > > work fine? > > > > > > > > > > On Tue, Dec 4, 2018, 16:16 Andrew Purtell > wrote: > > > > > > > > > > > I thought only the Avatica module aka PQS must be compiled using > > Java > > > > 8? > > > > > > Maybe you can script a Phoenix build that uses the Java 7 > compiler > > > for > > > > > the > > > > > > coprocessor modules and a Java 8 compiler only for the PQS? > > > > > > > > > > > > Otherwise, if all Phoenix modules including the coprocessor code > > are > > > > > > compiled with Java 8, then HBase must also be compiled with Java > 8. > > > If > > > > > you > > > > > > attempt to install Phoenix coprocessors compiled with Java 8 > into a > > > > Java > > > > > > 7 + HBase runtime then you will see fatal runtime errors related > to > > > > > linkage > > > > > > errors in the concurrent data types. > > > > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser > > wrote: > > > > > > > > > > > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > > > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser > > > > > wrote: > > > > > > > >> > > > > > > > >> Phoenix depends on Avatica for its Phoenix Query Server. > > Avatica > > > > > > > >> requires Java8. Thus, the impasse. > > > > > > > >> > > > > > > > > > > > > > > > > Generally our maintaining JDK7 support shouldn't limit the > > > ability > > > > of > > > > > > > > any downstream user to require a newer JDK. Is there some > > context > > > > I'm > > > > > > > > missing? Are they manually recompiling some part of our > > codebase > > > > that > > > > > > > > doesn't work with JDK8? Or do they not want to have folks > > > > complaining > > > > > > > > to them for requiring a JDK update that we don't require? > > > > > > > > > > > > > > Phoenix isn't doing anything exceptional. > > > > > > > > > > > > > > I'm struggling to find the Phoenix thread where Andrew P > > suggested > > > > that > > > > > > > I start this discussion in HBase. I don't remember if Phoenix > is > > > > using > > > > > > > JDK7 purely to build solely because HBase does, or if there is > a > > > > > > > technical reason that this is required. I'll have to keep > > looking. > > > > > > > > > > > > > > >> I'd pose the question: are we really doing our users a > service > > > for > > > > > > > >> continuing to allow them to use Java7? Not tryi
Re: [DISCUSS] JDK8 for >=1.5.x?
I think Phoenix might end up moving the connector modules (spark, kakfa etc) and the query server into a separate repos so that they can be compiled with Java 8. Phoenix core would continue to maintain the same java version compatibility as HBase. On Tue, Dec 4, 2018 at 4:37 PM Andrew Purtell wrote: > I'm not taking a position, just pointing out that dropping Phoenix > coprocessors compiled with Java 8 into a Java 7 runtime won't work. It is > unclear at least to me if mixing HBase compiled with Java 7 and > coprocessors compiled with Java 8 in a Java 8 runtime would be ok. > > Based on that I would say Phoenix, at least the coprocessor code, is > constrained by the compatibility choices that HBase makes. > > That said I don't know of an obstacle to a build process on the Phoenix > side that uses Java 7 to build coprocessors and Java 8 to build the PQS. > Maybe someone on the Phoenix project has tried it? Or can try it? > > On Tue, Dec 4, 2018 at 4:33 PM Zach York > wrote: > > > Isn't Phoenix's compatibility guarantees separate from HBase's? If > Phoenix > > only works with a Java8 environment, then couldn't Phoenix only support > > Java8 environments in releases after the Java 8 Avatica issue without > > requiring HBase's compatibility to drop support for Java7? > > > > While it's nice to have Phoenix and HBase compatibility guarantees remain > > in sync, I don't think that's a requirement for either project. > > I do feel like dropping support for a java version in a minor release is > > very questionable as a Java upgrade isn't always trivial. Unfortunately, > > this is a by product of our long lived release lines. > > > > On Tue, Dec 4, 2018 at 4:21 PM Andrew Purtell > wrote: > > > > > We have the current compatibility promise that we will build with Java > 7 > > so > > > folks using a Java 7 JRE can consume and deploy them. > > > > > > If we will be assuming Java 8 runtimes predominate, then we can build > > with > > > Java 8. > > > > > > > > > On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey wrote: > > > > > > > What about installing Phoenix coprocessors built with Java 8 into a > > Java > > > 8 > > > > runtime that's running the currently built for jdk7 HBase? Shouldn't > > that > > > > work fine? > > > > > > > > On Tue, Dec 4, 2018, 16:16 Andrew Purtell wrote: > > > > > > > > > I thought only the Avatica module aka PQS must be compiled using > Java > > > 8? > > > > > Maybe you can script a Phoenix build that uses the Java 7 compiler > > for > > > > the > > > > > coprocessor modules and a Java 8 compiler only for the PQS? > > > > > > > > > > Otherwise, if all Phoenix modules including the coprocessor code > are > > > > > compiled with Java 8, then HBase must also be compiled with Java 8. > > If > > > > you > > > > > attempt to install Phoenix coprocessors compiled with Java 8 into a > > > Java > > > > > 7 + HBase runtime then you will see fatal runtime errors related to > > > > linkage > > > > > errors in the concurrent data types. > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser > wrote: > > > > > > > > > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser > > > wrote: > > > > > > >> > > > > > > >> Phoenix depends on Avatica for its Phoenix Query Server. > Avatica > > > > > > >> requires Java8. Thus, the impasse. > > > > > > >> > > > > > > > > > > > > > > Generally our maintaining JDK7 support shouldn't limit the > > ability > > > of > > > > > > > any downstream user to require a newer JDK. Is there some > context > > > I'm > > > > > > > missing? Are they manually recompiling some part of our > codebase > > > that > > > > > > > doesn't work with JDK8? Or do they not want to have folks > > > complaining > > > > > > > to them for requiring a JDK update that we don't require? > > > > > > > > > > > > Phoenix isn't doing anything exceptional. > > > > > > > > > > > > I'm struggling to find the Phoenix thread where Andrew P > suggested > > > that > > > > > > I start this discussion in HBase. I don't remember if Phoenix is > > > using > > > > > > JDK7 purely to build solely because HBase does, or if there is a > > > > > > technical reason that this is required. I'll have to keep > looking. > > > > > > > > > > > > >> I'd pose the question: are we really doing our users a service > > for > > > > > > >> continuing to allow them to use Java7? Not trying to be > > > contentious > > > > in > > > > > > >> asking that -- I am just (happily) seeing fewer and fewer > folks > > > > still > > > > > on > > > > > > >> Java7. > > > > > > > > > > > > > > Yes. Making it so downstream folks don't have to worry about > > > certain > > > > > > > categories of changes is the entire advantage of us declaring > > > > backward > > > > > > > compatibility promises at all. JDK changes are a big deal in > any > > > > > > > deployment. > > > > > > > > > > > > > > As a project we already have a path forward for no longer > > worrying > > > > > > > about maintain
Re: [DISCUSS] JDK8 for >=1.5.x?
I'm not taking a position, just pointing out that dropping Phoenix coprocessors compiled with Java 8 into a Java 7 runtime won't work. It is unclear at least to me if mixing HBase compiled with Java 7 and coprocessors compiled with Java 8 in a Java 8 runtime would be ok. Based on that I would say Phoenix, at least the coprocessor code, is constrained by the compatibility choices that HBase makes. That said I don't know of an obstacle to a build process on the Phoenix side that uses Java 7 to build coprocessors and Java 8 to build the PQS. Maybe someone on the Phoenix project has tried it? Or can try it? On Tue, Dec 4, 2018 at 4:33 PM Zach York wrote: > Isn't Phoenix's compatibility guarantees separate from HBase's? If Phoenix > only works with a Java8 environment, then couldn't Phoenix only support > Java8 environments in releases after the Java 8 Avatica issue without > requiring HBase's compatibility to drop support for Java7? > > While it's nice to have Phoenix and HBase compatibility guarantees remain > in sync, I don't think that's a requirement for either project. > I do feel like dropping support for a java version in a minor release is > very questionable as a Java upgrade isn't always trivial. Unfortunately, > this is a by product of our long lived release lines. > > On Tue, Dec 4, 2018 at 4:21 PM Andrew Purtell wrote: > > > We have the current compatibility promise that we will build with Java 7 > so > > folks using a Java 7 JRE can consume and deploy them. > > > > If we will be assuming Java 8 runtimes predominate, then we can build > with > > Java 8. > > > > > > On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey wrote: > > > > > What about installing Phoenix coprocessors built with Java 8 into a > Java > > 8 > > > runtime that's running the currently built for jdk7 HBase? Shouldn't > that > > > work fine? > > > > > > On Tue, Dec 4, 2018, 16:16 Andrew Purtell > > > > > > I thought only the Avatica module aka PQS must be compiled using Java > > 8? > > > > Maybe you can script a Phoenix build that uses the Java 7 compiler > for > > > the > > > > coprocessor modules and a Java 8 compiler only for the PQS? > > > > > > > > Otherwise, if all Phoenix modules including the coprocessor code are > > > > compiled with Java 8, then HBase must also be compiled with Java 8. > If > > > you > > > > attempt to install Phoenix coprocessors compiled with Java 8 into a > > Java > > > > 7 + HBase runtime then you will see fatal runtime errors related to > > > linkage > > > > errors in the concurrent data types. > > > > > > > > > > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser wrote: > > > > > > > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser > > wrote: > > > > > >> > > > > > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica > > > > > >> requires Java8. Thus, the impasse. > > > > > >> > > > > > > > > > > > > Generally our maintaining JDK7 support shouldn't limit the > ability > > of > > > > > > any downstream user to require a newer JDK. Is there some context > > I'm > > > > > > missing? Are they manually recompiling some part of our codebase > > that > > > > > > doesn't work with JDK8? Or do they not want to have folks > > complaining > > > > > > to them for requiring a JDK update that we don't require? > > > > > > > > > > Phoenix isn't doing anything exceptional. > > > > > > > > > > I'm struggling to find the Phoenix thread where Andrew P suggested > > that > > > > > I start this discussion in HBase. I don't remember if Phoenix is > > using > > > > > JDK7 purely to build solely because HBase does, or if there is a > > > > > technical reason that this is required. I'll have to keep looking. > > > > > > > > > > >> I'd pose the question: are we really doing our users a service > for > > > > > >> continuing to allow them to use Java7? Not trying to be > > contentious > > > in > > > > > >> asking that -- I am just (happily) seeing fewer and fewer folks > > > still > > > > on > > > > > >> Java7. > > > > > > > > > > > > Yes. Making it so downstream folks don't have to worry about > > certain > > > > > > categories of changes is the entire advantage of us declaring > > > backward > > > > > > compatibility promises at all. JDK changes are a big deal in any > > > > > > deployment. > > > > > > > > > > > > As a project we already have a path forward for no longer > worrying > > > > > > about maintaining Java 7 support; it's Apache HBase 2+. Providing > > > > > > advantages for that move and reducing the risk of updating to it > > is a > > > > > > straight forward way to remove our limitation on JDK features. > > IMHO, > > > > > > making it riskier to upgrade to newer 1.y minor versions is an > > > > > > undesirable way to drive that change. > > > > > > > > > > I disagree with you on this point but that is fine. > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Andrew > > > > > > > > Words like orphans lost among the crosstalk, meaning torn
Re: [DISCUSS] JDK8 for >=1.5.x?
Isn't Phoenix's compatibility guarantees separate from HBase's? If Phoenix only works with a Java8 environment, then couldn't Phoenix only support Java8 environments in releases after the Java 8 Avatica issue without requiring HBase's compatibility to drop support for Java7? While it's nice to have Phoenix and HBase compatibility guarantees remain in sync, I don't think that's a requirement for either project. I do feel like dropping support for a java version in a minor release is very questionable as a Java upgrade isn't always trivial. Unfortunately, this is a by product of our long lived release lines. On Tue, Dec 4, 2018 at 4:21 PM Andrew Purtell wrote: > We have the current compatibility promise that we will build with Java 7 so > folks using a Java 7 JRE can consume and deploy them. > > If we will be assuming Java 8 runtimes predominate, then we can build with > Java 8. > > > On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey wrote: > > > What about installing Phoenix coprocessors built with Java 8 into a Java > 8 > > runtime that's running the currently built for jdk7 HBase? Shouldn't that > > work fine? > > > > On Tue, Dec 4, 2018, 16:16 Andrew Purtell > > > > I thought only the Avatica module aka PQS must be compiled using Java > 8? > > > Maybe you can script a Phoenix build that uses the Java 7 compiler for > > the > > > coprocessor modules and a Java 8 compiler only for the PQS? > > > > > > Otherwise, if all Phoenix modules including the coprocessor code are > > > compiled with Java 8, then HBase must also be compiled with Java 8. If > > you > > > attempt to install Phoenix coprocessors compiled with Java 8 into a > Java > > > 7 + HBase runtime then you will see fatal runtime errors related to > > linkage > > > errors in the concurrent data types. > > > > > > > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser wrote: > > > > > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser > wrote: > > > > >> > > > > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica > > > > >> requires Java8. Thus, the impasse. > > > > >> > > > > > > > > > > Generally our maintaining JDK7 support shouldn't limit the ability > of > > > > > any downstream user to require a newer JDK. Is there some context > I'm > > > > > missing? Are they manually recompiling some part of our codebase > that > > > > > doesn't work with JDK8? Or do they not want to have folks > complaining > > > > > to them for requiring a JDK update that we don't require? > > > > > > > > Phoenix isn't doing anything exceptional. > > > > > > > > I'm struggling to find the Phoenix thread where Andrew P suggested > that > > > > I start this discussion in HBase. I don't remember if Phoenix is > using > > > > JDK7 purely to build solely because HBase does, or if there is a > > > > technical reason that this is required. I'll have to keep looking. > > > > > > > > >> I'd pose the question: are we really doing our users a service for > > > > >> continuing to allow them to use Java7? Not trying to be > contentious > > in > > > > >> asking that -- I am just (happily) seeing fewer and fewer folks > > still > > > on > > > > >> Java7. > > > > > > > > > > Yes. Making it so downstream folks don't have to worry about > certain > > > > > categories of changes is the entire advantage of us declaring > > backward > > > > > compatibility promises at all. JDK changes are a big deal in any > > > > > deployment. > > > > > > > > > > As a project we already have a path forward for no longer worrying > > > > > about maintaining Java 7 support; it's Apache HBase 2+. Providing > > > > > advantages for that move and reducing the risk of updating to it > is a > > > > > straight forward way to remove our limitation on JDK features. > IMHO, > > > > > making it riskier to upgrade to newer 1.y minor versions is an > > > > > undesirable way to drive that change. > > > > > > > > I disagree with you on this point but that is fine. > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > >- A23, Crosstalk > > > > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands >- A23, Crosstalk >
Re: [DISCUSS] JDK8 for >=1.5.x?
We have the current compatibility promise that we will build with Java 7 so folks using a Java 7 JRE can consume and deploy them. If we will be assuming Java 8 runtimes predominate, then we can build with Java 8. On Tue, Dec 4, 2018 at 4:15 PM Sean Busbey wrote: > What about installing Phoenix coprocessors built with Java 8 into a Java 8 > runtime that's running the currently built for jdk7 HBase? Shouldn't that > work fine? > > On Tue, Dec 4, 2018, 16:16 Andrew Purtell > > I thought only the Avatica module aka PQS must be compiled using Java 8? > > Maybe you can script a Phoenix build that uses the Java 7 compiler for > the > > coprocessor modules and a Java 8 compiler only for the PQS? > > > > Otherwise, if all Phoenix modules including the coprocessor code are > > compiled with Java 8, then HBase must also be compiled with Java 8. If > you > > attempt to install Phoenix coprocessors compiled with Java 8 into a Java > > 7 + HBase runtime then you will see fatal runtime errors related to > linkage > > errors in the concurrent data types. > > > > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser wrote: > > > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser wrote: > > > >> > > > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica > > > >> requires Java8. Thus, the impasse. > > > >> > > > > > > > > Generally our maintaining JDK7 support shouldn't limit the ability of > > > > any downstream user to require a newer JDK. Is there some context I'm > > > > missing? Are they manually recompiling some part of our codebase that > > > > doesn't work with JDK8? Or do they not want to have folks complaining > > > > to them for requiring a JDK update that we don't require? > > > > > > Phoenix isn't doing anything exceptional. > > > > > > I'm struggling to find the Phoenix thread where Andrew P suggested that > > > I start this discussion in HBase. I don't remember if Phoenix is using > > > JDK7 purely to build solely because HBase does, or if there is a > > > technical reason that this is required. I'll have to keep looking. > > > > > > >> I'd pose the question: are we really doing our users a service for > > > >> continuing to allow them to use Java7? Not trying to be contentious > in > > > >> asking that -- I am just (happily) seeing fewer and fewer folks > still > > on > > > >> Java7. > > > > > > > > Yes. Making it so downstream folks don't have to worry about certain > > > > categories of changes is the entire advantage of us declaring > backward > > > > compatibility promises at all. JDK changes are a big deal in any > > > > deployment. > > > > > > > > As a project we already have a path forward for no longer worrying > > > > about maintaining Java 7 support; it's Apache HBase 2+. Providing > > > > advantages for that move and reducing the risk of updating to it is a > > > > straight forward way to remove our limitation on JDK features. IMHO, > > > > making it riskier to upgrade to newer 1.y minor versions is an > > > > undesirable way to drive that change. > > > > > > I disagree with you on this point but that is fine. > > > > > > > > > -- > > Best regards, > > Andrew > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > decrepit hands > >- A23, Crosstalk > > > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
Re: [DISCUSS] JDK8 for >=1.5.x?
What about installing Phoenix coprocessors built with Java 8 into a Java 8 runtime that's running the currently built for jdk7 HBase? Shouldn't that work fine? On Tue, Dec 4, 2018, 16:16 Andrew Purtell I thought only the Avatica module aka PQS must be compiled using Java 8? > Maybe you can script a Phoenix build that uses the Java 7 compiler for the > coprocessor modules and a Java 8 compiler only for the PQS? > > Otherwise, if all Phoenix modules including the coprocessor code are > compiled with Java 8, then HBase must also be compiled with Java 8. If you > attempt to install Phoenix coprocessors compiled with Java 8 into a Java > 7 + HBase runtime then you will see fatal runtime errors related to linkage > errors in the concurrent data types. > > > On Tue, Dec 4, 2018 at 1:54 PM Josh Elser wrote: > > > On 12/4/18 3:28 PM, Sean Busbey wrote: > > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser wrote: > > >> > > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica > > >> requires Java8. Thus, the impasse. > > >> > > > > > > Generally our maintaining JDK7 support shouldn't limit the ability of > > > any downstream user to require a newer JDK. Is there some context I'm > > > missing? Are they manually recompiling some part of our codebase that > > > doesn't work with JDK8? Or do they not want to have folks complaining > > > to them for requiring a JDK update that we don't require? > > > > Phoenix isn't doing anything exceptional. > > > > I'm struggling to find the Phoenix thread where Andrew P suggested that > > I start this discussion in HBase. I don't remember if Phoenix is using > > JDK7 purely to build solely because HBase does, or if there is a > > technical reason that this is required. I'll have to keep looking. > > > > >> I'd pose the question: are we really doing our users a service for > > >> continuing to allow them to use Java7? Not trying to be contentious in > > >> asking that -- I am just (happily) seeing fewer and fewer folks still > on > > >> Java7. > > > > > > Yes. Making it so downstream folks don't have to worry about certain > > > categories of changes is the entire advantage of us declaring backward > > > compatibility promises at all. JDK changes are a big deal in any > > > deployment. > > > > > > As a project we already have a path forward for no longer worrying > > > about maintaining Java 7 support; it's Apache HBase 2+. Providing > > > advantages for that move and reducing the risk of updating to it is a > > > straight forward way to remove our limitation on JDK features. IMHO, > > > making it riskier to upgrade to newer 1.y minor versions is an > > > undesirable way to drive that change. > > > > I disagree with you on this point but that is fine. > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands >- A23, Crosstalk >
Re: [DISCUSS] JDK8 for >=1.5.x?
I thought only the Avatica module aka PQS must be compiled using Java 8? Maybe you can script a Phoenix build that uses the Java 7 compiler for the coprocessor modules and a Java 8 compiler only for the PQS? Otherwise, if all Phoenix modules including the coprocessor code are compiled with Java 8, then HBase must also be compiled with Java 8. If you attempt to install Phoenix coprocessors compiled with Java 8 into a Java 7 + HBase runtime then you will see fatal runtime errors related to linkage errors in the concurrent data types. On Tue, Dec 4, 2018 at 1:54 PM Josh Elser wrote: > On 12/4/18 3:28 PM, Sean Busbey wrote: > > On Tue, Dec 4, 2018 at 2:14 PM Josh Elser wrote: > >> > >> Phoenix depends on Avatica for its Phoenix Query Server. Avatica > >> requires Java8. Thus, the impasse. > >> > > > > Generally our maintaining JDK7 support shouldn't limit the ability of > > any downstream user to require a newer JDK. Is there some context I'm > > missing? Are they manually recompiling some part of our codebase that > > doesn't work with JDK8? Or do they not want to have folks complaining > > to them for requiring a JDK update that we don't require? > > Phoenix isn't doing anything exceptional. > > I'm struggling to find the Phoenix thread where Andrew P suggested that > I start this discussion in HBase. I don't remember if Phoenix is using > JDK7 purely to build solely because HBase does, or if there is a > technical reason that this is required. I'll have to keep looking. > > >> I'd pose the question: are we really doing our users a service for > >> continuing to allow them to use Java7? Not trying to be contentious in > >> asking that -- I am just (happily) seeing fewer and fewer folks still on > >> Java7. > > > > Yes. Making it so downstream folks don't have to worry about certain > > categories of changes is the entire advantage of us declaring backward > > compatibility promises at all. JDK changes are a big deal in any > > deployment. > > > > As a project we already have a path forward for no longer worrying > > about maintaining Java 7 support; it's Apache HBase 2+. Providing > > advantages for that move and reducing the risk of updating to it is a > > straight forward way to remove our limitation on JDK features. IMHO, > > making it riskier to upgrade to newer 1.y minor versions is an > > undesirable way to drive that change. > > I disagree with you on this point but that is fine. > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
Re: [DISCUSS] JDK8 for >=1.5.x?
On 12/4/18 3:28 PM, Sean Busbey wrote: On Tue, Dec 4, 2018 at 2:14 PM Josh Elser wrote: Phoenix depends on Avatica for its Phoenix Query Server. Avatica requires Java8. Thus, the impasse. Generally our maintaining JDK7 support shouldn't limit the ability of any downstream user to require a newer JDK. Is there some context I'm missing? Are they manually recompiling some part of our codebase that doesn't work with JDK8? Or do they not want to have folks complaining to them for requiring a JDK update that we don't require? Phoenix isn't doing anything exceptional. I'm struggling to find the Phoenix thread where Andrew P suggested that I start this discussion in HBase. I don't remember if Phoenix is using JDK7 purely to build solely because HBase does, or if there is a technical reason that this is required. I'll have to keep looking. I'd pose the question: are we really doing our users a service for continuing to allow them to use Java7? Not trying to be contentious in asking that -- I am just (happily) seeing fewer and fewer folks still on Java7. Yes. Making it so downstream folks don't have to worry about certain categories of changes is the entire advantage of us declaring backward compatibility promises at all. JDK changes are a big deal in any deployment. As a project we already have a path forward for no longer worrying about maintaining Java 7 support; it's Apache HBase 2+. Providing advantages for that move and reducing the risk of updating to it is a straight forward way to remove our limitation on JDK features. IMHO, making it riskier to upgrade to newer 1.y minor versions is an undesirable way to drive that change. I disagree with you on this point but that is fine.
Re: [DISCUSS] JDK8 for >=1.5.x?
On Tue, Dec 4, 2018 at 2:14 PM Josh Elser wrote: > > Phoenix depends on Avatica for its Phoenix Query Server. Avatica > requires Java8. Thus, the impasse. > Generally our maintaining JDK7 support shouldn't limit the ability of any downstream user to require a newer JDK. Is there some context I'm missing? Are they manually recompiling some part of our codebase that doesn't work with JDK8? Or do they not want to have folks complaining to them for requiring a JDK update that we don't require? > I'd pose the question: are we really doing our users a service for > continuing to allow them to use Java7? Not trying to be contentious in > asking that -- I am just (happily) seeing fewer and fewer folks still on > Java7. Yes. Making it so downstream folks don't have to worry about certain categories of changes is the entire advantage of us declaring backward compatibility promises at all. JDK changes are a big deal in any deployment. As a project we already have a path forward for no longer worrying about maintaining Java 7 support; it's Apache HBase 2+. Providing advantages for that move and reducing the risk of updating to it is a straight forward way to remove our limitation on JDK features. IMHO, making it riskier to upgrade to newer 1.y minor versions is an undesirable way to drive that change.
Re: [DISCUSS] JDK8 for >=1.5.x?
Phoenix depends on Avatica for its Phoenix Query Server. Avatica requires Java8. Thus, the impasse. I'd pose the question: are we really doing our users a service for continuing to allow them to use Java7? Not trying to be contentious in asking that -- I am just (happily) seeing fewer and fewer folks still on Java7. On 11/30/18 4:55 PM, Sean Busbey wrote: I'm opposed if only because I don't like weakening our compatibility promises. What does Phoenix (or other downstream folks) lose out on because we work with jdk7 on our branch-1 releases? On Fri, Nov 30, 2018, 15:20 Josh Elser We came back across this in Phoenix recently and Andrew suggested we talk about it here in HBase. Can/should we move to requiring Java8 for HBase 1.5 and beyond? It would have my vote. - Josh
[jira] [Resolved] (HBASE-21531) race between region report and region move causes master to kill RS
[
https://issues.apache.org/jira/browse/HBASE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HBASE-21531.
--
Resolution: Duplicate
> race between region report and region move causes master to kill RS
> ---
>
> Key: HBASE-21531
> URL: https://issues.apache.org/jira/browse/HBASE-21531
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> In this case the delay between the ack from RS and the region report was
> 1.5s, so I'm not sure what caused the race (network hiccup? unreported retry
> by protobuf transport?) but in any case I don't see anything that prevents
> this from happening in a normal case, with a narrowed time window. Any delay
> (e.g. a GC pause on RS right after the report is built, and ack is sent for
> the close) or retries expands the window.
> Master starts moving the region and the source RS acks by 21:51,206
> {noformat}
> Master:
> 2018-11-21 21:21:49,024 INFO [master/6:17000.Chore.1] master.HMaster:
> balance hri=, source=,17020,1542754626176, destination= 2>,17020,1542863268158
> ...
> Server:
> 2018-11-21 21:21:49,394 INFO [RS_CLOSE_REGION-regionserver/:17020-1]
> handler.UnassignRegionHandler: Close
> ...
> 2018-11-21 21:21:51,095 INFO [RS_CLOSE_REGION-regionserver/:17020-1]
> handler.UnassignRegionHandler: Closed
> {noformat}
> By then the region is removed from onlineRegions, so the master proceeds.
> {noformat}
> 2018-11-21 21:21:51,206 INFO [PEWorker-4] procedure2.ProcedureExecutor:
> Finished subprocedure(s) of pid=667,
> state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true;
> TransitRegionStateProcedure table=, region=, REOPEN/MOVE;
> resume parent processing.
> 2018-11-21 21:21:51,386 INFO [PEWorker-13] assignment.RegionStateStore:
> pid=667 updating hbase:meta row=, regionState=OPENING,
> regionLocation=,17020,1542863268158
> {noformat}
> There are no obvious errors/delays that I see in RS log, and it doesn't log
> starting to send the report.
> However, at 21:52.853 the report is processed that still contains this region.
> {noformat}
> 2018-11-21 21:21:52,853 WARN
> [RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=17000]
> assignment.AssignmentManager: Killing ,17020,1542754626176:
> rit=OPENING, location=,17020,1542863268158, table=,
> region= reported OPEN on server=,17020,1542754626176 but
> state has otherwise.
> * ABORTING region server ,17020,1542754626176:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPENING, location= 2>,17020,1542863268158, table=, region= reported OPEN on
> server=,17020,1542754626176 but state has otherwise.
> {noformat}
> RS shuts down in an orderly manner and it can be seen from the log that this
> region is actually not present (there's no line indicating it's being closed,
> unlike for other regions).
> I think there needs to be some sort of versioning for region operations
> and/or in RS reports to allow master to account for concurrent operations and
> avoid races. Probably per region with either a grace period or an additional
> global version, so that master could avoid killing RS based on stale reports,
> but still kill RS if it did retain an old version of the region state due to
> some bug after acking a new version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: About to release 1.4.9
Needed another round with HBASE-21464. I plan to commit a fix for this today, to unblock the release, and roll a new RC. On Thu, Nov 29, 2018 at 5:55 PM Andrew Purtell wrote: > Now that HBASE-21464 is in it is time for 1.4.9 RC0. Preparing and > testing tomorrow. Should have a vote on it next week. > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands >- A23, Crosstalk > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
[jira] [Created] (HBASE-21548) CLONE - WAL writer for recovered.edits file in WalSplitting should not require hflush from filesystem
Josh Elser created HBASE-21548:
--
Summary: CLONE - WAL writer for recovered.edits file in
WalSplitting should not require hflush from filesystem
Key: HBASE-21548
URL: https://issues.apache.org/jira/browse/HBASE-21548
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.0.4
Been talking through this with a bunch of folks. [~enis] brought me back from
the cliff of despair though.
Context: running HBase on top of a filesystem that doesn't have hflush for
hfiles. In our case, on top of Azure's Hadoop-compatible filesystems (WASB,
ABFS).
When a RS fails and we have an SCP running for it, you'll see log splitting get
into an "infinite" loop where the master keeps resubmitting and the RS which
takes the action deterministically fails with the following:
{noformat}
2018-11-26 20:59:18,415 ERROR
[RS_LOG_REPLAY_OPS-regionserver/wn2-b831f9:16020-0-Writer-2]
wal.FSHLogProvider: The RegionServer write ahead log provider for FileSystem
implementations relies on the ability to call hflush for proper operation
during component failures, but the current FileSystem does not support doing
so. Please check the config value of 'hbase.wal.dir' and ensure it points to a
FileSystem mount that has suitable capabilities for output streams.
2018-11-26 20:59:18,415 WARN
[RS_LOG_REPLAY_OPS-regionserver/wn2-b831f9:16020-0-Writer-2]
wal.AbstractProtobufLogWriter: WALTrailer is null. Continuing with default.
2018-11-26 20:59:18,467 ERROR
[RS_LOG_REPLAY_OPS-regionserver/wn2-b831f9:16020-0-Writer-2] wal.WALSplitter:
Got while writing log entry to log
java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:96)
at
org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:61)
at
org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:370)
at
org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:804)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.createWAP(WALSplitter.java:1530)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.getWriterAndPath(WALSplitter.java:1501)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1584)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1566)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1090)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1082)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1052)
Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:99)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165)
at
org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:77)
... 10 more{noformat}
This is the sanity check added by HBASE-18784, failing on creating the writer
for the recovered.edits file.
The odd-ball here is that our recovered.edits writer is just a WAL writer
class. The WAL writer class thinks it always should have hflush support;
however, we don't _actually_ need that for writing out the recovered.edits
files. If {{close()}} on the recovered.edits file would fail, we're trash any
intermediate data in the filesystem and rerun the whole process.
It's my understanding that this check is over-bearing and we should not make
the check when the ProtobufLogWriter is being used for the recovered.edits file.
[~zyork], [~busbey] fyi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21547) Precommit uses master flaky list for other branches
Peter Somogyi created HBASE-21547:
-
Summary: Precommit uses master flaky list for other branches
Key: HBASE-21547
URL: https://issues.apache.org/jira/browse/HBASE-21547
Project: HBase
Issue Type: Bug
Components: test
Reporter: Peter Somogyi
Precommit job downloads the flaky exclude list for master branch when the
uploaded patch file is made for different branches.
As an example check [https://builds.apache.org/job/PreCommit-HBASE-Build/15192]
which was against branch-1 but the unit test downloaded master's flaky list.
{noformat}
15:26:05 [Tue Dec 4 14:26:04 UTC 2018 INFO]: Personality: patch unit
15:26:05 [Tue Dec 4 14:26:04 UTC 2018 INFO]:
EXCLUDE_TESTS_URL=https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/excludes/
15:26:05 [Tue Dec 4 14:26:04 UTC 2018 INFO]: INCLUDE_TESTS_URL=
15:26:05 --2018-12-04 14:26:04--
https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/excludes/
15:26:05 Resolving builds.apache.org (builds.apache.org)... 195.201.213.130,
2a01:4f8:c0:2cc9::2
15:26:05 Connecting to builds.apache.org
(builds.apache.org)|195.201.213.130|:443... connected.
15:26:06 HTTP request sent, awaiting response... 200
15:26:06 Length: 866 [application/octet-stream]
15:26:06 Saving to: 'excludes'
15:26:06
15:26:06 0K 100%
43.0M=0s
15:26:06
15:26:06 2018-12-04 14:26:06 (43.0 MB/s) - 'excludes' saved [866/866]
15:26:06
15:26:09 cd /testptch/hbase/hbase-thrift
15:26:09 mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/yetus-m2/hbase-branch-1-patch-1
-DHBasePatchProcess -Dhttps.protocols=TLSv1.2 -PrunAllTests
-Dtest.exclude.pattern=**/master.cleaner.TestSnapshotFromMaster.java,**/client.TestRestoreSnapshotFromClientAfterSplittingRegions.java,**/regionserver.TestRegionMergeTransactionOnCluster.java,**/client.TestCloneSnapshotFromClientAfterSplittingRegion.java,**/master.assignment.TestAssignmentManager.java,**/master.assignment.TestAMAssignWithRandExec.java,**/client.TestMobCloneSnapshotFromClientAfterSplittingRegion.java,**/regionserver.TestCompactingToCellFlatMapMemStore.java,**/replication.TestReplicationSmallTestsSync.java,**/TestMultiVersions.java,**/client.TestMobRestoreSnapshotFromClientAfterSplittingRegions.java,**/client.TestRestoreSnapshotFromClientWithRegionReplicas.java,**/regionserver.TestRegionServerAbortTimeout.java,**/replication.TestMasterReplication.java,**/backup.TestIncrementalBackupWithBulkLoad.java,**/master.replication.TestRegisterPeerWorkerWhenRestarting.java
clean test -fae > /testptch/patchprocess/patch-unit-hbase-thrift.txt 2>&1
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21546) ConnectException in TestThriftHttpServer
Peter Somogyi created HBASE-21546: - Summary: ConnectException in TestThriftHttpServer Key: HBASE-21546 URL: https://issues.apache.org/jira/browse/HBASE-21546 Project: HBase Issue Type: Bug Components: test, Thrift Affects Versions: 1.4.9, 1.5.0 Reporter: Peter Somogyi Assignee: Peter Somogyi TestThriftHttpServer is the first on the flaky list for branch-1 and branch-1.4 with approximately 60% failure rate. Thrift server is not yet accepting request at the time the test starts. java.net.ConnectException: Connection refused (Connection refused) at org.apache.hadoop.hbase.thrift.TestThriftHttpServer.checkHttpMethods(TestThriftHttpServer.java:275) at org.apache.hadoop.hbase.thrift.TestThriftHttpServer.testThriftServerHttpOptionsForbiddenWhenOptionsDisabled(TestThriftHttpServer.java:176) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
