[jira] [Updated] (PHOENIX-5951) IndexTool output logging for past-max-lookback rows should be configurable

2020-06-17 Thread Geoffrey Jacoby (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geoffrey Jacoby updated PHOENIX-5951:
-
Attachment: PHOENIX-5951-4.x.v2.patch

> IndexTool output logging for past-max-lookback rows should be configurable
> --
>
> Key: PHOENIX-5951
> URL: https://issues.apache.org/jira/browse/PHOENIX-5951
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Attachments: PHOENIX-5951-4.x.v1.patch, PHOENIX-5951-4.x.v2.patch
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> On a global mutable table with lots of writes, it can be possible for there 
> to be many rows flagged by index verification as invalid that are older than 
> the max lookback age. These are likely due to race conditions between 
> compaction in the base table and index table, but they can possibly show real 
> issues.
> Since heavy output logging to PHOENIX_INDEX_TOOL can cause hotspotting, some 
> operators may want to turn it off for this class of failures, while still 
> wanting output logging for other kinds of invalid rows. 
> As part of this change, we should start including in the output row a column 
> with a code identifying the type of error.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: No builds on Phoenix-Master jenkins since Feb 19th?

2020-06-17 Thread rajeshb...@apache.org 
Just my observation related tests hang or flaky ness is that when there are
connections created which involves zookeeper connection then at some point
of time zookeeper sees too many connections and not allow to create further
connections then the cluster won't be terminated immediately and tests
hangs. We can identify such cases and close the connections properly if any.

Thanks,
Rajeshbabu.

On Tue, Jun 16, 2020, 12:12 PM Istvan Toth  wrote:

> What we really need is getting our test suite (and infra) sorted out.
> Your suggestions are already part of the process, it's just that lately
> everyone ignores them, because the precommit tests have been next to
> useless for a good while.
>
> I think that if developers can be reasonably sure that the precommit
> failure that they see is not some kind of random/known flake, they (we)
> will take them seriously, and not commit until they(we) get it right.
>
> Blocking further commits until the tests are fixed is one drastic, but
> possibly necessary measure to achieve this.
> And once we get them fixed, we will indeed need to be more vigilant in not
> letting the situation deteriorate.
>
> Istvan
>
> On Mon, Jun 15, 2020 at 7:30 PM Andrew Purtell 
> wrote:
>
> > So it's time to enforce an automatic revert if build fails policy?
> > Ideally, no commit before a precommit passes.
> > If that fails, if someone discovers failing tests and can git bisect to
> the
> > cause, automatic revert of the offending commit.
> >
> > YDYT?
> >
> >
> > On Sun, Jun 14, 2020 at 10:20 PM Istvan Toth  >
> > wrote:
> >
> > > Unfortunately you are not missing anything, Lars :(
> > >
> > > What's worse, the tests are not only failing, they even hang since Jun
> 2.
> > > Master is in a similar state.
> > >
> > > regards
> > > Istvan
> > >
> > > On Sun, Jun 14, 2020 at 12:03 AM la...@apache.org 
> > > wrote:
> > >
> > > > Thanks Istvan.
> > > >
> > > > Just checked the builds. Looks like the 4.x builds have not passed a
> > > > single time since May 20th.
> > > >
> > > > I hope I am missing something... Otherwise this would be pretty
> > > > frustrating. :)
> > > > (Since pleading doesn't appear to help, maybe we should automatically
> > > > block all commits until all tests pass...?)
> > > >
> > > > -- Lars
> > > >
> > > >
> > > > On Tuesday, April 21, 2020, 5:33:52 AM PDT, Istvan Toth <
> > > st...@apache.org>
> > > > wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > I've deleted all but the four Phoenix jobs listed in the JIRA.
> > > >
> > > >
> > >
> > > --
> > > *István Tóth* | Staff Software Engineer
> > > st...@cloudera.com 
> > > [image: Cloudera] 
> > > [image: Cloudera on Twitter]  [image:
> > > Cloudera on Facebook]  [image:
> > Cloudera
> > > on LinkedIn] 
> > > 
> > > --
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

[jira] [Updated] (PHOENIX-5905) Reset user to hbase by changing rpc context before getting user permissions on access controller service

2020-06-17 Thread Rajeshbabu Chintaguntla (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajeshbabu Chintaguntla updated PHOENIX-5905:
-
Attachment: PHOENIX-5905.4.x.patch

> Reset user to hbase by changing rpc context before getting user permissions 
> on access controller service 
> -
>
> Key: PHOENIX-5905
> URL: https://issues.apache.org/jira/browse/PHOENIX-5905
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5905.4.x.patch, 
> PHOENIX-5905.master.addendum2.patch, PHOENIX-5905.master.addendum3.patch, 
> PHOENIX-5905.patch, PHOENIX-5905_addendum.patch, PHOENIX-5905_addendum2.patch
>
>
> Currently we are calling getUserPermissions with hbase user directly on 
> access controller service which is not a rpc call. If we don't reset user 
> system user will be considered and might expect extra privileges  to return 
> the user  permissions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5789) try to standardize on a JSON library

2020-06-17 Thread Chinmay Kulkarni (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kulkarni updated PHOENIX-5789:
--
Attachment: Screen Shot 2020-06-17 at 11.45.44 AM.png

> try to standardize on a JSON library
> 
>
> Key: PHOENIX-5789
> URL: https://issues.apache.org/jira/browse/PHOENIX-5789
> Project: Phoenix
>  Issue Type: Improvement
>  Components: core
>Reporter: Istvan Toth
>Assignee: Richard Antal
>Priority: Minor
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5789.4.x.v1.patch, PHOENIX-5789.4.x.v3.patch, 
> PHOENIX-5789.4.x.v4.patch, PHOENIX-5789.4.x.v5.patch, 
> PHOENIX-5789.4.x.v6.patch, PHOENIX-5789.4.x.v7.patch, 
> PHOENIX-5789.master.addendum.patch, PHOENIX-5789.master.addendum.v2.patch, 
> PHOENIX-5789.master.addendum.v3.patch, PHOENIX-5789.master.v1.patch, 
> PHOENIX-5789.master.v2.patch, PHOENIX-5789.master.v3.patch, 
> PHOENIX-5789.master.v4.patch, Screen Shot 2020-06-17 at 11.45.44 AM.png
>
>
> Phoenix uses at least the following JSON libraries:
>  * gson
>  * jackson
>  * jettison
> Of these, only the jackson usage is performance critical, as it is used 
> during bulk loading. 
> Try to standardize on a single one to reduce dependency hell.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5905) Reset user to hbase by changing rpc context before getting user permissions on access controller service

2020-06-17 Thread Rajeshbabu Chintaguntla (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajeshbabu Chintaguntla updated PHOENIX-5905:
-
Attachment: PHOENIX-5905.master.addendum3.patch

> Reset user to hbase by changing rpc context before getting user permissions 
> on access controller service 
> -
>
> Key: PHOENIX-5905
> URL: https://issues.apache.org/jira/browse/PHOENIX-5905
> Project: Phoenix
>  Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5905.master.addendum2.patch, 
> PHOENIX-5905.master.addendum3.patch, PHOENIX-5905.patch, 
> PHOENIX-5905_addendum.patch, PHOENIX-5905_addendum2.patch
>
>
> Currently we are calling getUserPermissions with hbase user directly on 
> access controller service which is not a rpc call. If we don't reset user 
> system user will be considered and might expect extra privileges  to return 
> the user  permissions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re:Re: Master branch does not compile

2020-06-17 Thread cheng...@apache.org 



Should we only support HBase 2.2 , which is a stable 2.x release ?  














At 2020-06-17 16:04:58, "Istvan Toth"  wrote:
>The 4.x branch has historically deprecated unsupported branches.
>
>I'd be fine with removing support for EOL 2.x branches in master,
>https://issues.apache.org/jira/browse/PHOENIX-5716 is open for this very
>issue,
>but would welcome some community input before working on it.
>
>I also like Josh's idea, https://issues.apache.org/jira/browse/PHOENIX-5828 is
>quite similar to it.
>
>
>On Wed, Jun 17, 2020 at 2:49 AM Guanghao Zhang  wrote:
>
>> HBase 2.0 and 2.1 has been EOL now. Does phoenix need to support them?
>>
>> swaroopa kadam  于2020年6月17日周三 上午12:32写道：
>>
>> > yes, that’s a good idea.
>> >
>> > On Tue, Jun 16, 2020 at 9:29 AM Josh Elser  wrote:
>> >
>> > > Sounds like we should try to update precommit to at least compile
>> > > against _a version_ in each line 2.1/2.2/2.3 for master. Thoughts?
>> > >
>> > > On 6/16/20 11:57 AM, swaroopa kadam wrote:
>> > > > Thank you for the replies everyone!
>> > > >
>> > > > It puts me at ease by knowing the issue has been identified and being
>> > > > fixed.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > On Tue, Jun 16, 2020 at 4:11 AM rajeshb...@apache.org <
>> > > > chrajeshbab...@gmail.com> wrote:
>> > > >
>> > > >> I am on the PHOENIX-5905 compilation issue and will fix it today.
>> > > >>
>> > > >> On Tue, Jun 16, 2020 at 3:07 PM cheng...@apache.org <
>> > > cheng...@apache.org>
>> > > >> wrote:
>> > > >>
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> PHOENIX-5905 caused the master branch compile broken, because
>> > > >>> org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest
>> is
>> > > only
>> > > >>> available in hbase 2.2.x,
>> > > >>> so when the hbase.profile=2.0 or  hbase.profile=2.1, the compiler
>> > > boken,
>> > > >>> Should we revert PHOENIX-5905 for the moment?
>> > > >>>
>> > > >>>
>> > > >>> The error messages are :
>> > > >>>
>> > > >>>
>> > > >>> [ERROR] COMPILATION ERROR :
>> > > >>> [INFO]
>> -
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[39,47]
>> > > >>> cannot find symbol
>> > > >>>symbol:   class GetUserPermissionsRequest
>> > > >>>location: package org.apache.hadoop.hbase.security.access
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[1452,48]
>> > > >>> cannot find symbol
>> > > >>>symbol:   method hasUserName()
>> > > >>>location: variable request of type
>> > > >>>
>> > > >>
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[1452,72]
>> > > >>> cannot find symbol
>> > > >>>symbol:   method getUserName()
>> > > >>>location: variable request of type
>> > > >>>
>> > > >>
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[1458,32]
>> > > >>> cannot find symbol
>> > > >>>symbol:   method hasColumnFamily()
>> > > >>>location: variable request of type
>> > > >>>
>> > > >>
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[1458,60]
>> > > >>> cannot find symbol
>> > > >>>symbol:   method getColumnFamily()
>> > > >>>location: variable request of type
>> > > >>>
>> > > >>
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
>> > > >>> :[1460,32]
>> > > >>> cannot find symbol
>> > > >>>symbol:   method hasColumnQualifier()
>> > > >>>location: variable request of type
>> > > >>>
>> > > >>
>> > >
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
>> > > >>> [ERROR] <
>> > > >>>
>> > > >>
>> > >
>> >
>>

[jira] [Assigned] (PHOENIX-5758) Apply cleanups to 4.x branch

2020-06-17 Thread Istvan Toth (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth reassigned PHOENIX-5758:


Assignee: (was: Istvan Toth)

> Apply cleanups to 4.x branch
> 
>
> Key: PHOENIX-5758
> URL: https://issues.apache.org/jira/browse/PHOENIX-5758
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Istvan Toth
>Priority: Major
>
> There are number of  cleanups that are only present on master, mostly because 
> backporting them to all of the 4.x branches would be too cumbersome.
> Once we have the unified 4.x branch, we should apply these there too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5962) Stabilize builds

2020-06-17 Thread Richard Antal (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Antal updated PHOENIX-5962:
---
Attachment: PHOENIX-5962.master.v4.patch

> Stabilize builds
> 
>
> Key: PHOENIX-5962
> URL: https://issues.apache.org/jira/browse/PHOENIX-5962
> Project: Phoenix
>  Issue Type: Task
>Reporter: Richard Antal
>Assignee: Richard Antal
>Priority: Major
> Attachments: PHOENIX-5962.master.v1.patch, 
> PHOENIX-5962.master.v2.patch, PHOENIX-5962.master.v3.patch, 
> PHOENIX-5962.master.v4.patch
>
>
> Last 2 build was aborted because Build timed out (after 360 minutes).
>  Search the possible cause of the problem and try to fix it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5928) Index rebuilds without replaying data table mutations

2020-06-17 Thread Kadir OZDEMIR (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kadir OZDEMIR updated PHOENIX-5928:
---
Attachment: PHOENIX-5928.4.x.008.patch

> Index rebuilds without replaying data table mutations
> -
>
> Key: PHOENIX-5928
> URL: https://issues.apache.org/jira/browse/PHOENIX-5928
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 5.0.0, 4.14.3
>Reporter: Kadir OZDEMIR
>Assignee: Kadir OZDEMIR
>Priority: Major
> Attachments: PHOENIX-5928.4.x.001.patch, PHOENIX-5928.4.x.002.patch, 
> PHOENIX-5928.4.x.003.patch, PHOENIX-5928.4.x.004.patch, 
> PHOENIX-5928.4.x.005.patch, PHOENIX-5928.4.x.006.patch, 
> PHOENIX-5928.4.x.007.patch, PHOENIX-5928.4.x.008.patch
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Index rebuilds are done currently by reading data table mutations (in 
> UngroupedAggregateRegionObserver) and then replaying them on the data table 
> (in IndexRegionObser) -- without actually applying mutations on the data 
> table -- in order to generate the corresponding index mutations and apply 
> them on index tables. IndexRegionObserver sets the operation status for these 
> data table mutations to NOWRITE so that they are ignored by HBase after 
> generating index table mutations from these data table mutations. Since we do 
> not apply these mutations on the data table, there is no need to send these 
> mutations to the data table regions if index mutations are generated and sent 
> to index tables by UngroupedAggregateRegionObserver. By doing so, we 
> eliminate going through data table update path and its overhead (its 
> interaction with flushes, row locking etc). 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (PHOENIX-5965) Remove Guava from Phoenix-Connectors

2020-06-17 Thread Beata Sudi (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beata Sudi reassigned PHOENIX-5965:
---

Assignee: Beata Sudi

> Remove Guava from Phoenix-Connectors
> 
>
> Key: PHOENIX-5965
> URL: https://issues.apache.org/jira/browse/PHOENIX-5965
> Project: Phoenix
>  Issue Type: Bug
>  Components: connectors
>Reporter: Beata Sudi
>Assignee: Beata Sudi
>Priority: Major
>
> Guava should be removed as a dependency, as it's kind of difficult to 
> maintain and update. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (PHOENIX-5965) Remove Guava from Phoenix-Connectors

2020-06-17 Thread Beata Sudi (Jira) 
Beata Sudi created PHOENIX-5965:
---

 Summary: Remove Guava from Phoenix-Connectors
 Key: PHOENIX-5965
 URL: https://issues.apache.org/jira/browse/PHOENIX-5965
 Project: Phoenix
  Issue Type: Bug
  Components: connectors
Reporter: Beata Sudi


Guava should be removed as a dependency, as it's kind of difficult to maintain 
and update. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Public Python PhoenixDB releases

2020-06-17 Thread Istvan Toth 
I agree with everything that you said.

However by 'source release', I was merely referring to a Python/PyPI
packaging technicality, where we distribute the package in the form that is
generated by *python setup.py sdist*,
as opposed to the pre-built binary egg or wheel formats.

But since you brought this up, do we need to create actual zip/tarballs of
the source during release, or is creating a git tag enough ?
And if we do, do we package the full queryserver repo, or just the python
subdirectory ?
The docs in the package assume that the source is built from the full repo.


On Tue, Jun 16, 2020 at 6:24 PM Josh Elser  wrote:

> Yup, let's just draw a number (whatever we'd like, as long as it follows
> the standard PEP for versioning). When we release it, we can advertise
> whatever level of stability we expect it to have (i.e. "This is an
> alpha, you can try it if you want").
>
> ASF is very strict when it comes to releases in source form. In the
> legalese for the foundation, _only_ source releases are actually
> considered a release. That means, all of the foundation documentation
> about release policies applies to just the source release, further,
> that's really all we have to vote on.
>
> My current interpretation of the policy is that we should also try to
> validate any binaries we build to make sure that they still follow
> proper licensing guidelines for the foundation (but this is a "should"
> and not a "must" because they are not technically real releases).
>
> All that to say, when we do a vote in the ASF, it must be on source. If
> we have an egg to also use for testing, great. But, the pushing an
> approved ASF release to PyPi is a post-vote step.
>
> Shout if this doesn't make sense. It reads like you're on the right path :)
>
> On 6/16/20 3:06 AM, Istvan Toth wrote:
> > The Beam guide looks interesting, we could certainly borrow from it.
> >
> > I'm fine with your suggested process. Here's my interpretation of it:
> >
> > - No dev releases on PyPI
> > - Decoupling PhoenixDB versioning from PQS versioning
> > - Making a proper minor/patch release when important features/fixes
> land
> > - Testing the release process /making RC releases on TestPyPI
> >
> > One more detail that we haven't discussed yet:
> > The previous PyPI releases were source packages.
> > I suggest that we can keep releasing PhoenixDB as a source package, as it
> > is pure python.
> >
> > Istvan
> >
> > On Tue, Jun 16, 2020 at 3:32 AM Josh Elser  wrote:
> >
> >> ASF releases do not have to be on the order of years. As long as we have
> >> three people to vote, we can do releases as often as we'd like. The
> >> lower-bound on release cadence is probably on the order of "weeks" (as
> >> opposed to days) but that's primarily limited by bandwidth of our
> >> volunteers :)
> >>
> >> Anyways, if that's the major worry, let's just push to doing proper
> >> votes. We should not have the problem in having people turn out to vote.
> >>
> >> I'll put L review to the top of my list to do (my) tomorrow morning.
> >> Sorry I haven't gotten to it yet.
> >>
> >> On 6/15/20 1:42 PM, Istvan Toth wrote:
> >>> TestPyPi certainly seems useful to practice the release process, and
> >> avoid
> >>> mistakes with it.
> >>>
> >>> However, I don't think that it is a substitute for a .devN release,
> which
> >>> (in my mind) is meant to give the users something to use until we get
> all
> >>> our ducks in a row to do a proper 1.0 release. The primary user in this
> >>> case would be Hue, which needs the new features, and is currently
> >> mirroring
> >>> our dev sources into their own repo. (TBH, that's their normal modus
> >>> operandi anyway, but for their Python 3 version, they'd like to switch
> to
> >>> PyPI  modules)
> >>>
> >>> To use a dev package, the users have to explicitly specify the exact
> >>> development version, (or go all-out, and use development versions of
> all
> >>> packages), so there is no chance of them accidentally upgrading from a
> >>> stable release. Making them specify another repo as well to install a
> dev
> >>> release sounds like too much to me.
> >>>
> >>> If on the other hand we plan on releasing  the current state of
> PhoenixDB
> >>> as 1.0 soon (like in month), and then do frequent-ish releases as new
> >>> features/fixes are (hopefully) added, then we can skip the dev releases
> >>> altogether. The Phoenix norm so far is more like yearly releases (if we
> >> are
> >>> lucky).
> >>>
> >>> regards
> >>> István
> >>>
> >>> On Mon, Jun 15, 2020 at 7:08 PM Josh Elser  wrote:
> >>>
>  Did a little searching..
> 
>  * Found a 2011 blog post from sqlalchemy which said (as a project)
> they
>  would not post devN releases to pypi
>  * There's a TestPyPi [1] instead which seems to be for staging work.
> 
>  Could we play with staging there? And then push to pypi (real) after
> we
>  do a normal vote?
> 
>  I think we can keep our python release super-low

Re: Master branch does not compile

2020-06-17 Thread Istvan Toth 
The 4.x branch has historically deprecated unsupported branches.

I'd be fine with removing support for EOL 2.x branches in master,
https://issues.apache.org/jira/browse/PHOENIX-5716 is open for this very
issue,
but would welcome some community input before working on it.

I also like Josh's idea, https://issues.apache.org/jira/browse/PHOENIX-5828 is
quite similar to it.


On Wed, Jun 17, 2020 at 2:49 AM Guanghao Zhang  wrote:

> HBase 2.0 and 2.1 has been EOL now. Does phoenix need to support them?
>
> swaroopa kadam  于2020年6月17日周三 上午12:32写道：
>
> > yes, that’s a good idea.
> >
> > On Tue, Jun 16, 2020 at 9:29 AM Josh Elser  wrote:
> >
> > > Sounds like we should try to update precommit to at least compile
> > > against _a version_ in each line 2.1/2.2/2.3 for master. Thoughts?
> > >
> > > On 6/16/20 11:57 AM, swaroopa kadam wrote:
> > > > Thank you for the replies everyone!
> > > >
> > > > It puts me at ease by knowing the issue has been identified and being
> > > > fixed.
> > > >
> > > > Thanks!
> > > >
> > > > On Tue, Jun 16, 2020 at 4:11 AM rajeshb...@apache.org <
> > > > chrajeshbab...@gmail.com> wrote:
> > > >
> > > >> I am on the PHOENIX-5905 compilation issue and will fix it today.
> > > >>
> > > >> On Tue, Jun 16, 2020 at 3:07 PM cheng...@apache.org <
> > > cheng...@apache.org>
> > > >> wrote:
> > > >>
> > > >>>
> > > >>>
> > > >>>
> > > >>> PHOENIX-5905 caused the master branch compile broken, because
> > > >>> org.apache.hadoop.hbase.security.access.GetUserPermissionsRequest
> is
> > > only
> > > >>> available in hbase 2.2.x,
> > > >>> so when the hbase.profile=2.0 or  hbase.profile=2.1, the compiler
> > > boken,
> > > >>> Should we revert PHOENIX-5905 for the moment?
> > > >>>
> > > >>>
> > > >>> The error messages are :
> > > >>>
> > > >>>
> > > >>> [ERROR] COMPILATION ERROR :
> > > >>> [INFO]
> -
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[39,47]
> > > >>> cannot find symbol
> > > >>>symbol:   class GetUserPermissionsRequest
> > > >>>location: package org.apache.hadoop.hbase.security.access
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1452,48]
> > > >>> cannot find symbol
> > > >>>symbol:   method hasUserName()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1452,72]
> > > >>> cannot find symbol
> > > >>>symbol:   method getUserName()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1458,32]
> > > >>> cannot find symbol
> > > >>>symbol:   method hasColumnFamily()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1458,60]
> > > >>> cannot find symbol
> > > >>>symbol:   method getColumnFamily()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1460,32]
> > > >>> cannot find symbol
> > > >>>symbol:   method hasColumnQualifier()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos.GetUserPermissionsRequest
> > > >>> [ERROR] <
> > > >>>
> > > >>
> > >
> >
> https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/ws/phoenix-core/src/it/java/org/apache/phoenix/end2end/BasePermissionsIT.java
> > > >>> :[1460,63]
> > > >>> cannot find symbol
> > > >>>symbol:   method getColumnQualifier()
> > > >>>location: variable request of type
> > > >>>
> > > >>
> > >
> >
>

[jira] [Created] (PHOENIX-5964) Rename queryserver subprojects

2020-06-17 Thread Istvan Toth (Jira) 
Istvan Toth created PHOENIX-5964:


 Summary: Rename queryserver subprojects
 Key: PHOENIX-5964
 URL: https://issues.apache.org/jira/browse/PHOENIX-5964
 Project: Phoenix
  Issue Type: Improvement
  Components: queryserver
Affects Versions: queryserver-1.0.0
Reporter: Istvan Toth
Assignee: Istvan Toth


The current phoenix-queryserver maven subproject/artifact names are quite 
arbitrary and inconsistent.

The current structure:
phoenix-queryserver
assembly
load-balancer
queryserver
queryserver-client
queryerrver-it
queryserver-orchestrator

I propose that we change this to either:

phoenix-queryserver
queryserver-assembly
queryserver-load-balancer
queryserver
queryserver-client
queryerrver-it
queryserver-orchestrator

or

phoenix-queryserver-parent
phoenix-queryserver-assembly
phoenix-queryserver-load-balancer
phoenix-queryserver
phoenix-queryserver-client
phoenix-queryerrver-it
phoenix-queryserver-orchestrator

The latter is more consistent with hadoop conventions, and the pre-split 
artifact names, but is also quite long.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5963) Enable loadbalancer for standalone PQS assembly

2020-06-17 Thread Istvan Toth (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-5963:
-
Description: 
The load-balancer does not work with the current PQS assembly, due to 
dependency (and script problems)

The proposal below would - hopefully - make it feasible to re-introduce and 
support it as an optional dependecy for PQS:

 

As for the load-balancer, you are right that this setup doesn't allow for a 
working load-balancer. However, it wouldn't work even if the load-balancer jar 
was included in the classpath. The load-balancer depends on (unshaded) 
zookeeper, and unshaded zookeeper is generally not available on the PQS 
classpath anyway (it may have been available in some old phoenix-client 
version, but currently both master and 4.x shades it).

Loading the load-balancer as an external service is nice in theory, but since 
it's compiled separately from phoenix-client, the dependency conflicts that it 
introduces are hard to solve.

If I were to do it, I'd:
 * Refactor/Simplify loadbalancer, so that doesn't need Hadoop and Hbase deps
 ** Pass the Hbase Config object as in Iterable to registerServer
 ** Ditch the config singleton/cache, it is not useful in this context
 ** Preferably ditch Curator as well
 * create a shaded loadbalancer jar with ZK (and curator) shaded into it
 * Then prepare the PQS startup script for it, and document how to add this to 
PQS as an optional dependency

  was:
The load-balancer does not work with the current PQS assembly, due to 
dependency (and script problems)

The proposal below would -hopefully- make it feasible to re-introduce and 
support it as an optional dependecy for PQS:

 

As for the load-balancer, you are right that this setup doesn't allow for a 
working load-balancer. However, it wouldn't work even if the load-balancer jar 
was included in the classpath. The load-balancer depends on (unshaded) 
zookeeper, and unshaded zookeeper is generally not available on the PQS 
classpath anyway (it may have been available in some old phoenix-client 
version, but currently both master and 4.x shades it).

Loading the load-balancer as an external service is nice in theory, but since 
it's compiled separately from phoenix-client, the dependency conflicts that it 
introduces are hard to solve.

If I were to do it, I'd:
 * Refactor/Simplify loadbalancer, so that doesn't need Hadoop and Hbase deps
 ** Pass the Hbase Config object as in Iterable to registerServer
 ** Ditch the config singleton/cache, it is not useful in this context
 ** Preferably ditch Curator as well
 * create a shaded loadbalancer jar with ZK (and curator) shaded into it
 * Then prepare the PQS startup script for it, and document how to add this to 
PQS as an optional dependency


> Enable loadbalancer for standalone PQS assembly
> ---
>
> Key: PHOENIX-5963
> URL: https://issues.apache.org/jira/browse/PHOENIX-5963
> Project: Phoenix
>  Issue Type: Bug
>  Components: queryserver
>Affects Versions: queryserver-1.0.0
>Reporter: Istvan Toth
>Priority: Major
>
> The load-balancer does not work with the current PQS assembly, due to 
> dependency (and script problems)
> The proposal below would - hopefully - make it feasible to re-introduce and 
> support it as an optional dependecy for PQS:
>  
> As for the load-balancer, you are right that this setup doesn't allow for a 
> working load-balancer. However, it wouldn't work even if the load-balancer 
> jar was included in the classpath. The load-balancer depends on (unshaded) 
> zookeeper, and unshaded zookeeper is generally not available on the PQS 
> classpath anyway (it may have been available in some old phoenix-client 
> version, but currently both master and 4.x shades it).
> Loading the load-balancer as an external service is nice in theory, but since 
> it's compiled separately from phoenix-client, the dependency conflicts that 
> it introduces are hard to solve.
> If I were to do it, I'd:
>  * Refactor/Simplify loadbalancer, so that doesn't need Hadoop and Hbase deps
>  ** Pass the Hbase Config object as in Iterable to registerServer
>  ** Ditch the config singleton/cache, it is not useful in this context
>  ** Preferably ditch Curator as well
>  * create a shaded loadbalancer jar with ZK (and curator) shaded into it
>  * Then prepare the PQS startup script for it, and document how to add this 
> to PQS as an optional dependency



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5962) Stabilize builds

2020-06-17 Thread Richard Antal (Jira) 


 [ 
https://issues.apache.org/jira/browse/PHOENIX-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Antal updated PHOENIX-5962:
---
Attachment: PHOENIX-5962.master.v3.patch

> Stabilize builds
> 
>
> Key: PHOENIX-5962
> URL: https://issues.apache.org/jira/browse/PHOENIX-5962
> Project: Phoenix
>  Issue Type: Task
>Reporter: Richard Antal
>Assignee: Richard Antal
>Priority: Major
> Attachments: PHOENIX-5962.master.v1.patch, 
> PHOENIX-5962.master.v2.patch, PHOENIX-5962.master.v3.patch
>
>
> Last 2 build was aborted because Build timed out (after 360 minutes).
>  Search the possible cause of the problem and try to fix it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (PHOENIX-5963) Enable loadbalancer for standalone PQS assembly

2020-06-17 Thread Istvan Toth (Jira) 
Istvan Toth created PHOENIX-5963:


 Summary: Enable loadbalancer for standalone PQS assembly
 Key: PHOENIX-5963
 URL: https://issues.apache.org/jira/browse/PHOENIX-5963
 Project: Phoenix
  Issue Type: Bug
  Components: queryserver
Affects Versions: queryserver-1.0.0
Reporter: Istvan Toth


The load-balancer does not work with the current PQS assembly, due to 
dependency (and script problems)

The proposal below would -hopefully- make it feasible to re-introduce and 
support it as an optional dependecy for PQS:

 

As for the load-balancer, you are right that this setup doesn't allow for a 
working load-balancer. However, it wouldn't work even if the load-balancer jar 
was included in the classpath. The load-balancer depends on (unshaded) 
zookeeper, and unshaded zookeeper is generally not available on the PQS 
classpath anyway (it may have been available in some old phoenix-client 
version, but currently both master and 4.x shades it).

Loading the load-balancer as an external service is nice in theory, but since 
it's compiled separately from phoenix-client, the dependency conflicts that it 
introduces are hard to solve.

If I were to do it, I'd:
 * Refactor/Simplify loadbalancer, so that doesn't need Hadoop and Hbase deps
 ** Pass the Hbase Config object as in Iterable to registerServer
 ** Ditch the config singleton/cache, it is not useful in this context
 ** Preferably ditch Curator as well
 * create a shaded loadbalancer jar with ZK (and curator) shaded into it
 * Then prepare the PQS startup script for it, and document how to add this to 
PQS as an optional dependency



--
This message was sent by Atlassian Jira
(v8.3.4#803005)