from:"Josh Elser"



 [ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-25449.

Hadoop Flags: Reviewed
Release Note: The presence of HDFS short-circuit read configuration 
properties in hbase-default.xml inadvertently causes short-circuit reads to not 
happen inside of RegionServers, despite short-circuit reads being enabled in 
hdfs-site.xml.
  Resolution: Fixed

Thanks for a great fix (and test), [~shenshengli]!

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml



 [ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-25449.

Hadoop Flags: Reviewed
Release Note: The presence of HDFS short-circuit read configuration 
properties in hbase-default.xml inadvertently causes short-circuit reads to not 
happen inside of RegionServers, despite short-circuit reads being enabled in 
hdfs-site.xml.
  Resolution: Fixed

Thanks for a great fix (and test), [~shenshengli]!

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml



 [ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25449:
---
Fix Version/s: 2.3.5
   2.4.1
   2.5.0
   2.2.7
   3.0.0-alpha-1

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.4.1, 2.3.5
>
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml



[ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261583#comment-17261583
 ] 

Josh Elser commented on HBASE-25449:


Tested this on branch-2 and master, locally, just to make sure the metrics from 
HBASE-8868 align. branch-2.4 works fine (with Hadoop 3.1.2), but master looks a 
little weird. I'm not entirely convinced it's a problem – it may just be us 
being smarter and I've not yet caught up. I was able to observe 
shortCircuitBytesRead in the logs with master, so I believe the config is 
correct in that respect (and something is just happening that obviates any read 
from HDFS).

Either way, pushing this now.

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml

2021-01-07 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260929#comment-17260929
 ] 

Josh Elser commented on HBASE-25449:


I'll plan to merge this to all active 2.x+ branches tomorrow. Shout if there 
are concerns.

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25449) 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml

2021-01-06 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259961#comment-17259961
 ] 

Josh Elser commented on HBASE-25449:


Our [~karthikshva...@gmail.com] found this one over the weekend, too. We've 
been scrambling to realize the impact of this. It's bad.

Worse, really, because on 2.1 based releases, we don't have HBASE-8868 to even 
tell us (via metrics) that SCRs aren't happening.

Discussions with Busbey so far seem to indicate that our best course of action 
is to remove these from hbase-default.xml and that will fix a bunch of other 
things. The workaround is to re-set all of these properties from hdfs-site.xml 
back into hbase-site.xml.

> 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
> -
>
> Key: HBASE-25449
> URL: https://issues.apache.org/jira/browse/HBASE-25449
> Project: HBase
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.0.1
>Reporter: shenshengli
>Assignee: shenshengli
>Priority: Major
>
> I think this parameter is not suitable for in hbase-default.xml, because in 
> this case, HDFS explicitly set to "dfs.client.read.shortcircuit=true", hbase 
> rely on HDFS configuration, the parameters in hbase service still is 
> false.Must be explicitly in hbase-site.xml is set to 
> "dfs.client.read.shortcircuit=true" to take effect.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Phoenix committer Richárd Antal

2021-01-06 Thread Josh Elser


Congrats, Richard!

On 1/4/21 3:31 PM, Ankit Singhal wrote:
On behalf of the Apache Phoenix PMC, I'm pleased to announce that 
Richárd Antal

has accepted the PMC's invitation to become a committer on Apache Phoenix.

We appreciate all of the great contributions Richárd has made to the
community thus far and we look forward to his continued involvement.

Congratulations and welcome, Richárd Antal!

[jira] [Commented] (HBASE-24813) ReplicationSource should clear buffer usage on ReplicationSourceManager upon termination

2021-01-04 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258356#comment-17258356
 ] 

Josh Elser commented on HBASE-24813:


This should be re-resolved since the bug was addressed in HBASE-25117, yes?

> ReplicationSource should clear buffer usage on ReplicationSourceManager upon 
> termination
> 
>
> Key: HBASE-24813
> URL: https://issues.apache.org/jira/browse/HBASE-24813
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.6
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0
>
> Attachments: TestReplicationSyncUpTool.log, 
> image-2020-10-09-10-50-00-372.png
>
>
> Following investigations on the issue described by [~elserj] on HBASE-24779, 
> we found out that once a peer is removed, thus killing peers related 
> *ReplicationSource* instance, it may leave 
> *ReplicationSourceManager.totalBufferUsed* inconsistent. This can happen if 
> *ReplicationSourceWALReader* had put some entries on its queue to be 
> processed by *ReplicationSourceShipper,* but the peer removal killed the 
> shipper before it could process the pending entries. When 
> *ReplicationSourceWALReader* thread add entries to the queue, it increments 
> *ReplicationSourceManager.totalBufferUsed* with the sum of the entries sizes. 
> When those entries are read by *ReplicationSourceShipper,* 
> *ReplicationSourceManager.totalBufferUsed* is then decreased. We should also 
> decrease *ReplicationSourceManager.totalBufferUsed* when *ReplicationSource* 
> is terminated, otherwise those unprocessed entries size would be consuming 
> *ReplicationSourceManager.totalBufferUsed __*indefinitely, unless the RS gets 
> restarted. This may be a problem for deployments with multiple peers, or if 
> new peers are added.**



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



[ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257140#comment-17257140
 ] 

Josh Elser commented on CALCITE-4152:
-

Linking my draft PR. Needs cleanup, testing, doc updates, and performance 
validation.

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>    Assignee: Josh Elser
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



[ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257137#comment-17257137
 ] 

Josh Elser commented on CALCITE-4152:
-

{code:java}
2020-12-31 23:21:35,831 [qtp2048434399-16] DEBUG - COMMIT for / on 
HttpChannelOverHttp@584ac69e{s=HttpChannelState@5cea67c6{s=HANDLING 
rs=COMPLETING os=COMMITTED is=READY awp=false se=false i=false 
al=0},r=2,c=false/false,a=HANDLING,uri=//localhost:51706/,age=283}
200 null HTTP/1.1
Date: Fri, 01 Jan 2021 04:21:35 GMT
WWW-Authenticate: Negotiate 
oYH1MIHyoAMKAQChCwYJKoZIhvcSAQICom4EbGBqBgkqhkiG9xIBAgICAG9bMFmgAwIBBaEDAgEPok0wS6ADAgERokQEQtpZnCRCej2MpfcD4oGTteO70BdUVSdd7Y4o/hqCP7ZB6YcXORaqxcEHjVjRLCZk1MLueoDiUO/YQh2CruAbVWMIBaNuBGxgagYJKoZIhvcSAQICAgBvWzBZoAMCAQWhAwIBD6JNMEugAwIBEaJEBELaWZwkQno9jKX3A+KBk7Xju9AXVFUnXe2OKP4agj+2QemHFzkWqsXBB41Y0SwmZNTC7nqA4lDv2EIdgq7gG1VjCAU=
Set-Cookie: JSESSIONID=node01mx0ketk9hfx2166mjptrygys60.node0; Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/octet-stream;charset=utf-8 {code}
With the new ConfigurableSpnegoAuthenticator/LoginService, Jetty will 
automatically send back a JSESSIONID cookie and use that, as long as the 
provided "duration" for cookie validity is not exceeded. Pretty slick.

We'll have to go through the other stuff that hadoop-auth does and make sure 
that we don't need anything else (like {{Secure}} or {{HttpOnly}} options on 
that cookie.).

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>Assignee: Josh Elser
>Priority: Major
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



[ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257124#comment-17257124
 ] 

Josh Elser commented on CALCITE-4152:
-

Hah, well, maybe back this whole train up. (I think Kevin suggested this 
elsewhere)

The ConfigurableSpnegoAuthenticator already does exactly what we want here :)

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>    Assignee: Josh Elser
>Priority: Major
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25279) Non-daemon thread in ZKWatcher



 [ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25279:
---
Fix Version/s: 2.4.1

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25279) Non-daemon thread in ZKWatcher



 [ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25279:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Fell off my radar, but trying to get this committed.

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher



[ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257120#comment-17257120
 ] 

Josh Elser commented on HBASE-25279:


[~vjasani], it's been a while, but I believe I was able to reproduce this, yes. 
I think the ZK threads are already daemonized, so, while I agree with you long 
term that we should have a good understanding of ZK cnxn management inside our 
daemons, I think this is a good "cover our butts".

[~bharathv], yeah, if we are ever shutting down without the ZK client being 
closed. I didn't exhaustively map that out.

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures

2020-12-30 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved CALCITE-4367.
-
Fix Version/s: avatica-1.18.0
   Resolution: Fixed

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Reporter: John Bodley
>    Assignee: Josh Elser
>Priority: Trivial
> Fix For: avatica-1.18.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



[ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254396#comment-17254396
 ] 

Josh Elser commented on CALCITE-4152:
-

I guess this approach is really just a JWT but not following that spec 
[https://jwt.io/introduction]

Maybe step 5 should be "make it a JWT"

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>Priority: Major
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



 [ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned CALCITE-4152:
---

Assignee: Josh Elser

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>    Assignee: Josh Elser
>Priority: Major
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4152) Avoid SPNEGO re-negotiation for each request



[ 
https://issues.apache.org/jira/browse/CALCITE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254375#comment-17254375
 ] 

Josh Elser commented on CALCITE-4152:
-

Looking at this for fun, the general wag at what Hadoop is doing is this...
 * After a successful SPNEGO auth'n, they send a SetCookie header back to the 
client
 * The cookie looks something like {{Set-Cookie: 
hadoop.auth="u=guest=guest/c6401.ambari.apache@example.com=kerberos=1487947765114=fNpq9FYy2DA19Rah7586rgsAieI=";
 Path=gateway/default; Domain=ambari.apache.org; Secure; HttpOnly}}
 * The token data is "username", (kerberos) "principal", authentication type, 
expiration time
 * This token data is signed with HmacSHA256 and that's included as 
"{{fNpq9FYy2DA19Rah7586rgsAieI="}}
 * The signature is used when the token is passed back to the server to 
validate that the token itself wasn't changed (e.g. user doesn't modify it and 
say they're someone else)

 * If the user doesn't provide the token (via the cookie), spnego authn happens 
normally. When spnego authn succeeds, it sets a new cookie
 * If the user provides the token (via the cookie) and the token is valid (the 
signature matches), then user is marked as "authenticated" (as the user who is 
specified in that auth token).

I think I can break this up into a couple of steps:
 # Show that we can bypass spnego successfully with a cookie that just has 
basic info. Will have to add indirection in AbstractAvaticaHandler to not pull 
the user directly from the HttpServletRequest. Update the client, maybe (the 
http client we use may automatically pass it along)?
 # Make the plan cookie data into a protobuf or other serializable data 
structure
 # Add signing of the cookie data
 # Add expiration of the auth cookie

> Avoid SPNEGO re-negotiation for each request
> 
>
> Key: CALCITE-4152
> URL: https://issues.apache.org/jira/browse/CALCITE-4152
> Project: Calcite
>  Issue Type: Improvement
>  Components: avatica
>Reporter: Istvan Toth
>Priority: Major
>
> When using SPNEGO authentication with Avatica, every HTTP request 
> re-initiates the negotiation, doubling the number HTTP requests.
> Consider switching to cookies after the initial SPNEGO authentication 
> succeeds.
> Jetty ticket that discusses the issue: 
> [https://github.com/eclipse/jetty.project/issues/2868]
> Description of the Knox implementation
> [https://cwiki.apache.org/confluence/display/KNOX/2017/02/24/Hadoop+Auth+%28SPNEGO+and+delegation+token+based+authentication%29+with+Apache+Knox]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures



[ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254361#comment-17254361
 ] 

Josh Elser commented on CALCITE-4367:
-

Pull request is up. If I don't get a review, I'll plan on committing this in a 
few days.

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Reporter: John Bodley
>    Assignee: Josh Elser
>Priority: Trivial
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures



[ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254357#comment-17254357
 ] 

Josh Elser edited comment on CALCITE-4367 at 12/24/20, 1:47 AM:


{quote}
`RpcMetadata` is actually a response as opposed to a miscellaneous type per 
[here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
 and thus requires a {{response}} field. Note I'm not certain if this was 
intentional, i.e., it being a response, however it it is it seems that it 
should be renamed to {{RpcMetadataResponse}} for consistency.
{quote}

 It's intentional that it isn't a response, as a "Response" was intended to be 
top-level messages which might be sent back by the Avatica server. However, 
it's not used by any requests, so it lives in responses.proto to avoid 
polluting common.proto (for no reason). Hope this makes sense. I'll update the 
comment to reflect this.

bq. The supplied ConnectionProperties contains an undocumented dirty field 
(is_dirty for protobuf).

Good catch. Another funny one -- this shouldn't ever be sent over the wire. 
It's just internal state used to avoid unnecessary RPCs (to sync the client's 
properties with the properties in the Avatica server). I'll update the docs for 
now, but we should circle back to avoid this getting serialized at all (for 
json and proto)

The rest of these were spot-on. Thanks for the easy-to-incorporate feedback, 
[~john.bod...@gmail.com]


was (Author: elserj):
{quote}
`RpcMetadata` is actually a response as opposed to a miscellaneous type per 
[here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
 and thus requires a {{response}} field. Note I'm not certain if this was 
intentional, i.e., it being a response, however it it is it seems that it 
should be renamed to {{RpcMetadataResponse}} for consistency.

 It's intentional that it isn't a response, as a "Response" was intended to be 
top-level messages which might be sent back by the Avatica server. However, 
it's not used by any requests, so it lives in responses.proto to avoid 
polluting common.proto (for no reason). Hope this makes sense. I'll update the 
comment to reflect this.

bq. The supplied ConnectionProperties contains an undocumented dirty field 
(is_dirty for protobuf).

Good catch. Another funny one -- this shouldn't ever be sent over the wire. 
It's just internal state used to avoid unnecessary RPCs (to sync the client's 
properties with the properties in the Avatica server). I'll update the docs for 
now, but we should circle back to avoid this getting serialized at all (for 
json and proto)

The rest of these were spot-on. Thanks for the easy-to-incorporate feedback, 
[~john.bod...@gmail.com]

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>    Reporter: John Bodley
>Assignee: Josh Elser
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures



[ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17254357#comment-17254357
 ] 

Josh Elser commented on CALCITE-4367:
-

{quote}
`RpcMetadata` is actually a response as opposed to a miscellaneous type per 
[here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
 and thus requires a {{response}} field. Note I'm not certain if this was 
intentional, i.e., it being a response, however it it is it seems that it 
should be renamed to {{RpcMetadataResponse}} for consistency.

 It's intentional that it isn't a response, as a "Response" was intended to be 
top-level messages which might be sent back by the Avatica server. However, 
it's not used by any requests, so it lives in responses.proto to avoid 
polluting common.proto (for no reason). Hope this makes sense. I'll update the 
comment to reflect this.

bq. The supplied ConnectionProperties contains an undocumented dirty field 
(is_dirty for protobuf).

Good catch. Another funny one -- this shouldn't ever be sent over the wire. 
It's just internal state used to avoid unnecessary RPCs (to sync the client's 
properties with the properties in the Avatica server). I'll update the docs for 
now, but we should circle back to avoid this getting serialized at all (for 
json and proto)

The rest of these were spot-on. Thanks for the easy-to-incorporate feedback, 
[~john.bod...@gmail.com]

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Reporter: John Bodley
>Assignee: Josh Elser
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures



 [ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned CALCITE-4367:
---

Assignee: Josh Elser

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Reporter: John Bodley
>    Assignee: Josh Elser
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4367) Incorrect documentation for Avatica JSON request/response signatures

2020-12-15 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/CALCITE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249864#comment-17249864
 ] 

Josh Elser commented on CALCITE-4367:
-

Thanks for reporting, John. Will see if we can get the docs updated.

> Incorrect documentation for Avatica JSON request/response signatures
> 
>
> Key: CALCITE-4367
> URL: https://issues.apache.org/jira/browse/CALCITE-4367
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Reporter: John Bodley
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I noticed a few inconsistencies between what is documented in the [Avatica 
> JSON Reference|https://calcite.apache.org/avatica/docs/json_reference.html] 
> and what the Avatica JDBC driver provides, specifically:
> # The {{DatabasePropertyRequest}} was missing the {{connection_id}} field in 
> the example signature.
> # `RpcMetadata` is actually a response as opposed to a miscellaneous type per 
> [here|https://github.com/apache/calcite-avatica/blob/4b7eee5bf430b916c7c07897b6f60d2b6b6dabb7/core/src/main/protobuf/responses.proto#L114-L116]
>  and thus requires a {{response}} field. Note I'm not certain if this was 
> intentional, i.e., it being a response, however it it is it seems that it 
> should be renamed to {{RpcMetadataResponse}} for consistency.
> # The supplied {{ConnectionProperties}} contains an undocumented {{dirty}} 
> field ({{is_dirty}} for protobuf).
> # For the {{SchemasRequest}} the {{catalog}} and {{schemaPattern}} are 
> optional rather than required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Clarifying guidance around Signed-off-by in commit messages

2020-12-01 Thread Josh Elser

The ask of this thread was originally to change the semantics of
"signed-off-by" to only be "A committer who gave an explicit +1". That
was the ask from a community member and why I started this.

I want to tease this apart from the "reviewed-by" suggestion, as this
obviously needs a little more polishing. Specifically, this would change
what you had stated as what we "don't care about" today -- a committer
or community member can (today) be listed as the target of a Signed-off-by.

Are you OK with just the change of who can be listed in Signed-off-by,
Nick, while we continue to circle around on how to ensure contributor
reviews recognize merit? I would agree that we need to be sure that we
have the tooling in place to ensure that merit is assigned equally
(especially in a pull-request-centric world we now live in).

On 11/30/20 2:16 PM, Nick Dimiduk wrote:

Nice discussion here.

For my part, I am +1 for our community to define our meaning around this
aspect of metadata.

However, I don't like using both "signed-off-by" and "reviewed-by" as a
manual annotation on the part of the committer, because we as a community
don't care about the distinction between a committer and a community member
in this context. I think that enforcing correct usage of two metadata
annotations, without automation, will be error-prone. If we had some plan
to make use of this metadata, maybe it's worth it, but so far I see no
concrete plan to use this information. So why increase the burden on
committers?

On Sun, Nov 22, 2020 at 11:41 PM Yu Li wrote:

TL;DR: +1 for document rules / guidance of review trailers in commit
message, and +1 for continuing using the signed-off-by message for
"reviewed by" and/or "co-authored-by" semantic (committers only), adding
explicit preamble in the "Git best practice" chapter in our hbase book [1].

I did some research around signed-off-by [2] [3], reviewed-by [3] and
co-authored-by [4], and would like to share my thoughts here:

1. We have been using signed-off-by as the "reviewed by" and/or
"co-authored by" semantic for a long time, starting from the review-board
era (long before github PR).
2. I second that our usage of signed-off-by is a bit of a perversion of the
original [2], thus adding preamble as clarification is necessary.
3. Git offers a signed-off-by switch (-s/--signoff) while no reviewed-by or
co-authored-by support yet, so we need to manually type the message if
choose to use Reviewed-by or Co-authored-by trailers, which means
additional efforts.
4. Based on #3, I suggest that contributors / committers are free but not
required to add "Reviewed-by" and / or "Co-authored-by" trailers manually.
5. Regarding recognizing the review efforts of (new) non-committer
contributors, I suggest we use the Github search [5] (and the commit
efforts as well [6]).

Best Regards,
Yu

[1] http://hbase.apache.org/book.html#git.best.practices
[2]

https://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for
[3] https://wiki.samba.org/index.php/CodeReview#commit_message_tags
[4]

https://docs.github.com/en/free-pro-team@latest/github/committing-changes-to-your-project/creating-a-commit-with-multiple-authors
[5] https://github.com/apache/hbase/pulls?q=is%3Apr+involves%3Acarp84
[6] https://github.com/apache/hbase/commits?author=carp84

On Mon, 23 Nov 2020 at 04:06, Sean Busbey wrote:

I expressly would like to see non-commiters given credit for reviews and
have made a point of including them in prior commits for signed-off-by to
do that.

I'm fine with the idea of us using some other means to indicate this, but
I'd like us to make sure there's not some already widely used bit of git
metadata we could use before picking our own.

It's kind of like when we moved away from amending author (I think that

was

the phrase?) To co authored by when github started pushing that as a way

show multiple authors on a commit.

One thing to keep in mind also is that a big stumbling block to our
consistent crediting of reviewers is a lack of tooling. Having to
distinguish between binding and non binding reviews for putting together
commit metadata will make that more complicated.

On Fri, Nov 20, 2020, 18:15 Stack wrote:

Thanks for taking the time to do a write up Josh.

Looks good to me.

When Sean started in on the 'Signed-off-by:' I didn't get it

(especially

after reading the git definition). Sean then set me straight explaining

our

use is a bit of a perversion of the original. I notice his definition

not in the refguide. Suggest a sentence preamble definition of
'Signed-off-by:' and that we intentionally are different from the
definition cited by Bharath.

I like the Bharath idea on 'Reviewed-by' too. We can talk up

'Reviewed-by'

credits as a way to earn standing in the community, of how they are

given

weight evaluating whe

Re: [DISCUSS] Clarifying guidance around Signed-off-by in commit messages

2020-12-01 Thread Josh Elser

Yeah, that's the intent of what Bharath had suggested and I liked. In 
parallel, see other part of thread from Yu and Nick.


On 11/21/20 11:31 AM, Reid Chan wrote:

Does that mean:
Signed-off-by for binding +1 (from committer),
Reviewed-by for non-binding +1 (from volunteer)?

Sounds good to me.





--

Best regards,
R.C




From: Jan Hentschel 
Sent: 21 November 2020 19:37
To: dev@hbase.apache.org
Subject: Re: [DISCUSS] Clarifying guidance around Signed-off-by in commit 
messages

Also +1 for both suggestions as long as it is clear when to use which. Starting 
point (after the discussion) probably would be to include it in our ref guide.

From: Wellington Chevreuil 
Reply-To: "dev@hbase.apache.org" 
Date: Saturday, November 21, 2020 at 11:37 AM
To: dev 
Subject: Re: [DISCUSS] Clarifying guidance around Signed-off-by in commit 
messages

+1 for both suggestions ('Signed-off-by' and 'Reviewed-by');

Em sáb., 21 de nov. de 2020 às 00:15, Stack 
mailto:st...@duboce.net>> escreveu:

Thanks for taking the time to do a write up Josh.

Looks good to me.

When Sean started in on the 'Signed-off-by:' I didn't get it (especially
after reading the git definition). Sean then set me straight explaining our
use is a bit of a perversion of the original. I notice his definition is
not in the refguide. Suggest a sentence preamble definition of
'Signed-off-by:' and that we intentionally are different from the
definition cited by Bharath.

I like the Bharath idea on 'Reviewed-by' too. We can talk up 'Reviewed-by'
credits as a way to earn standing in the community, of how they are given
weight evaluating whether to make a candidate a committer/PMC'er or not.

S

On Fri, Nov 20, 2020 at 3:13 PM Josh Elser 
mailto:els...@apache.org>> wrote:


On 11/20/20 1:07 PM, Bharath Vissapragada wrote:

* All individuals mentioned in a sign-off*must*  be capable of giving

a

binding vote (i.e. they are an HBase committer)


It appears that the original intent
<



http://web.archive.org/web/20160507011446/http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html<http://web.archive.org/web/20160507011446/http:/gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html>

of
this sign-off feature in git mandates that the signing-off party to be

a

maintainer. So agree with you in theory. However, most times

non-committers

also give great feedback and help with the code review process (code
reviews, testing, perf etc). I think acknowledging their contribution

in

some form would be nice and that encourages potential-future-committers

to

actively review PRs IMO. So how about we annotate their names with
Reviewed-by tags? A related discussion
<https://lists.x.org/archives/xorg-devel/2009-October/003036.html>

on a

different open source project has more tag definitions if we are

interested

in taking that route.

(I know you are only talking about the "signed-off by" tag but I

thought

this discussion would be relevant when documenting this in the dev
guidelines, hence bringing it up). What do you think?


I would be happy with distinguishing Signed-off-by and Reviewed-by as a
way to better track metrics on contributors who review others' code.

Great idea!

Re: [VOTE] Release of Apache Tephra 0.16.0RC2 as Apache Tephra 0.16.0

2020-11-30 Thread Josh Elser


+1 (binding)

* xsums/sigs OK
* CHANGES has reasonable content
* mvn apache-rat:check passes and `find . -type f` doesn't show anything 
unreasonable

* Can build the code
* Ran many unit tests, but cancelled at ~1hr mark.
* Can build Phoenix master against 0.16.0rc2

On 11/30/20 8:15 AM, Istvan Toth wrote:

Please vote on this Apache Phoenix Tephra release candidate,
phoenix-tephra-0.16.0RC2

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache phoenix tephra 0.16.0
[ ] -1 Do not release this package because ...

The tag to be voted on is 0.16.0RC2

   https://github.com/apache/phoenix-tephra/tree/0.16.0RC2

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

   https://dist.apache.org/repos/dist/dev/phoenix/phoenix-tephra-0.16.0RC2/

Maven artifacts are available in a staging repository at:

   https://repository.apache.org/content/repositories/orgapachephoenix-1206/

Artifacts were signed with the 0x794433C7 key which can be found in:

   https://dist.apache.org/repos/dist/release/phoenix/KEYS

To learn more about Apache Phoenix Tephra, please see

   https://tephra.incubator.apache.org/

Thanks,
Istvan

Re: Please describe website update procedure for updating Phoenix and Omid

2020-11-30 Thread Josh Elser

Should we just incorporate the omid and tephra websites into phoenix.a.o 
going forward?


I'm surprised infra didn't kill the old websites when the IPMC 
"graduated" them.


ASF websites are either updated via svn pub-sub, git pub-sub, or via ASF 
CMS (which might be EOL, I forget).


svn/git pub-sub essentially work the same way: when you commit to a 
certain branch in a repository, magic happens on the infra side to 
update the corresponding website. For SVN pubsub, this is just some 
repo. For Git pubsub, the website branch defaults to "asf-site".


ASF CMS was a home-grown content management system in which you would go 
into your website, click a special bookmark, use a WYSIWYG editor, 
stage, and publish changes.


If we don't know what Omid (and Tephra) are using, we can ask infra.

On 11/25/20 2:35 AM, Istvan Toth wrote:

I have documented my unsuccessful attempt to update the Omid website in
https://issues.apache.org/jira/browse/OMID-190

Still looking for any information on update procedure for both websites.

regards
Istvan

On Mon, Nov 9, 2020 at 1:30 PM Istvan Toth  wrote:


Hi!

We have documented how to update the Phoenix website on the website itself.
However, I could not find similar documentation for Omid and Tephra.

If you have updated either website in the past, or know the procedure to
update either, please describe the procedure, or link the relevant
documentation.

thanks in advance
Istvan

[jira] [Resolved] (HBASE-24268) REST and Thrift server do not handle the "doAs" parameter case insensitively



 [ 
https://issues.apache.org/jira/browse/HBASE-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-24268.

Hadoop Flags: Reviewed
Release Note: This change allows the REST and Thrift servers to handle the 
"doAs" parameter case-insensitively, which is deemed as correct per the 
"specification" provided by the Hadoop community.
  Resolution: Fixed

Thanks for the contribution, [~RichardAntal]!

> REST and Thrift server do not handle the "doAs" parameter case insensitively
> 
>
> Key: HBASE-24268
> URL: https://issues.apache.org/jira/browse/HBASE-24268
> Project: HBase
>  Issue Type: Bug
>  Components: REST, Thrift
>Reporter: Istvan Toth
>Assignee: Richard Antal
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Hadoop does a case-insensitve comparison on the doAs parameter name when 
> handling principal impersonation.
> The HBase Rest and Thrift servers do not do that, they only accept the "doAs" 
> form.
> According to HADOOP-11083, the the correct Hadoop behaviour is accepting doAs 
> in any case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24268) REST and Thrift server do not handle the "doAs" parameter case insensitively



 [ 
https://issues.apache.org/jira/browse/HBASE-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-24268.

Hadoop Flags: Reviewed
Release Note: This change allows the REST and Thrift servers to handle the 
"doAs" parameter case-insensitively, which is deemed as correct per the 
"specification" provided by the Hadoop community.
  Resolution: Fixed

Thanks for the contribution, [~RichardAntal]!

> REST and Thrift server do not handle the "doAs" parameter case insensitively
> 
>
> Key: HBASE-24268
> URL: https://issues.apache.org/jira/browse/HBASE-24268
> Project: HBase
>  Issue Type: Bug
>  Components: REST, Thrift
>Reporter: Istvan Toth
>Assignee: Richard Antal
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Hadoop does a case-insensitve comparison on the doAs parameter name when 
> handling principal impersonation.
> The HBase Rest and Thrift servers do not do that, they only accept the "doAs" 
> form.
> According to HADOOP-11083, the the correct Hadoop behaviour is accepting doAs 
> in any case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24268) REST and Thrift server do not handle the "doAs" parameter case insensitively



 [ 
https://issues.apache.org/jira/browse/HBASE-24268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-24268:
---
Fix Version/s: 2.4.0
   3.0.0-alpha-1

> REST and Thrift server do not handle the "doAs" parameter case insensitively
> 
>
> Key: HBASE-24268
> URL: https://issues.apache.org/jira/browse/HBASE-24268
> Project: HBase
>  Issue Type: Bug
>  Components: REST, Thrift
>Reporter: Istvan Toth
>Assignee: Richard Antal
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Hadoop does a case-insensitve comparison on the doAs parameter name when 
> handling principal impersonation.
> The HBase Rest and Thrift servers do not do that, they only accept the "doAs" 
> form.
> According to HADOOP-11083, the the correct Hadoop behaviour is accepting doAs 
> in any case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb



 [ 
https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25278:
---
Release Note: A new option, CACHE_BLOCKS, was added to the `count` shell 
command which will force the data for a table to be loaded into the block 
cache. By default, the `count` command will not cache any blocks. This option 
can serve as a means to for a table's data to be loaded into block cache on 
demand. See the help message on the count shell command for usage details.

> Add option to toggle CACHE_BLOCKS in count.rb
> -
>
> Key: HBASE-25278
> URL: https://issues.apache.org/jira/browse/HBASE-25278
> Project: HBase
>  Issue Type: New Feature
>  Components: shell
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is 
> to edit table.rb so that the `count` shell command will not instruct 
> RegionServers to not cache any data blocks. This is a quick+dirty way to 
> force a table to be loaded into block cache (i.e. for performance testing).
> We can easily add another option to avoid having to edit the ruby files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb



 [ 
https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25278:
---
Hadoop Flags: Reviewed
  Resolution: Fixed
  Status: Resolved  (was: Patch Available)

Thanks Peter and Zach for the reviews!

> Add option to toggle CACHE_BLOCKS in count.rb
> -
>
> Key: HBASE-25278
> URL: https://issues.apache.org/jira/browse/HBASE-25278
> Project: HBase
>  Issue Type: New Feature
>  Components: shell
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is 
> to edit table.rb so that the `count` shell command will not instruct 
> RegionServers to not cache any data blocks. This is a quick+dirty way to 
> force a table to be loaded into block cache (i.e. for performance testing).
> We can easily add another option to avoid having to edit the ruby files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Clarifying guidance around Signed-off-by in commit messages

2020-11-20 Thread Josh Elser


On 11/20/20 1:07 PM, Bharath Vissapragada wrote:

* All individuals mentioned in a sign-off*must*  be capable of giving a
binding vote (i.e. they are an HBase committer)


It appears that the original intent
of
this sign-off feature in git mandates that the signing-off party to be a
maintainer. So agree with you in theory. However, most times non-committers
also give great feedback and help with the code review process (code
reviews, testing, perf etc). I think acknowledging their contribution in
some form would be nice and that encourages potential-future-committers to
actively review PRs IMO. So how about we annotate their names with
Reviewed-by tags? A related discussion
  on a
different open source project has more tag definitions if we are interested
in taking that route.

(I know you are only talking about the "signed-off by" tag but I thought
this discussion would be relevant when documenting this in the dev
guidelines, hence bringing it up). What do you think?


I would be happy with distinguishing Signed-off-by and Reviewed-by as a 
way to better track metrics on contributors who review others' code.


Great idea!

[DISCUSS] Clarifying guidance around Signed-off-by in commit messages

2020-11-20 Thread Josh Elser


Hi!

As most of you know, we've been using the "Signed-off-by:  
" line in out commit messages more and more lately to indicate 
who reviewed some change.


We've recently had an event in which one of these Signed-off-by lines 
showed up with someone's name who didn't consider themselves to have 
signed-off on the change. This is akin to saying someone gave a +1 for 
some change when they did not. As an RTC community, that's worrisome.


I went reading the HBase book and was surprised to not find guidance on 
how we expect this to work, so I'd like to have some discussion about 
how we should treat these lines. I'll start this off by making 
suggestions about what seems reasonable to me.


When a committer is applying some change in a commit:

* All individuals mentioned in a sign-off *must* be capable of giving a 
binding vote (i.e. they are an HBase committer)
* Any individual in a sign-off *must* have given approval via an 
explicit "+1" or the "Approved" via the Github Pull Request review function.
* Approval *must* be publicly visible and memorialized on the code 
review (e.g. no private emails or chat message to give approval)
* The committer _should_ (not *must*) create a sign-off line for each 
binding reviewer who gave approval


I think these are generally how we have been operating, but it would be 
good to make sure they are documented as such.


Thoughts/concerns?

- Josh

Re: [VOTE] Release of Apache Omid 1.0.2

2020-11-20 Thread Josh Elser


(Because I accidentally sent just to Istvan the first time)

On 11/19/20 3:27 PM, Josh Elser wrote:

+1 binding

* NOTICE has out of date copyright year. Fix for later. License appears 
fine (some copyright retained by Yahoo on some benchmark files, it 
seems, but still apache licensed)

* xsums/sigs OK
* Some errors reported by `mvn apache-rat:check`. Fine to ship in this, 
IMO. Let's get fixed for next release.


```
   misc/findbugs-exclude.xml
   misc/omid_checks.xml
   bintray-settings.xml
   doc/images/ModuleDependencies.graffle
   doc/site/site.xml
   .travis.yml
```

* Can run unit tests
* Can build Phoenix's master branch with 1.0.2rc0 (have to use the 
profile `-Phbase-2` on phoenix-omid when building). A smell we can fix 
later in omid.

* Phoenix unit tests pass with 1.0.2rc0

Looks great to me by that! Nice work, Istvan.

- Josh

On 11/17/20 10:24 AM, Istvan Toth wrote:

Please vote on this Apache phoenix omid release candidate,
phoenix-omid-1.0.2RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache phoenix omid 1.0.2
[ ] -1 Do not release this package because ...

The tag to be voted on is 1.0.2RC0:

   https://github.com/apache/phoenix-omid/tree/1.0.2RC0

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

   https://dist.apache.org/repos/dist/dev/phoenix/phoenix-omid-1.0.2RC0/

Maven artifacts are available in a staging repository at:

   
https://repository.apache.org/content/repositories/orgapachephoenix-1203/


Artifacts were signed with the 0x794433C7 key which can be found in:

   https://dist.apache.org/repos/dist/release/phoenix/KEYS

To learn more about Apache phoenix omid, please see

   https://omid.incubator.apache.org/

Thanks,
Istvan

[jira] [Resolved] (CALCITE-4379) Meta.Frame created with java float values in rows hits a ClassCastException in toProto()

2020-11-13 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/CALCITE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved CALCITE-4379.
-
Fix Version/s: avatica-1.18.0
   Resolution: Fixed

Pushed! Thanks for the contribution, Dmitri!

> Meta.Frame created with java float values in rows hits a ClassCastException 
> in toProto()
> 
>
> Key: CALCITE-4379
> URL: https://issues.apache.org/jira/browse/CALCITE-4379
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Affects Versions: 1.17.0
>Reporter: Dmitri Bourlatchkov
>Assignee: Dmitri Bourlatchkov
>Priority: Major
>  Labels: pull-request-available
> Fix For: avatica-1.18.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Use case:
> * Custom {{Meta}} implementation
> * {{Meta.Frame.create(offset, done, rows)}} is called with a row (a {{List}}) 
> containing a java {{Float}} value
> ** Note: the float value is fetched from Apache Cassandra, which has 32-bit 
> float types (unlike SQL).
> ** Related [email 
> discussion|https://mail-archives.apache.org/mod_mbox/calcite-dev/202011.mbox/%3CCAPSgeEREWLTNpBTpNBe4TY4hvwBWR9x-5BxAckKOTAE5QpoP9Q%40mail.gmail.com%3E]
>  
> * The {{Frame}} is serialized by calling its {{.toProto()}} method
> * ClassCastException occurs
> Exception snippet:
> {noformat}
> java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Float
> at 
> org.apache.calcite.avatica.remote.TypedValue.writeToProtoWithType(TypedValue.java:600)
> at org.apache.calcite.avatica.remote.TypedValue.toProto(TypedValue.java:805)
> at org.apache.calcite.avatica.Meta$Frame.serializeScalar(Meta.java:991)
> at org.apache.calcite.avatica.Meta$Frame.parseColumn(Meta.java:977)
> at org.apache.calcite.avatica.Meta$Frame.toProto(Meta.java:942)
> at 
> org.apache.calcite.avatica.remote.Service$FetchResponse.serialize(Service.java:1468)
> [...snip...]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher



[ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231052#comment-17231052
 ] 

Josh Elser commented on HBASE-25279:


Yeah, I think you're right. I didn't go looking to see if it should've been 
closed down. In general, having any non-daemon threads in HBase processes is a 
smell (which is what caught my attention).

Looks like HMaster creates an ZKWatcher but doesn't close it. Sounds like 
something else to follow-up on.

I'm not sure if we get into any troubles just marking this thread as daemon. I 
would think not, it's just a smell to not be proactively closing stuff down 
that we spawn :)

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25279) Non-daemon thread in ZKWatcher



 [ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25279:
---
Status: Patch Available  (was: Open)

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher



[ 
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230984#comment-17230984
 ] 

Josh Elser commented on HBASE-25279:


[~bharathv], [~vjasani], be careful here please, gents. Marking the threads as 
daemon is super important.

> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
>  Issue Type: Bug
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
> which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25279) Non-daemon thread in ZKWatcher

Josh Elser created HBASE-25279:
--

 Summary: Non-daemon thread in ZKWatcher
 Key: HBASE-25279
 URL: https://issues.apache.org/jira/browse/HBASE-25279
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 3.0.0-alpha-1


ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25279) Non-daemon thread in ZKWatcher

Josh Elser created HBASE-25279:
--

 Summary: Non-daemon thread in ZKWatcher
 Key: HBASE-25279
 URL: https://issues.apache.org/jira/browse/HBASE-25279
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 3.0.0-alpha-1


ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons 
which will prevent clean shut downs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (CALCITE-4379) Meta.Frame created with java float values in rows hits a ClassCastException in toProto()



 [ 
https://issues.apache.org/jira/browse/CALCITE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned CALCITE-4379:
---

Assignee: Dmitri Bourlatchkov

> Meta.Frame created with java float values in rows hits a ClassCastException 
> in toProto()
> 
>
> Key: CALCITE-4379
> URL: https://issues.apache.org/jira/browse/CALCITE-4379
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Affects Versions: 1.17.0
>Reporter: Dmitri Bourlatchkov
>Assignee: Dmitri Bourlatchkov
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Use case:
> * Custom {{Meta}} implementation
> * {{Meta.Frame.create(offset, done, rows)}} is called with a row (a {{List}}) 
> containing a java {{Float}} value
> ** Note: the float value is fetched from Apache Cassandra, which has 32-bit 
> float types (unlike SQL).
> ** Related [email 
> discussion|https://mail-archives.apache.org/mod_mbox/calcite-dev/202011.mbox/%3CCAPSgeEREWLTNpBTpNBe4TY4hvwBWR9x-5BxAckKOTAE5QpoP9Q%40mail.gmail.com%3E]
>  
> * The {{Frame}} is serialized by calling its {{.toProto()}} method
> * ClassCastException occurs
> Exception snippet:
> {noformat}
> java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Float
> at 
> org.apache.calcite.avatica.remote.TypedValue.writeToProtoWithType(TypedValue.java:600)
> at org.apache.calcite.avatica.remote.TypedValue.toProto(TypedValue.java:805)
> at org.apache.calcite.avatica.Meta$Frame.serializeScalar(Meta.java:991)
> at org.apache.calcite.avatica.Meta$Frame.parseColumn(Meta.java:977)
> at org.apache.calcite.avatica.Meta$Frame.toProto(Meta.java:942)
> at 
> org.apache.calcite.avatica.remote.Service$FetchResponse.serialize(Service.java:1468)
> [...snip...]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb



 [ 
https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-25278:
---
Status: Patch Available  (was: Open)

> Add option to toggle CACHE_BLOCKS in count.rb
> -
>
> Key: HBASE-25278
> URL: https://issues.apache.org/jira/browse/HBASE-25278
> Project: HBase
>  Issue Type: New Feature
>  Components: shell
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is 
> to edit table.rb so that the `count` shell command will not instruct 
> RegionServers to not cache any data blocks. This is a quick+dirty way to 
> force a table to be loaded into block cache (i.e. for performance testing).
> We can easily add another option to avoid having to edit the ruby files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb

Josh Elser created HBASE-25278:
--

 Summary: Add option to toggle CACHE_BLOCKS in count.rb
 Key: HBASE-25278
 URL: https://issues.apache.org/jira/browse/HBASE-25278
 Project: HBase
  Issue Type: New Feature
  Components: shell
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 3.0.0-alpha-1, 2.4.0


A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is 
to edit table.rb so that the `count` shell command will not instruct 
RegionServers to not cache any data blocks. This is a quick+dirty way to force 
a table to be loaded into block cache (i.e. for performance testing).

We can easily add another option to avoid having to edit the ruby files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb

Josh Elser created HBASE-25278:
--

 Summary: Add option to toggle CACHE_BLOCKS in count.rb
 Key: HBASE-25278
 URL: https://issues.apache.org/jira/browse/HBASE-25278
 Project: HBase
  Issue Type: New Feature
  Components: shell
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: 3.0.0-alpha-1, 2.4.0


A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is 
to edit table.rb so that the `count` shell command will not instruct 
RegionServers to not cache any data blocks. This is a quick+dirty way to force 
a table to be loaded into block cache (i.e. for performance testing).

We can easily add another option to avoid having to edit the ruby files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Phoenix connection issue

2020-10-26 Thread Josh Elser

If you are not using Kerberos authentication, then you need to figure 
out how your DNS names are resolving for your VM. A 
SocketTimeoutException means that your client could not open an socket 
to that host:port which was specified. Likely, this is something you 
need to figure out with your VM.

On 10/23/20 10:29 AM, Varun Rajavelu wrote:

Hey Hi,

Thanks for the response. Yeah I'm not using in any Kerberos 
authentication hbase still in simple authentication method only. Can you 
please tell me what I'm made any mistakes in these presto catalog 
configuration, while communication between hbase and presto. Please help 
me or suggestions on this issue. I can able to query hbase table from 
phoenix but issue comes only from connecting presto phoenix only.

On Fri, 23 Oct, 2020, 7:27 pm Josh Elser, <mailto:els...@apache.org>> wrote:

(-to: dev@phoenix, +bcc: dev@phoenix, +to: user@phoenix)

I've taken the liberty of moving this over to the user list.

Typically, such an exception is related to Kerberos authentication,
when
the HBase service denies in an incoming, non-authenticated client.
However, since you're running inside of a VM, I'd suggest you should
validate that your environment is sane, first.

HDP 2.6 never shipped an HBase 1.5 release, so it seems like you have
chosen to own your own set of versions. I'll assume that you have
already validated that the versions of Hadoop, HBase, and Phoenix that
you are running are compatible with one another. A common debugging
trick is to strip away the layers of complexity so that you can isolate
your problem. Can you talk to HBase or Phoenix not within Presto?

On 10/23/20 8:58 AM, Varun Rajavelu wrote:
 > Hi,,
 >
 > I'm currently using HDP 2.6 and hbase1.5 and phoenix 4.7 and
presto-server
 > which i have been using presto344. Im facing issue while connecting
 > prestoserver with hbase;
 >
 > *./presto --server localhost:14100 --catalog phoenix*
 > *Issue:*
 > Fri Oct 23 18:11:06 IST 2020, null, java.net
<http://java.net>.SocketTimeoutException:
 > callTimeout=6, callDuration=69066: Call to
 > sandbox-hdp.hortonworks.com/127.0.0.1:16020
<http://sandbox-hdp.hortonworks.com/127.0.0.1:16020> failed on local
exception:
 > java.io.EOFException row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
 > region=hbase:meta,,1.1588230740,
 > hostname=sandbox-hdp.hortonworks.com
<http://sandbox-hdp.hortonworks.com>,16020,1603447852070,
 > seqNum=0
 >
 > *My presto catalog Config:*
 > connector.name <http://connector.name>=phoenix
 > phoenix.connection-url=jdbc:phoenix:localhost:2181:/hbase-unsecure
 >
phoenix.config.resources=/home/desa/Downloads/hbase_schema/hbase-site.xml
 >
 > Kindly please have a look and help me to resolve this issue.
 >

Re: Phoenix connection issue

2020-10-23 Thread Josh Elser


(-to: dev@phoenix, +bcc: dev@phoenix, +to: user@phoenix)

I've taken the liberty of moving this over to the user list.

Typically, such an exception is related to Kerberos authentication, when 
the HBase service denies in an incoming, non-authenticated client. 
However, since you're running inside of a VM, I'd suggest you should 
validate that your environment is sane, first.


HDP 2.6 never shipped an HBase 1.5 release, so it seems like you have 
chosen to own your own set of versions. I'll assume that you have 
already validated that the versions of Hadoop, HBase, and Phoenix that 
you are running are compatible with one another. A common debugging 
trick is to strip away the layers of complexity so that you can isolate 
your problem. Can you talk to HBase or Phoenix not within Presto?


On 10/23/20 8:58 AM, Varun Rajavelu wrote:

Hi,,

I'm currently using HDP 2.6 and hbase1.5 and phoenix 4.7 and presto-server
which i have been using presto344. Im facing issue while connecting
prestoserver with hbase;

*./presto --server localhost:14100 --catalog phoenix*
*Issue:*
Fri Oct 23 18:11:06 IST 2020, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=69066: Call to
sandbox-hdp.hortonworks.com/127.0.0.1:16020 failed on local exception:
java.io.EOFException row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=sandbox-hdp.hortonworks.com,16020,1603447852070,
seqNum=0

*My presto catalog Config:*
connector.name=phoenix
phoenix.connection-url=jdbc:phoenix:localhost:2181:/hbase-unsecure
phoenix.config.resources=/home/desa/Downloads/hbase_schema/hbase-site.xml

Kindly please have a look and help me to resolve this issue.

Re: Phoenix connection issue

2020-10-23 Thread Josh Elser


(-to: dev@phoenix, +bcc: dev@phoenix, +to: user@phoenix)

I've taken the liberty of moving this over to the user list.

Typically, such an exception is related to Kerberos authentication, when 
the HBase service denies in an incoming, non-authenticated client. 
However, since you're running inside of a VM, I'd suggest you should 
validate that your environment is sane, first.


HDP 2.6 never shipped an HBase 1.5 release, so it seems like you have 
chosen to own your own set of versions. I'll assume that you have 
already validated that the versions of Hadoop, HBase, and Phoenix that 
you are running are compatible with one another. A common debugging 
trick is to strip away the layers of complexity so that you can isolate 
your problem. Can you talk to HBase or Phoenix not within Presto?


On 10/23/20 8:58 AM, Varun Rajavelu wrote:

Hi,,

I'm currently using HDP 2.6 and hbase1.5 and phoenix 4.7 and presto-server
which i have been using presto344. Im facing issue while connecting
prestoserver with hbase;

*./presto --server localhost:14100 --catalog phoenix*
*Issue:*
Fri Oct 23 18:11:06 IST 2020, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=69066: Call to
sandbox-hdp.hortonworks.com/127.0.0.1:16020 failed on local exception:
java.io.EOFException row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=sandbox-hdp.hortonworks.com,16020,1603447852070,
seqNum=0

*My presto catalog Config:*
connector.name=phoenix
phoenix.connection-url=jdbc:phoenix:localhost:2181:/hbase-unsecure
phoenix.config.resources=/home/desa/Downloads/hbase_schema/hbase-site.xml

Kindly please have a look and help me to resolve this issue.

Re: [DISCUSS] Phoenix bot account for GitHub

2020-10-21 Thread Josh Elser


Sounds good to me. Yes, would say that private SVN space is good.

I can't find any existing private space, so I think it would be an ask 
to infra, e.g. https://issues.apache.org/jira/browse/INFRA-15461



On 10/21/20 1:23 AM, Istvan Toth wrote:

Hi!

I've recently implemented the Yetus PR checks for GitHub PR, which for the
most part seem to work well.

However, it seems that none of the available GitHub credentials in Jenkins
let the job comment on the PR, so ATM the jobs are using my GitHub account.
It is not very professional looking, and I get spammed with mail on every
entry on every ticket that has a PR, which makes life difficult for me.

Looking at HBase (as ever), they have created a bot account, and are using
it for the same purpose.

I propose that we do similarly. The GitHub docs

seem
to indicate that it is OK.

One open question is how to share the credentials, so that I am not not the
only one with access. I seem to recall that we have a private SVN or git
repo for Phoenix members somewhere, that we could use to store the
login/password for it, but I cannot find it now.

Please share your opinion, and point me to the private repo, or the docs
that describes it.

regards
Istvan

Re: [VOTE] Release of phoenix-thirdparty 1.0.0

2020-10-21 Thread Josh Elser


+1 (binding)

* L seem fine
* RELEASENOTES.md is empty, maybe we'll have content in there later? 
(Mentioning in case there should have been content)

* phoenix-shaded-guava jar looks sane
* apache-rat:check passes
* Nice content in CHANGES.md
* Built phoenix.git with this release

On 10/14/20 3:05 AM, Istvan Toth wrote:

Please vote on this Apache phoenix-thirdparty release candidate,
phoenix-thirdparty-1.0.0RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache phoenix-thirdparty 1.0.0
[ ] -1 Do not release this package because ...

The tag to be voted on is 1.0.0RC0

   https://github.com/apache/phoenix-thirdparty/tree/1.0.0RC0

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:


https://dist.apache.org/repos/dist/dev/phoenix/phoenix-thirdparty-1.0.0RC0/

Artifacts were signed with the ${GPG_KEY} key which can be found in:

   https://dist.apache.org/repos/dist/release/phoenix/KEYS

To learn more about Apache Phoenix, please see

   http://phoenix.apache.org/

Thanks,
Istvan

[jira] [Commented] (PHOENIX-5066) The TimeZone is incorrectly used during writing or reading data

2020-10-19 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216965#comment-17216965
 ] 

Josh Elser commented on PHOENIX-5066:
-

{quote}Let's say I want to write 2020-05-12 12:26:26 (and I am in GMT+2 
timezone) the server has it's phoenix.query.dateFormatTimeZone property on the 
default GMT.
 So what I would like to see is that the object that we store in HBase should 
be the byte[] equivalent of 2020-05-12 10:26:26 GMT.
 But when we read this data again it should be represented in our local 
timezone and get 2020-05-12 12:26:26 GMT+2 back even if we use the getObject or 
getString methods.
{quote}
IMO this is the correct solution to the problem that actually scales out to 
different groups using a database. I worry that Janaai's suggestion won't work 
when you have multiple teams/applications trying to read the same table. If 
everyone can agree "date/time is converted to GMT through Phoenix", then each 
team can make sure that their application's timezone is set appropriately and 
they should be able to get "stable" results from Phoenix.

Let me take another look at your PR :)

> The TimeZone is incorrectly used during writing or reading data
> ---
>
> Key: PHOENIX-5066
> URL: https://issues.apache.org/jira/browse/PHOENIX-5066
> Project: Phoenix
>  Issue Type: Bug
>Affects Versions: 5.0.0, 4.14.1
>Reporter: Jaanai Zhang
>Assignee: Richard Antal
>Priority: Critical
> Fix For: 5.1.1, 4.16.0
>
> Attachments: DateTest.java, PHOENIX-5066.4x.v1.patch, 
> PHOENIX-5066.4x.v2.patch, PHOENIX-5066.4x.v3.patch, 
> PHOENIX-5066.master.v1.patch, PHOENIX-5066.master.v2.patch, 
> PHOENIX-5066.master.v3.patch, PHOENIX-5066.master.v4.patch, 
> PHOENIX-5066.master.v5.patch, PHOENIX-5066.master.v6.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have two methods to write data when uses JDBC API.
> #1. Uses _the exceuteUpdate_ method to execute a string that is an upsert SQL.
> #2. Uses the _prepareStatement_ method to set some objects and execute.
> The _string_ data needs to convert to a new object by the schema information 
> of tables. we'll use some date formatters to convert string data to object 
> for Date/Time/Timestamp types when writes data and the formatters are used 
> when reads data as well.
>  
> *Uses default timezone test*
>  Writing 3 records by the different ways.
> {code:java}
> UPSERT INTO date_test VALUES (1,'2018-12-10 15:40:47','2018-12-10 
> 15:40:47','2018-12-10 15:40:47') 
> UPSERT INTO date_test VALUES (2,to_date('2018-12-10 
> 15:40:47'),to_time('2018-12-10 15:40:47'),to_timestamp('2018-12-10 15:40:47'))
> stmt.setInt(1, 3);stmt.setDate(2, date);stmt.setTime(3, 
> time);stmt.setTimestamp(4, ts);
> {code}
> Reading the table by the getObject(getDate/getTime/getTimestamp) methods.
> {code:java}
> 1 | 2018-12-10 | 23:45:07 | 2018-12-10 23:45:07.0 
> 2 | 2018-12-10 | 23:45:07 | 2018-12-10 23:45:07.0 
> 3 | 2018-12-10 | 15:45:07 | 2018-12-10 15:45:07.66 
> {code}
> Reading the table by the getString methods 
> {code:java}
> 1 | 2018-12-10 15:45:07.000 | 2018-12-10 15:45:07.000 | 2018-12-10 
> 15:45:07.000 
> 2 | 2018-12-10 15:45:07.000 | 2018-12-10 15:45:07.000 | 2018-12-10 
> 15:45:07.000 
> 3 | 2018-12-10 07:45:07.660 | 2018-12-10 07:45:07.660 | 2018-12-10 
> 07:45:07.660
> {code}
>  *Uses GMT+8 test*
>  Writing 3 records by the different ways.
> {code:java}
> UPSERT INTO date_test VALUES (1,'2018-12-10 15:40:47','2018-12-10 
> 15:40:47','2018-12-10 15:40:47')
> UPSERT INTO date_test VALUES (2,to_date('2018-12-10 
> 15:40:47'),to_time('2018-12-10 15:40:47'),to_timestamp('2018-12-10 15:40:47'))
> stmt.setInt(1, 3);stmt.setDate(2, date);stmt.setTime(3, 
> time);stmt.setTimestamp(4, ts);
> {code}
> Reading the table by the getObject(getDate/getTime/getTimestamp) methods.
> {code:java}
> 1 | 2018-12-10 | 23:40:47 | 2018-12-10 23:40:47.0 
> 2 | 2018-12-10 | 15:40:47 | 2018-12-10 15:40:47.0 
> 3 | 2018-12-10 | 15:40:47 | 2018-12-10 15:40:47.106 {code}
> Reading the table by the getString methods
> {code:java}
>  1 | 2018-12-10 23:40:47.000 | 2018-12-10 23:40:47.000 | 2018-12-10 
> 23:40:47.000
> 2 | 2018-12-10 15:40:47.000 | 2018-12-10 15:40:47.000 | 2018-12-10 
> 15:40:47.000
> 3 | 2018-12-10 15:40:47.106 | 2018-12-10 15:40:47.106 | 2018-12-10 
> 15:40:47.106
> {code}
>  
> _We_ have a historical problem,  we'll parse the string to 
> Date/Time/Timestamp objects with timezone in #1, which means the actual data 
> is going to be changed when stored in HBase table。



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-25142) Auto-fix 'Unknown Server'

2020-10-14 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-25142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214044#comment-17214044
 ] 

Josh Elser commented on HBASE-25142:


{quote}The 'unkown server' means we have bugs in code so we are not 100% sure 
what is the real problem and whether it is 100% safe to fix.
{quote}
I keep coming back and thinking more about this. I've not made an exhaustive 
search (so maybe I'll rely on y'all to vet the idea), but I think we have two 
paths which might lead to an "unknown" server:
 * We lost WALs: both of Stack's example in the description are indicative of 
that. Just, one was intentional and the other was circumstantial.
 * We have a bug. I can respect Duo's caution around trying to automatically 
fix something that we don't yet understand.

Is there a third (or more) that I'm missing?

The second thing I keep wondering, say we do hit Duo's concern and have some 
unknown bug where we incorrectly clean up the WALs for a RS before the Master 
can submit an SCP (say it crashed as the same time and didn't see the ZK 
expiration). Is marking a server as unknown and reassigning regions that were 
on it going to cause problems _beyond_ the problems we already have (WALs went 
missing)? I can't come up with a situation that (as long as the server isn't 
registered in ZK), where starting to reassign regions which were once on that 
server is going to make matters worse. I fully acknowledge that I may be 
missing a subtle condition :)
{quote}I would suggest that we should at least provide an option for the 'auto' 
fix.
{quote}
Just making sure. Duo – you're saying you'd be happy with a solution in HBase 
to automatically fix this, as long as it was off-by-default?

> Auto-fix 'Unknown Server'
> -
>
> Key: HBASE-25142
> URL: https://issues.apache.org/jira/browse/HBASE-25142
> Project: HBase
>  Issue Type: Improvement
>Reporter: Michael Stack
>Priority: Major
>
> Addressing reports of 'Unknown Server' has come up in various conversations 
> lately. This issue is about fixing instances of 'Unknown Server' 
> automatically as part of the tasks undertaken by CatalogJanitor when it runs.
> First though, would like to figure a definition for 'Unknown Server' and a 
> list of ways in which they arise. We need this to figure how to do safe 
> auto-fixing.
> Currently an 'Unknown Server' is a server found in hbase:meta that is not 
> online (no recent heartbeat) and that is not mentioned in the dead servers 
> list.
> In outline, I'd think CatalogJanitor could schedule an expiration of the RS 
> znode in zk (if exists) and then an SCP if it finds an 'Unknown Server'. 
> Perhaps it waits for 2x or 10x the heartbeat interval just-in-case (or not). 
> The SCP would clean up any references in hbase:meta by reassigning Regions 
> assigned the 'Unknown Server' after replaying any WALs found in hdfs 
> attributed to the dead server.
> As to how they arise:
>  * A contrived illustration would be a large online cluster crashes down with 
> a massive backlog of WAL files – zk went down for some reason say. The replay 
> of the WALs look like it could take a very long time  (lets say the cluster 
> was badly configured and a bug and misconfig made it so each RS was carrying 
> hundreds of WALs and there are hundreds of servers). To get the service back 
> online, the procedure store and WALs are moved aside (for later replay with 
> WALPlayer). The cluster comes up. meta is onlined but refers to server 
> instances that are no longer around. Can schedule an SCP per server mentioned 
> in the 'HBCK Report' by scraping and scripting hbck2 or, better, 
> catalogjanitor could just do it.
>  * HBASE-24286 HMaster won't become healthy after after cloning... describes 
> starting a cluster over data that is hfile-content only. In this case the 
> original servers used manufacture the hfile cluster data are long dead yet 
> meta still refers to the old servers. They will not make the 'dead servers' 
> list.
> Let this issue stew awhile. Meantime collect how 'Unknown Server' gets 
> created and best way to fix.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate

2020-09-23 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201004#comment-17201004
 ] 

Josh Elser commented on HBASE-23887:


{quote}could you explain please, how to enable AdaptiveLruBlockCache 
{quote}
The suggestion is that you create a new class called AdaptiveLruBlockCache (or 
any new name you think is appropriate) which either extends LruBlockCache and 
contains the modifications over the LruBlockCache or copies the code from 
LruBlockCache and modifies the copied code inside AdaptiveLruBlockCache. Then, 
to use your new feature, the user would specify that the new 
AdaptiveLruBlockCache via configuration.

> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg, 
> 1582801838065_rs_metrics_new.png, BC_LongRun.png, 
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png, 
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png, 
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png, 
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png, 
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png, 
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png, 
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png, 
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png, 
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png, 
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png, 
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than 
> BlockChache (usual story in BigData). The idea - caching only part of DATA 
> blocks. It is good becouse LruBlockCache starts to work and save huge amount 
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a 
> high rate of evictions. In this case we can skip cache a block N and insted 
> cache the N+1th block. Anyway we would evict N block quite soon and that why 
> that skipping good for performance.
> ---
> Some information below isn't  actual
> ---
>  
>  
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to 
> read 3 blocks with offsets:
>  124
>  198
>  223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict 
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we 
> divide by modulus we got:
>  124 -> 24
>  198 -> 98
>  223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set 
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip 
> others. It means we will not try to handle the block 198 and save CPU for 
> other job. In the result - we put block 124, then put 223, evict 124 (3 
> actions).
> See the picture in attachment with test below. Requests per second is higher, 
> GC is lower.
>  
>  The key point of the code:
>  Added the parameter: *hbase.lru.cache.data.block.percent* which by default = 
> 100
>   
>  But if we set it 1-99, then will work the next logic:
>   
>   
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
> inMemory) {   
>   if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())      
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) 
>   return;    
> ... 
> // the same code as usual
> }
> {code}
>  
> Other parameters help to control when this logic will be enabled. It means it 
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run 
> eviction process that start to avoid of putting data to BlockCache
>  hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to 
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time) 
> then we start

[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate

2020-09-22 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200283#comment-17200283
 ] 

Josh Elser commented on HBASE-23887:


bq. While looking at the patch, I was wondering if we should implement this as 
a separate Cache implementation that extends LruBlockCache, say 
AdaptiveLruBlockCache or something? (restructure the code to override certain 
methods like evict() etc). I think that helps with the following

While I want to find the time to come back and look some more at this, I think 
Bharath's suggestion here is a good one. If you can tease this apart from 
LruBlockCache, it will be easier to integrate this change. "Enabling" the 
feature is a bit more sensical (e.g. we just use AdaptiveLruBlockCache instead 
of changing the config property) and it removes (nearly?) all risk of changing 
runtime semantics for users who don't want to use this.

I'll give a soft message of support for committing this to master, given the 
analysis you've done (the google doc is great). I'll make a pass through to 
make sure it all makes sense to me, but I don't see any big impediments if we 
can decouple this from LruBlockCache.

ps: we _can_ leave the changes in LruBlockCache, but I think it will just 
require more buy-in.

> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg, 
> 1582801838065_rs_metrics_new.png, BC_LongRun.png, 
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png, 
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png, 
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png, 
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png, 
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png, 
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png, 
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png, 
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png, 
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png, 
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than 
> BlockChache (usual story in BigData). The idea - caching only part of DATA 
> blocks. It is good becouse LruBlockCache starts to work and save huge amount 
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a 
> high rate of evictions. In this case we can skip cache a block N and insted 
> cache the N+1th block. Anyway we would evict N block quite soon and that why 
> that skipping good for performance.
> ---
> Some information below isn't  actual
> ---
>  
>  
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to 
> read 3 blocks with offsets:
>  124
>  198
>  223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict 
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we 
> divide by modulus we got:
>  124 -> 24
>  198 -> 98
>  223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set 
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip 
> others. It means we will not try to handle the block 198 and save CPU for 
> other job. In the result - we put block 124, then put 223, evict 124 (3 
> actions).
> See the picture in attachment with test below. Requests per second is higher, 
> GC is lower.
>  
>  The key point of the code:
>  Added the parameter: *hbase.lru.cache.data.block.percent* which by default = 
> 100
>   
>  But if we set it 1-99, then will work the next logic:
>   
>   
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
> inMemory) {   
>   if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())      
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent) 
>   return;    
> ... 
> // the same code as usual
> }
> {code}
>  
&g

[jira] [Commented] (PHOENIX-6151) Switch phoenix-client to shade-by-default mode

2020-09-22 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200265#comment-17200265
 ] 

Josh Elser commented on PHOENIX-6151:
-

bq. Instead of shading packages explicitly, try the approach of shading 
everything, and only keeping unshaded the libraries that cannot be shaded.

+1. Giving you some more history. I remember when we were first trying to shade 
phoenix jars. We approached it with a mindset of "who knows what's going to 
break, let's shade as much as we can"

We had some pain in making tiny-changes as it broke for the "next user". Aside 
from the known painful dependencies (e.g. Hadoop and Protobuf), I can't think 
of an example where our shading _caused_ a problem. I think this "inversion" 
makes sense, given where we've come in the past years. Draft looks good so far!

> Switch phoenix-client to shade-by-default mode
> --
>
> Key: PHOENIX-6151
> URL: https://issues.apache.org/jira/browse/PHOENIX-6151
> Project: Phoenix
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 5.1.0, 4.16.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Phoenix-client contains Phoenix, plus most of a Hadoop+Hbase stack.
> We keep running into difficulties, where phoenix-client conflicts with user 
> code, or in the case of connectors with different components.
> Instead of shading packages explicitly, try the approach of shading 
> everything, and only keeping unshaded the libraries that cannot be shaded.
> This is the approach taken by Hadoop.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Remove Omid and Tephra server component from Phoenix distribution

2020-09-11 Thread Josh Elser


Sounds reasonable to me.

We have the same kind of thing going since phoenix-queryserver was moved 
out to its own repository. Maybe we can come up with some conventions as 
to how we "overlay" things so that Omid, PQS, and Tephra can all have 
some semblance of familiarity?


I think that would mean, daemons and such that Omid/Tephra need to run 
would be launched via an assembly in their respective code-bases. I 
guess the difference to PQS is that each of them have jar(s) that would 
also need to be added to the HBase RegionServer classpath?


On 9/11/20 11:09 AM, Istvan Toth wrote:

Hi!

We are currently shipping startup scripts and [some of] the JARs necessary
to start the server components of Omid and Tephra in the Phoenix
distribution.

The JARs for OMID are manually enumerated to be added to the distribution,
and have to be updated whenever the Omid dependencies change (and are
currently not enough to start Omid TSO), while the JARS for Tephra seem to
be completely missing.

I propose that we remove both from the Phoenix assembly, and instead
document (link to the corresponding project documentations) how to install
and run the server components for Omid and Tephra.

This would free us from the burden of having to duplicate and maintain the
Omid and Tephra server runtimes in our distribution, and save our users the
frustration when we fail to do so.

Looking forward to hearing your thoughts on this.

best regards
Istvan

[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x

2020-09-10 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193919#comment-17193919
 ] 

Josh Elser commented on PHOENIX-6067:
-

bq. Let me scope this out before I close the laptop for today.

Wait, where was this? I thought you meant you had a new patch here. I don't see 
any recent commits either. You talking about a PR somewhere?

Either way, getting the additional deactivations somewhere else is just fine.

IMO, let's get this in so we can let the stuff start to shake out. +1

> (5.x) Global Secondary Index Parity with 4.x
> 
>
> Key: PHOENIX-6067
> URL: https://issues.apache.org/jira/browse/PHOENIX-6067
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> A large number of indexing JIRAs were done for Phoenix 4.16 but were 
> originally unable to be ported to 5.x because of the lack of HBase 2.x 
> support for PHOENIX-5645, max lookback age. This was eventually fixed in 
> HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones 
> _were_ ported to 5.x, applying them all one-by-one and testing each 
> intermediate version would be impractical.
> This JIRA will import the 4.16 global index implementation into 5.1.0, then 
> fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x 
> compatibility shim between minor versions. (For example, max lookback age 
> features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and 
> incremental index validation will be deactivated when running against HBase 
> 2.1, because of the lack of HBASE-22710 to use raw Filters.) 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x

2020-09-10 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193912#comment-17193912
 ] 

Josh Elser commented on PHOENIX-6067:
-

bq. I'd rather tackle those as a separate JIRA because the same logic is in 4.x 
and would need to be fixed both places.

+1

bq. As for a new VerifyType coming from a newer client, I believe best practice 
is to upgrade Phoenix server first and clients second (this is enforced in the 
code for minor and major version upgrades), and I don't see a new VerifyType 
coming in on a patch release.

Great! Was more concerned about making sure there was some kind of precedence 
around this. Sounds great.

bq. I've pushed up additional changes deactivating under 2.1 a lot of tests 
using raw filters in some way (either view index rebuilds or incremental index 
rebuilds), including the IndexUpgradeToolTest you found.

Excellent! Let me scope this out before I close the laptop for today.

> (5.x) Global Secondary Index Parity with 4.x
> 
>
> Key: PHOENIX-6067
> URL: https://issues.apache.org/jira/browse/PHOENIX-6067
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> A large number of indexing JIRAs were done for Phoenix 4.16 but were 
> originally unable to be ported to 5.x because of the lack of HBase 2.x 
> support for PHOENIX-5645, max lookback age. This was eventually fixed in 
> HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones 
> _were_ ported to 5.x, applying them all one-by-one and testing each 
> intermediate version would be impractical.
> This JIRA will import the 4.16 global index implementation into 5.1.0, then 
> fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x 
> compatibility shim between minor versions. (For example, max lookback age 
> features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and 
> incremental index validation will be deactivated when running against HBase 
> 2.1, because of the lack of HBASE-22710 to use raw Filters.) 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release of phoenixdb 1.0.0 RC0

2020-09-10 Thread Josh Elser


+1 (binding)

* Xsums/sigs are good
* dist/dev/phoenix/KEYS was updated but dist/release/phoenix is the 
official KEYS file that you need to add your key to, Istvan

* dev-support's RAT check passed
* No unexpected files in the source release
* Was able to run tests against all Python versions in tox.ini
- I did run into issues with the QueryServerBasicsIT going into a 
fail-loop state (where HBase was stuck in ConnectionLoss), but I think 
that's just stressing the tiny JVM, not a product defect.


Great work Istvan!

On 9/9/20 5:09 AM, Istvan Toth wrote:

Hello Everyone,

This is a call for a vote on phoenixdb 1.0.0 RC0.

PhoenixDB is native Python driver for accessing Phoenix via Phoenix Query
Server.

This version is the first version to be released by Apache Phoenix Project,
and contains the following improvements compared to the previous 0.7
release by the original author.

- Replaced bundled requests_kerberos with request_gssapi library
- Use default SPNEGO Auth settings from request_gssapi
- Refactored authentication code
- Added support for specifying server certificate
- Added support for BASIC and DIGEST authentication
- Fixed HTTP error parsing
- Added transaction support
- Added list support
- Rewritten type handling
- Refactored test suite
- Removed shell example, as it was python2 only
- Updated documentation
- Added SQLAlchemy dialect
- Implemented Avatica Metadata API
- Misc fixes
- Licensing cleanup

The source release consists of the contents of the python-phoenixdb
directory of the phoenix-queryserver repository.

The source tarball, including signatures, digests, etc can be found at:

https://dist.apache.org/repos/dist/dev/phoenix/python-phoenixdb-1.0.0-rc0/src/python-phoenixdb-1.0.0-src.tar.gz
https://dist.apache.org/repos/dist/dev/phoenix/python-phoenixdb-1.0.0-rc0/src/python-phoenixdb-1.0.0-src.tar.gz.asc
https://dist.apache.org/repos/dist/dev/phoenix/python-phoenixdb-1.0.0-rc0/src/python-phoenixdb-1.0.0-src.tar.gz.sha256
https://dist.apache.org/repos/dist/dev/phoenix/python-phoenixdb-1.0.0-rc0/src/python-phoenixdb-1.0.0-src.tar.gz.sha512

Artifacts are signed with my "CODE SIGNING KEY":
825203A70405BC83AECF5F7D97351C1B794433C7

KEYS file available here:
https://dist.apache.org/repos/dist/dev/phoenix/KEYS


The hash and tag to be voted upon:
https://gitbox.apache.org/repos/asf?p=phoenix-queryserver.git;a=commit;h=3360154858e27cabe258dfb33b37ec31ed3bd210
https://gitbox.apache.org/repos/asf?p=phoenix-queryserver.git;a=tag;h=refs/tags/python-phoenixdb-1.0.0-rc0

Vote will be open for at least 72 hours. Please vote:

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
The Apache Phoenix Team

[jira] [Assigned] (HBASE-20598) Upgrade to JRuby 9.2

2020-09-10 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned HBASE-20598:
--

Assignee: Norbert Kalmár  (was: Jack Bearden)

> Upgrade to JRuby 9.2
> 
>
> Key: HBASE-20598
> URL: https://issues.apache.org/jira/browse/HBASE-20598
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, shell
>    Reporter: Josh Elser
>Assignee: Norbert Kalmár
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-20598.001.patch, HBASE-20598.002.patch
>
>
> [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we 
> can get ourselves onto that from our current 9.1 release line.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x



[ 
https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193097#comment-17193097
 ] 

Josh Elser commented on PHOENIX-6067:
-

{noformat}
[ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 
484.192 s <<< FAILURE! - in 
org.apache.phoenix.end2end.IndexToolForNonTxGlobalIndexIT
[ERROR] 
testEnableOutputLoggingForMaxLookback[mutable=true](org.apache.phoenix.end2end.IndexToolForNonTxGlobalIndexIT)
  Time elapsed: 10.66 s  <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at 
org.apache.phoenix.end2end.IndexToolForNonTxGlobalIndexIT.testEnableOutputLoggingForMaxLookback(IndexToolForNonTxGlobalIndexIT.java:1087)
{noformat}

IndexToolForNonTxGlobalIndexIT failed for me running {{mvn clean verify 
-Dhbase.profile=2.2}} with hbase 2.2.5 rebuilt locally against hadoop 3.1.2.

> (5.x) Global Secondary Index Parity with 4.x
> 
>
> Key: PHOENIX-6067
> URL: https://issues.apache.org/jira/browse/PHOENIX-6067
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> A large number of indexing JIRAs were done for Phoenix 4.16 but were 
> originally unable to be ported to 5.x because of the lack of HBase 2.x 
> support for PHOENIX-5645, max lookback age. This was eventually fixed in 
> HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones 
> _were_ ported to 5.x, applying them all one-by-one and testing each 
> intermediate version would be impractical.
> This JIRA will import the 4.16 global index implementation into 5.1.0, then 
> fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x 
> compatibility shim between minor versions. (For example, max lookback age 
> features aren't supported in 2.1 and 2.2 because they lack HBASE-24321, and 
> incremental index validation will be deactivated when running against HBase 
> 2.1, because of the lack of HBASE-22710 to use raw Filters.) 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6067) (5.x) Global Secondary Index Parity with 4.x



[ 
https://issues.apache.org/jira/browse/PHOENIX-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193088#comment-17193088
 ] 

Josh Elser commented on PHOENIX-6067:
-

Tried to get through as much of this as I could before my eyes glazed over :). 
Thanks for your hard work on this.

{code:java}
[INFO] Running org.apache.phoenix.index.VerifySingleIndexRowTest
[ERROR] 
testIfOptionsArePassedToIndexTool[IndexUpgradeToolTest_mutable={1}](org.apache.phoenix.index.IndexUpgradeToolTest)
  Time elapsed: 0.026 s  <<< ERROR!
java.lang.IllegalStateException: Can't do incremental index verification on 
this version of HBase because raw skip scan filters are not supported.
at 
org.apache.phoenix.mapreduce.index.IndexTool.parseOptions(IndexTool.java:388)
at 
org.apache.phoenix.index.IndexUpgradeToolTest.testIfOptionsArePassedToIndexTool(IndexUpgradeToolTest.java:124)
 {code}

Should we {{Assume}} this test off (rather than fail) when running against 
HBase 2.1?

{code:java}
-System.out.println(current);
+System.out.println(current + "column= " +
{code}

nit: missing a space

{code:java}
+if (!familyMap.containsKey(family)) {
+return false;
+}
+NavigableSet set = familyMap.get(family);
+if (set == null || set.isEmpty()) {
+return true;
+}
{code}

This is a little strange at a glance. If we don't have an entry in the map, we 
return false. But if there is a mapping (but the value is null) we return true.

{code:java}
+familyMap = scan.getFamilyMap();
+if (familyMap.isEmpty()) {
+familyMap = null;
+}
{code}
This code will null out `familyMap` but `isColumnIncluded(Cell)` in the same 
class isn't checking for a null `familyMap`. Looks like a bug waiting to happen?

{code:java}
+byte[] valueBytes = 
scan.getAttribute(BaseScannerRegionObserver.INDEX_REBUILD_VERIFY_TYPE);
+if (valueBytes != null) {
+verifyType = IndexTool.IndexVerifyType.fromValue(valueBytes);
+if (verifyType != IndexTool.IndexVerifyType.NONE) {
+verify = true;
+}
+}
{code}

What happens if the client gives a serialized value which we can't parse (e.g. 
newer client with a new IndexVerifyType enum value)? Or if they just give 
garbage data. I'm assuming that we'll fail gracefully back to the client (and 
not hit some retry loop).

{code:java}
 public void close() throws IOException {
 innerScanner.close();
+if (indexRowKeyforReadRepair != null) {
+hTableFactory.shutdown();
+indexHTable.close();
+return;
+}
{code}

Closing `hTableFactory` should be done in the parent GlobalIndexRegionScanner, 
not in child IndexRebuildRegionScanner? Looks like the htableFactory is never 
cleaned up in the parent, nor the `indexHTable`. Any reason we to not put a 
`close()` on GlobalIndexRegionScanner and have IndexRebuildRegionScanner call 
super.close()? Looks like the same could be applied to IndexerRegionScanner.

{code:java}
-keys.remove(keyRange);
+indexKeyToMutationMap.remove(result.getRow());
+//keys.remove(keyRange);
{code}

Remove commented code?

All in all, most of these are nit-picky or me worrying about edge-cases. What 
else do you all think should be done before this lands in master? Lars, you 
mentioned "your wringer" -- is that something we should run to make sure the 
implementation is reasonably solid before committing?

> (5.x) Global Secondary Index Parity with 4.x
> 
>
> Key: PHOENIX-6067
> URL: https://issues.apache.org/jira/browse/PHOENIX-6067
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-6067.v1.patch, PHOENIX-6067.v2.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> A large number of indexing JIRAs were done for Phoenix 4.16 but were 
> originally unable to be ported to 5.x because of the lack of HBase 2.x 
> support for PHOENIX-5645, max lookback age. This was eventually fixed in 
> HBASE-24321 and PHOENIX-5881. Because some JIRAs later than the missing ones 
> _were_ ported to 5.x, applying them all one-by-one and testing each 
> intermediate version would be impractical.
> This JIRA will import the 4.16 global index implementation into 5.1.0, then 
> fix HBase API usage to conform to HBase 2.x standards and Phoenix's HBase 2.x 
> compatibility shim between minor versions. (For example, max lookback age 
> features aren't supported in 2.1 and 2.2 because th

[jira] [Commented] (PHOENIX-6065) Add OWASP dependency check, and update the flagged direct dependencies



[ 
https://issues.apache.org/jira/browse/PHOENIX-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192996#comment-17192996
 ] 

Josh Elser commented on PHOENIX-6065:
-

Approved the two PR's. Want to make sure we have some kind of docs to cover how 
people can use these, Istvan?

Doesn't matter to me whether it's markdown in the repo itself or on the website.

> Add OWASP dependency check, and update the flagged direct dependencies
> --
>
> Key: PHOENIX-6065
> URL: https://issues.apache.org/jira/browse/PHOENIX-6065
> Project: Phoenix
>  Issue Type: Task
>  Components: connectors, core, queryserver
>Affects Versions: 5.1.0, 4.16.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Attachments: PHOENIX-6065.4.x.v1.patch, PHOENIX-6065.master.v1.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5881) Port MaxLookbackAge logic to 5.x



 [ 
https://issues.apache.org/jira/browse/PHOENIX-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated PHOENIX-5881:

Attachment: PHOENIX-5881.v4.patch

> Port MaxLookbackAge logic to 5.x
> 
>
> Key: PHOENIX-5881
> URL: https://issues.apache.org/jira/browse/PHOENIX-5881
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-5881.v1.patch, PHOENIX-5881.v2.patch, 
> PHOENIX-5881.v3.patch, PHOENIX-5881.v4.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> PHOENIX-5645 wasn't included in the master (5.x) branch because an HBase 2.x 
> change prevented the logic from being useful in the case of deletes, since 
> HBase 2.x no longer allows us to show deleted cells on an SCN query before 
> the point of deletion. Unfortunately, PHOENIX-5645 wound up requiring a lot 
> of follow-up work in the IndexTool and IndexScrutinyTool to deal with its 
> implications, and because of that, the 4.x and 5.x codebases around indexes 
> have diverged a good bit. 
> This work item is to get them back in sync, even though the behavior in the 
> face of deletes will be somewhat different, and so most likely some tests 
> will have to be changed or Ignored. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-5881) Port MaxLookbackAge logic to 5.x



[ 
https://issues.apache.org/jira/browse/PHOENIX-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192964#comment-17192964
 ] 

Josh Elser commented on PHOENIX-5881:
-

{quote}the logic is basically saying "if the timestamp is super-high, that 
means Tephra's running this Scan, so don't mess with how HBase treats delete 
markers."
{quote}
Thanks Geoffrey. The assumption here is that folks don't write timestamps in 
the future for HBase. Generally, I think that sounds appropriate, but it would 
be good to write that down somewhere. It looks like the 1.1x is about 5 years 
in the future, so probably low risk :)

I can't find the re-kicked job I tried to launch. Reattaching v3 again.

> Port MaxLookbackAge logic to 5.x
> 
>
> Key: PHOENIX-5881
> URL: https://issues.apache.org/jira/browse/PHOENIX-5881
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-5881.v1.patch, PHOENIX-5881.v2.patch, 
> PHOENIX-5881.v3.patch, PHOENIX-5881.v4.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> PHOENIX-5645 wasn't included in the master (5.x) branch because an HBase 2.x 
> change prevented the logic from being useful in the case of deletes, since 
> HBase 2.x no longer allows us to show deleted cells on an SCN query before 
> the point of deletion. Unfortunately, PHOENIX-5645 wound up requiring a lot 
> of follow-up work in the IndexTool and IndexScrutinyTool to deal with its 
> implications, and because of that, the 4.x and 5.x codebases around indexes 
> have diverged a good bit. 
> This work item is to get them back in sync, even though the behavior in the 
> face of deletes will be somewhat different, and so most likely some tests 
> will have to be changed or Ignored. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (CALCITE-2795) New Avatica version doesn't support "list" type in query

2020-09-04 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/CALCITE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated CALCITE-2795:

Priority: Critical  (was: Blocker)

> New Avatica version doesn't support "list" type in query
> 
>
> Key: CALCITE-2795
> URL: https://issues.apache.org/jira/browse/CALCITE-2795
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>Affects Versions: 1.14.0, 1.17.0, 1.18.0
>Reporter: Ayelet Morris
>Priority: Critical
>
> I created a simple POJO that has an id and a list and created a simple select 
> query from it, I received the following exception, it seems that in previous 
> versions (we used avatica 1.9 with calcite 1.11 before this upgrade) the list 
> type was detected as "OTHER" type and the query worked, now it is marked as a 
> Scalar but somehow finds its way to the "array" type of the types switch, 
> then when trying to parse the column TYPE it fails (it doesn't even try to 
> fetch the list itself)
> {noformat}
> java.lang.RuntimeException: exception while executing [SELECT * FROM 
> TypeWithList]
> at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1458)
>  at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1426)
>  at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returnsUnordered(CalciteAssert.java:1464)
>  at com.gigaspaces.jdbc.TypesTest.testSelectWithList(TypesTest.java:44)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>  at 
> org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
>  at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>  at 
> org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
>  at 
> org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
>  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>  at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
>  at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
>  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>  at 
> org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
>  at 
> org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
>  at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>  at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
>  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>  at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>  at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>  at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>  at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
>  Caused by: java.lang.RuntimeException: With materializationsEnabled=false, 
> limit=0
>  at org.apache.calcite.test.CalciteAssert.assertQuery(CalciteAssert.java:573)
>  at 
> org.apache.calcite.test.CalciteAssert$AssertQuery.returns(CalciteAssert.java:1450)
>  ... 33 more
>  Caus

[jira] [Commented] (PHOENIX-5032) add Apache Yetus to Phoenix

2020-08-29 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186958#comment-17186958
 ] 

Josh Elser commented on PHOENIX-5032:
-

Artem sent me a note offline saying that he has no objections but that he also 
has no starting point. Asked me to pass that along as he's away from his normal 
computer.

> add Apache Yetus to Phoenix
> ---
>
> Key: PHOENIX-5032
> URL: https://issues.apache.org/jira/browse/PHOENIX-5032
> Project: Phoenix
>  Issue Type: Task
>Reporter: Artem Ervits
>Assignee: Istvan Toth
>Priority: Major
>
> Spoke with [~elserj], will benefit greatly from Yetus.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6074) Let PQS Act As An Admin Tool Rest Endpoint

2020-08-28 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186678#comment-17186678
 ] 

Josh Elser commented on PHOENIX-6074:
-

{quote}I'm looking around to find a clean way to solve this issue, one idea 
could be to run the indexTool via command line which should spin up a new 
process and won't impact PQS directly. But the implementation can be a 
complicated and I need to think more about that.
{quote}
Yeah, that is a hard problem. Don't forget that you can't always assume to have 
YARN configuration on the PQS classpath to be able to submit a Tool. If you can 
guarantee that we're only submitting Jobs to YARN from PQS, I think I'm less 
worried (than if we were rebuilding an index table inside PQS itself).

I think as long as you are tracking the tasks that you run, we can at least 
understand what PQS has launched. You could make a nice HTML page that 
summarizes that PQS is running and on behalf of whom (e.g. who submitted the 
request for an IndexTool).
{quote}we can do something like SPENGO to make sure it is available only 
available to an authorized user.
{quote}
SPNEGO is just doing authentication in PQS. It does not solve the authorization 
problem.

> Let PQS Act As An Admin Tool Rest Endpoint
> --
>
> Key: PHOENIX-6074
> URL: https://issues.apache.org/jira/browse/PHOENIX-6074
> Project: Phoenix
>  Issue Type: Improvement
>  Components: queryserver
>Reporter: Mehdi Salarkia
>Assignee: Mehdi Salarkia
>Priority: Minor
>
> In our production environment we need to create a lot of indexes and use 
> indexTool to build the index also sometime use tools like indexScrutiny to 
> verify the health and status of indexes, etc.
> PQS can act as a REST API end point (proxy) that allows developers to call 
> and run commands that phoenix currently support via command line only:
>  * IndexTool
>  * IndexScrutiny
> Benefits:
>  # Allow developers to develop tools in their application to run and 
> integrate phoenix command line tools into their application without a human 
> intervention.
>  # Remove unnecessary access permission to production from non admins.
>  # Simplify the Index management (or any other future command line tool that 
> will be added to phoenix) and remove the possibility of human error, etc.
> I was looking at the implementation PHOENIX-5827 as an example. I think we 
> can simply define a new context and use that to trigger phoenix command line 
> tools from PQS and return the result (perhaps the MR job link,...) to the 
> client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6107) Discuss speed up of BaseQueryIT

2020-08-27 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186129#comment-17186129
 ] 

Josh Elser commented on PHOENIX-6107:
-

A belated +1 on doing this. Thanks for suggesting Lars and thanks for 
implementing Istvan.

> Discuss speed up of BaseQueryIT
> ---
>
> Key: PHOENIX-6107
> URL: https://issues.apache.org/jira/browse/PHOENIX-6107
> Project: Phoenix
>  Issue Type: Wish
>Reporter: Lars Hofhansl
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: 6107-master.txt, 6107-proposal.txt, 
> PHOENIX-6107-stoty.master.v1.patch, PHOENIX-6107.master.v2.patch
>
>
> All 14 tests derived from BaseQueryIT are some of the slowest we have.
> I noticed that all these tests run 7 times and each time create a table and 6 
> indexes.
> So just in terms of setup there are 14*7 = 98 tables created and 14*7*6 = 588 
> indexes created.
> It's not clear to me that that the runtime is justified, especially since we 
> have so many other index ITs.
> I think we can reduce this to run with one global index and one local index, 
> for a repeat of only 3 times, instead of 7. That would benefit all derived 
> test and shave of probably around 50% of the overall Phoenix test runtime.
> I.e. 14*3 = 42 tables, and 14*3*2 = 84 indexes.
> Could even go as far and test with no indexes here.
> Yes, it would potentially reduce coverage. Hence a discussion.
> Thoughts?
> (Marked as "Wish" so that we can discuss)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (CALCITE-4196) Avatica server responds with HTTP/401 prior to consuming all data written by client

2020-08-27 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/CALCITE-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved CALCITE-4196.
-
Resolution: Fixed

Fixed in f9837420cfabf88874eeb2c0a5b9642ebe2c2461. Thanks to Kevin (and 
Vladimir) for the reviews.

> Avatica server responds with HTTP/401 prior to consuming all data written by 
> client
> ---
>
> Key: CALCITE-4196
> URL: https://issues.apache.org/jira/browse/CALCITE-4196
> Project: Calcite
>  Issue Type: Bug
>  Components: avatica
>    Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: avatica-1.18.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> First off, big thanks to [~krisden] for pointing me to HIVE-22231 which was 
> the similar problem on the Hive side.
> Symptoms: the client, when sending a large HTTP request to the Avatica server 
> which is configured for SPNEGO authentication, e.g. an ExecuteBatchRequest 
> with 100's to 1000's of rows to execute, will receive an HTTP/401 response as 
> a part of the normal SPNEGO negotiation (described in 
> [https://tools.ietf.org/html/rfc4559#section-5]). The client will observe an 
> error similar to the following, indicate "Broken pipe".
> {noformat}
> 2020-08-24 17:21:54,512 DEBUG http.wire: http-outgoing-1 >> "[write] I/O 
> error: Broken pipe (Write failed)"
> 2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection: 
> http-outgoing-1: Close connection
> 2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection: 
> http-outgoing-1: Shutdown connection
> 2020-08-24 17:21:54,512 DEBUG execchain.MainClientExec: Connection discarded
> 2020-08-24 17:21:54,512 DEBUG conn.PoolingHttpClientConnectionManager: 
> Connection released: [id: 1][route: {}->http://avatica-server:8765][total 
> kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 100]
> 2020-08-24 17:21:54,512 INFO execchain.RetryExec: I/O exception 
> (java.net.SocketException) caught when processing request to 
> {}->http://avatica-server:8765: Broken pipe (Write failed)
> 2020-08-24 17:21:54,512 DEBUG execchain.RetryExec: Broken pipe (Write failed)
> java.net.SocketException: Broken pipe (Write failed)
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.conn.LoggingOutputStream.write(LoggingOutputStream.java:74)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
> at 
> org.apache.calcite.avatica.org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
> at 
> org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
> at 
> org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at 
> org.apache.calcite.avatica.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at 
> org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:129)
> at 
> org

[jira] [Created] (CALCITE-4196) Avatica server responds with HTTP/401 prior to consuming all data written by client

2020-08-26 Thread Josh Elser (Jira)

Josh Elser created CALCITE-4196:
---

 Summary: Avatica server responds with HTTP/401 prior to consuming 
all data written by client
 Key: CALCITE-4196
 URL: https://issues.apache.org/jira/browse/CALCITE-4196
 Project: Calcite
  Issue Type: Bug
  Components: avatica
Reporter: Josh Elser
Assignee: Josh Elser
 Fix For: avatica-1.18.0


First off, big thanks to [~krisden] for pointing me to HIVE-22231 which was the 
similar problem on the Hive side.

Symptoms: the client, when sending a large HTTP request to the Avatica server 
which is configured for SPNEGO authentication, e.g. an ExecuteBatchRequest with 
100's to 1000's of rows to execute, will receive an HTTP/401 response as a part 
of the normal SPNEGO negotiation (described in 
[https://tools.ietf.org/html/rfc4559#section-5]). The client will observe an 
error similar to the following, indicate "Broken pipe".
{noformat}
2020-08-24 17:21:54,512 DEBUG http.wire: http-outgoing-1 >> "[write] I/O error: 
Broken pipe (Write failed)"
2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection: 
http-outgoing-1: Close connection
2020-08-24 17:21:54,512 DEBUG conn.DefaultManagedHttpClientConnection: 
http-outgoing-1: Shutdown connection
2020-08-24 17:21:54,512 DEBUG execchain.MainClientExec: Connection discarded
2020-08-24 17:21:54,512 DEBUG conn.PoolingHttpClientConnectionManager: 
Connection released: [id: 1][route: {}->http://avatica-server:8765][total kept 
alive: 0; route allocated: 0 of 25; total allocated: 0 of 100]
2020-08-24 17:21:54,512 INFO execchain.RetryExec: I/O exception 
(java.net.SocketException) caught when processing request to 
{}->http://avatica-server:8765: Broken pipe (Write failed)
2020-08-24 17:21:54,512 DEBUG execchain.RetryExec: Broken pipe (Write failed)
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at 
org.apache.calcite.avatica.org.apache.http.impl.conn.LoggingOutputStream.write(LoggingOutputStream.java:74)
at 
org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at 
org.apache.calcite.avatica.org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
at 
org.apache.calcite.avatica.org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at 
org.apache.calcite.avatica.org.apache.http.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:112)
at 
org.apache.calcite.avatica.org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at 
org.apache.calcite.avatica.org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
at 
org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at 
org.apache.calcite.avatica.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at 
org.apache.calcite.avatica.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at 
org.apache.calcite.avatica.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at 
org.apache.calcite.avatica.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at 
org.apache.calcite.avatica.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at 
org.apache.calcite.avatica.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at 
org.apache.calcite.avatica.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at 
org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:129)
at 
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient$1.run(DoAsAvaticaHttpClient.java:39)
at 
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient$1.run(DoAsAvaticaHttpClient.java:37)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.calcite.avatica.remote.DoAsAvaticaHttpClient.send(DoAsAvaticaHttpClient.java:37)
at 
org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:44)
at 
org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:117)
at 
org.apache.calcite.avatica.remote.RemoteMeta$20.call(RemoteMeta.java:430)
at 
org.apache.calcite.avatica.remote.Remote

[jira] [Resolved] (PHOENIX-6084) Error in public package when builded

2020-08-25 Thread Josh Elser (Jira)



 [ 
https://issues.apache.org/jira/browse/PHOENIX-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved PHOENIX-6084.
-
Resolution: Invalid

It seems like your complaint is that the _Guava_ code {{InternetDomainName}} 
does not work correctly when it has been shaded into the Phoenix client jar.

First and most importantly, we do not package Guava for downstream users to 
leverage. It is present for Phoenix to use internally. You should not be 
relying on any bundled dependencies that come with Phoenix jars as they are 
subject to change across _any_ release.

Second, if this Guava code does not work for some reason which is due to how we 
shade Guava into the phoenix-client jar, I think we would generally be 
interested in fixing that. However, as Phoenix does not use anything from this 
class, it is likely not a priority of any of the regular developers. If you 
have interest in debugging why your example doesn't work and can create a fix, 
please open a new Jira issue which explains the issue, includes a change, and 
demonstrates how the change fixes your issue.

> Error in public package when builded
> 
>
> Key: PHOENIX-6084
> URL: https://issues.apache.org/jira/browse/PHOENIX-6084
> Project: Phoenix
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Alejandro Anadon
>Priority: Major
>
> It is very easy to reproduce. Just download the 5.0.0-HBase-2.0 
> ([https://phoenix.apache.org/download.html)] and make the next  (and very 
> simple) class in java using the "phoenix-5.0.0-HBase-2.0-client.jar" that is 
> inside of the package :
> -
> import com.google.common.net.InternetDomainName;
> public class TestBug{
>  public static void main(String[] args){
>  InternetDomainName domainName;
>  domainName=InternetDomainName.from("www.mydomain.com"); 
>  System.out.println(domainName.isUnderPublicSuffix());
>  
>  domainName=InternetDomainName.from("www.mydomain.net");
>  System.out.println(domainName.isUnderPublicSuffix());
>  
>  
> domainName=InternetDomainName.from("nonopublicsufix.org.apache.phoenix.shaded.net.al");
>  System.out.println(domainName.isUnderPublicSuffix());
>  
>  domainName=InternetDomainName.from("org.apache.phoenix.shaded.net.al");
>  System.out.println(domainName.isPublicSuffix());
>  }
>  }
> --
>  
> Expected result:
>  
> true   -> [www.mydomain.com|http://www.mydomain.com/] is under .com public 
> suffix
> true -> [www.mydomain.net|http://www.mydomain.com/] is under .net public 
> suffix
> false -> nonopublicsufix.org.apache.phoenix.shaded.net.al is under NON public 
> suffix org.apache.phoenix.shaded.net.al
> false -> org.apache.phoenix.shaded.net.al is NOT a public sufix
>  
> Actually result:
> true  -> ok
> false   -> Error
> true  -> Error
> true  -> Error
>  
> I found the error in jar (I had not the knowledge to find it in the source 
> code).
> The error is in the actually class "com.google.common.net.TldPatterns" that 
> is in the current jar  "phoenix-5.0.0-HBase-2.0-client.jar".
>  
> If you decompile it , you can see something like this:
>EXACT = ImmutableSet.of((Object)"ac", (Object)"com.ac", 
> (Object)"edu.ac", (Object)"gov.ac", 
> (Object)"org.apache.phoenix.shaded.net.ac", (Object)"mil.ac", (Object[])new 
> String[] { "org.ac", "ad", "nom.ad", "ae", "co.ae", 
> "org.apache.phoenix.shaded.net.ae", " 
> and it seems that all domains that should begings with "net" (p.e. "net.ac"), 
> when building the file (it is a generated file), it makes a mistake and add 
> "org.apache.phoenix.shaded" leaving it as "org.apache.phoenix.shaded.net.ac" 
> and this makes the error.
> I guest that as this is a generated class, there should be something bad when 
> building the complete package.
>  
> I didn't test it in other versions (I solved it building my own classes; but 
> it is not a clean solution).
>  
>  
> I know that this bug is NOT from the core of Phoenix; but in my case, I uses 
> the "phoenix-5.0.0-HBase-2.0-client.jar" for access to phenix, and the 
> "com.google.common.net.InternetDomainName" class for domains funtions.
> So I do not know if this is the right place to create the issue.
> (I take the opportunity to ask something:
> I have been able to conect phoenix using hibernate ORM satisfactorily .
> I created a "Dialect" and make 3 o 4 changes in hibernate core to change 
> "insert" to "upsert".
> It is large to splain how I did it and it is not correct to do it here, but 
> if somebody tell me the right place to do it, I'll do it).
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6064) Make Tephra support optional

2020-08-25 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184646#comment-17184646
 ] 

Josh Elser commented on PHOENIX-6064:
-

{quote}
I do not agree with that statement. At all!
Next time we find a hard problem with (say) local indexes we also just going to 
make them optional?
{quote}

We're not talking about a core feature of Phoenix. We're talking about a 
thirdparty library which the development community has disappeared. I don't 
think this is a fair analogy.

> Make Tephra support optional
> 
>
> Key: PHOENIX-6064
> URL: https://issues.apache.org/jira/browse/PHOENIX-6064
> Project: Phoenix
>  Issue Type: Improvement
>  Components: core, tephra
>Affects Versions: 5.1.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: PHOENIX-6064.master.v1.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Tephra has and old Guava dependency, that cannot be removed due to Twill 
> depending on it. Removing the Twill dependency from Tephra is possible, but 
> not trivial. 
> This Guava has CVEs, which will show up in static analysis tools, which will 
> cause some potential users not to adopt Phoenix.
> Provide an option to build Phoenix without Tephra, and its problematic 
> dependencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6074) Let PQS Act As An Admin Tool Rest Endpoint

2020-08-25 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184123#comment-17184123
 ] 

Josh Elser commented on PHOENIX-6074:
-

Restating, you're thinking about adding Phoenix-specific administrative stuff 
into PQS?

Being able to report on health via IndexScrutiny sounds reasonable to me. I'm 
worried about running IndexTool inside of PQS -- how would we be able to make 
sure that IndexTool tasks don't interrupt normal user queries?

I think the idea of this is simple and reasonable. I think the devil is in the 
details of how you will implement this, how clients will use it (How is the 
REST api defined?), how will you secure it? Lots of open questions :)

> Let PQS Act As An Admin Tool Rest Endpoint
> --
>
> Key: PHOENIX-6074
> URL: https://issues.apache.org/jira/browse/PHOENIX-6074
> Project: Phoenix
>  Issue Type: Improvement
>  Components: queryserver
>Reporter: Mehdi Salarkia
>Assignee: Mehdi Salarkia
>Priority: Minor
>
> In our production environment we need to create a lot of indexes and use 
> indexTool to build the index also sometime use tools like indexScrutiny to 
> verify the health and status of indexes, etc.
> PQS can act as a REST API end point (proxy) that allows developers to call 
> and run commands that phoenix currently support via command line only:
>  * IndexTool
>  * IndexScrutiny
> Benefits:
>  # Allow developers to develop tools in their application to run and 
> integrate phoenix command line tools into their application without a human 
> intervention.
>  # Remove unnecessary access permission to production from non admins.
>  # Simplify the Index management (or any other future command line tool that 
> will be added to phoenix) and remove the possibility of human error, etc.
> I was looking at the implementation PHOENIX-5827 as an example. I think we 
> can simply define a new context and use that to trigger phoenix command line 
> tools from PQS and return the result (perhaps the MR job link,...) to the 
> client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-5881) Port MaxLookbackAge logic to 5.x

2020-08-24 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183637#comment-17183637
 ] 

Josh Elser commented on PHOENIX-5881:
-

{code:java}
+if (!isMaxLookbackSupported) {
+return;
+} {code}
nit: JUnit has the {{Assume}} class which you could use to skip this test 
instead of letting it "pass".
{code:java}
+CANNOT_QUERY_TABLE_WITH_SCN_OLDER_THAN_MAX_LOOKBACK_AGE(538, "42915",
+"Cannot use SCN to look further back in the past beyond the configured 
max lookback age"),{code}
nit: Looks like indentation is off here.

{code:java}
+private boolean isTransactionalTimestamp(long ts) {
+//have to use the HBase edge manager because the Phoenix one is in 
phoenix-core
+return ts > (long) (EnvironmentEdgeManager.currentTime() * 1.1);
+}
{code}

This looks pretty quirky. What's going on here?

Let me see if I can rekick QA.

> Port MaxLookbackAge logic to 5.x
> 
>
> Key: PHOENIX-5881
> URL: https://issues.apache.org/jira/browse/PHOENIX-5881
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: PHOENIX-5881.v1.patch, PHOENIX-5881.v2.patch, 
> PHOENIX-5881.v3.patch
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> PHOENIX-5645 wasn't included in the master (5.x) branch because an HBase 2.x 
> change prevented the logic from being useful in the case of deletes, since 
> HBase 2.x no longer allows us to show deleted cells on an SCN query before 
> the point of deletion. Unfortunately, PHOENIX-5645 wound up requiring a lot 
> of follow-up work in the IndexTool and IndexScrutinyTool to deal with its 
> implications, and because of that, the 4.x and 5.x codebases around indexes 
> have diverged a good bit. 
> This work item is to get them back in sync, even though the behavior in the 
> face of deletes will be somewhat different, and so most likely some tests 
> will have to be changed or Ignored. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (PHOENIX-6098) IndexPredicateAnalyzer wrongly handles pushdown predicates and residual predicates

2020-08-24 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/PHOENIX-6098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183551#comment-17183551
 ] 

Josh Elser commented on PHOENIX-6098:
-

{quote}It's very hard to add tests to phoenix-hive module of 
phoenix-connectors, so I didn't add any tests
{quote}
Do you have an example SQL query which would could be validated by hand? I can 
see how the code is broken given the context you've shared, but i'm not sure 
how I would write such a query to demonstrate that Phoenix-Hive does the 
correct thing after your change :)

> IndexPredicateAnalyzer wrongly handles pushdown predicates and residual 
> predicates
> --
>
> Key: PHOENIX-6098
> URL: https://issues.apache.org/jira/browse/PHOENIX-6098
> Project: Phoenix
>  Issue Type: Bug
>  Components: hive-connector
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> Currently, the following code of IndexPredicateAnalyzer is assuming that 
> GenericUDFOPAnd always has 2 children nodes. I think this is wrong and it 
> leads wrong results:
> https://github.com/apache/phoenix-connectors/blob/5bd23ae2a0f70c3b3edf92a53780dafa643faf26/phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java#L346-L359



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24858) [hbase-thirdparty] Builds without a `clean` phase fail on hbase-shaded-jetty

2020-08-24 Thread Josh Elser (Jira)



[ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183540#comment-17183540
 ] 

Josh Elser commented on HBASE-24858:


Thanks Duo!!

> [hbase-thirdparty] Builds without a `clean` phase fail on hbase-shaded-jetty
> 
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
> Fix For: thirdparty-3.4.0
>
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24528) Improve balancer decision observability



[ 
https://issues.apache.org/jira/browse/HBASE-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176538#comment-17176538
 ] 

Josh Elser commented on HBASE-24528:


This JSON looks excellent to me.

> Improve balancer decision observability
> ---
>
> Key: HBASE-24528
> URL: https://issues.apache.org/jira/browse/HBASE-24528
> Project: HBase
>  Issue Type: New Feature
>  Components: Admin, Balancer, Operability, shell, UI
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2020-08-12 at 11.50.43 PM.png
>
>
> We provide detailed INFO and DEBUG level logging of balancer decision 
> factors, outcome, and reassignment planning, as well as similarly detailed 
> logging of the resulting assignment manager activity. However, an operator 
> may need to perform online and interactive observation, debugging, or 
> performance analysis of current balancer activity. Scraping and correlating 
> the many log lines resulting from a balancer execution is labor intensive and 
> has a lot of latency (order of ~minutes to acquire and index, order of 
> ~minutes to correlate). 
> The balancer should maintain a rolling window of history, e.g. the last 100 
> region move plans, or last 1000 region move plans submitted to the assignment 
> manager. This history should include decision factor details and weights and 
> costs. The rsgroups balancer may be able to provide fairly simple decision 
> factors, like for example "this table was reassigned to that regionserver 
> group". The underlying or vanilla stochastic balancer on the other hand, 
> after a walk over random assignment plans, will have considered a number of 
> cost functions with various inputs (locality, load, etc.) and multipliers, 
> including custom cost functions. We can devise an extensible class structure 
> that represents explanations for balancer decisions, and for each region move 
> plan that is actually submitted to the assignment manager, we can keep the 
> explanations of all relevant decision factors alongside the other details of 
> the assignment plan like the region name, and the source and destination 
> regionservers. 
> This history should be available via API for use by new shell commands and 
> admin UI widgets.
> The new shell commands and UI widgets can unpack the representation of 
> balancer decision components into human readable output. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Rename master branch to main

2020-08-12 Thread Josh Elser


+1 change it.

On 7/28/20 1:43 PM, Julian Hyde wrote:

I am in favor of renaming ‘master’ to ‘main’. To most people it doesn’t make 
any difference. To some, such as potential members currently outside the 
community, it makes the project more welcoming.

Very little effort or disruption is required. We’ve identified a potential 
source of friction, so let’s fix it and move on.

Julian


On Jul 28, 2020, at 10:31 AM, Michael Mior  wrote:

Hi all,

You can find some background on this discussion at the link below [0].
This is a topic that has come up regularly among D folks at the ASF.
The short summary is that the term "master" when referring to a git
branch is a reference to terminology related to slavery. I'm
suggesting main because this seems to be what the developer community
as a whole is gravitating towards. See for example, GitHub's public
roadmap [1] where there are plans to make this change.

I'm hoping that this discussion can be focused not on whether anyone
has been impacted by such terminology, but how we can move forward. I
personally believe that if a single person feels more welcome to
contribute because of the change, it's a win. I also don't think
making this change needs to be painful. (There are less than 20
relevant references to "master" in the Calcite code.) Apache Mahout
and I believe others have already made this change.

I think this is a relatively low impact change that can potentially
make us even more welcoming to new contributors, which is a benefit to
us all :)

[0] http://www.kapwing.com/blog/how-to-rename-your-master-branch-to-main-in-git/
[1] https://github.com/github/roadmap/issues/63

--
Michael Mior
mm...@apache.org

Re: EOL 1.3.x?

2020-08-12 Thread Josh Elser


It's gone both ways, if memory serves.

On 8/11/20 2:43 PM, Zach York wrote:

+1

Do we typically do a final release or can we just EOL as is?

On Tue, Aug 11, 2020 at 9:43 AM Geoffrey Jacoby  wrote:


+1 (non-binding)

Geoffrey

On Tue, Aug 11, 2020 at 8:19 AM Josh Elser  wrote:


Sounds good to me.

On 8/11/20 4:01 AM, 张铎(Duo Zhang) wrote:

The last release for 1.3.x is 2019.10.20, which means we do not have a
release for this release line for about 10 months.

Let's make it EOL and tell users to at least upgrade to 1.4.x?

Thanks.

[jira] [Updated] (CALCITE-4095) Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead of `SslContextFactory`



 [ 
https://issues.apache.org/jira/browse/CALCITE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated CALCITE-4095:

Component/s: avatica

> Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead 
> of `SslContextFactory`
> --
>
> Key: CALCITE-4095
> URL: https://issues.apache.org/jira/browse/CALCITE-4095
> Project: Calcite
>  Issue Type: Task
>  Components: avatica
>Reporter: Santiago Velasco
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: avatica-1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> `SslContextFactory` is deprecated at Jetty 9.4. This issue replaces it with 
> `SslContextFactory.Server`.
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.4.19.v20190610/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.3.24.v20180605/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  
> [https://github.com/apache/calcite-avatica/blob/e4711cb0ac8e72894c0c5b381892539673b348c2/server/src/main/java/org/apache/calcite/avatica/server/HttpServer.java#L816]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (CALCITE-4095) Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead of `SslContextFactory`



 [ 
https://issues.apache.org/jira/browse/CALCITE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated CALCITE-4095:

Fix Version/s: avatica-1.18.0

> Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead 
> of `SslContextFactory`
> --
>
> Key: CALCITE-4095
> URL: https://issues.apache.org/jira/browse/CALCITE-4095
> Project: Calcite
>  Issue Type: Task
>Reporter: Santiago Velasco
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: avatica-1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> `SslContextFactory` is deprecated at Jetty 9.4. This issue replaces it with 
> `SslContextFactory.Server`.
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.4.19.v20190610/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.3.24.v20180605/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  
> [https://github.com/apache/calcite-avatica/blob/e4711cb0ac8e72894c0c5b381892539673b348c2/server/src/main/java/org/apache/calcite/avatica/server/HttpServer.java#L816]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (CALCITE-4095) Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead of `SslContextFactory`



 [ 
https://issues.apache.org/jira/browse/CALCITE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved CALCITE-4095.
-
Resolution: Fixed

> Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead 
> of `SslContextFactory`
> --
>
> Key: CALCITE-4095
> URL: https://issues.apache.org/jira/browse/CALCITE-4095
> Project: Calcite
>  Issue Type: Task
>  Components: avatica
>Reporter: Santiago Velasco
>Assignee: Peter Somogyi
>Priority: Major
> Fix For: avatica-1.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> `SslContextFactory` is deprecated at Jetty 9.4. This issue replaces it with 
> `SslContextFactory.Server`.
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.4.19.v20190610/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.3.24.v20180605/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  
> [https://github.com/apache/calcite-avatica/blob/e4711cb0ac8e72894c0c5b381892539673b348c2/server/src/main/java/org/apache/calcite/avatica/server/HttpServer.java#L816]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CALCITE-4095) Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead of `SslContextFactory`



[ 
https://issues.apache.org/jira/browse/CALCITE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176505#comment-17176505
 ] 

Josh Elser commented on CALCITE-4095:
-

Looks like Peter gave us a PR for this. Reassigning to him. Sorry if you were 
already working on this [~Aron.tao]

> Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead 
> of `SslContextFactory`
> --
>
> Key: CALCITE-4095
> URL: https://issues.apache.org/jira/browse/CALCITE-4095
> Project: Calcite
>  Issue Type: Task
>Reporter: Santiago Velasco
>Assignee: Peter Somogyi
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> `SslContextFactory` is deprecated at Jetty 9.4. This issue replaces it with 
> `SslContextFactory.Server`.
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.4.19.v20190610/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.3.24.v20180605/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  
> [https://github.com/apache/calcite-avatica/blob/e4711cb0ac8e72894c0c5b381892539673b348c2/server/src/main/java/org/apache/calcite/avatica/server/HttpServer.java#L816]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (CALCITE-4095) Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead of `SslContextFactory`



 [ 
https://issues.apache.org/jira/browse/CALCITE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned CALCITE-4095:
---

Assignee: Peter Somogyi  (was: Jiatao Tao)

> Update Jetty to 9.4.31.v20200723 and use `SslContextFactory.Server` instead 
> of `SslContextFactory`
> --
>
> Key: CALCITE-4095
> URL: https://issues.apache.org/jira/browse/CALCITE-4095
> Project: Calcite
>  Issue Type: Task
>Reporter: Santiago Velasco
>Assignee: Peter Somogyi
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> `SslContextFactory` is deprecated at Jetty 9.4. This issue replaces it with 
> `SslContextFactory.Server`.
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.4.19.v20190610/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  - 
> [https://www.eclipse.org/jetty/javadoc/9.3.24.v20180605/org/eclipse/jetty/util/ssl/SslContextFactory.html]
>  
> [https://github.com/apache/calcite-avatica/blob/e4711cb0ac8e72894c0c5b381892539673b348c2/server/src/main/java/org/apache/calcite/avatica/server/HttpServer.java#L816]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24860) Bump copyright year in NOTICE



 [ 
https://issues.apache.org/jira/browse/HBASE-24860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-24860.

Resolution: Fixed

Pushed without review.

FYI [~zhangduo], in case you have to do a 3.4.0-rc2

> Bump copyright year in NOTICE
> -
>
> Key: HBASE-24860
> URL: https://issues.apache.org/jira/browse/HBASE-24860
> Project: HBase
>  Issue Type: Task
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Trivial
> Fix For: thirdparty-3.5.0
>
>
> year in NOTICE is 2018, should be 2020



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24860) Bump copyright year in NOTICE



 [ 
https://issues.apache.org/jira/browse/HBASE-24860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-24860.

Resolution: Fixed

Pushed without review.

FYI [~zhangduo], in case you have to do a 3.4.0-rc2

> Bump copyright year in NOTICE
> -
>
> Key: HBASE-24860
> URL: https://issues.apache.org/jira/browse/HBASE-24860
> Project: HBase
>  Issue Type: Task
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Trivial
> Fix For: thirdparty-3.5.0
>
>
> year in NOTICE is 2018, should be 2020



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty



 [ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-24858:
---
Fix Version/s: thirdparty-3.5.0

> Builds without a `clean` phase fail on hbase-shaded-jetty
> -
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
> Fix For: thirdparty-3.5.0
>
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24860) Bump copyright year in NOTICE



 [ 
https://issues.apache.org/jira/browse/HBASE-24860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-24860:
---
Fix Version/s: thirdparty-3.5.0

> Bump copyright year in NOTICE
> -
>
> Key: HBASE-24860
> URL: https://issues.apache.org/jira/browse/HBASE-24860
> Project: HBase
>  Issue Type: Task
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Trivial
> Fix For: thirdparty-3.5.0
>
>
> year in NOTICE is 2018, should be 2020



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24860) Bump copyright year in NOTICE

Josh Elser created HBASE-24860:
--

 Summary: Bump copyright year in NOTICE
 Key: HBASE-24860
 URL: https://issues.apache.org/jira/browse/HBASE-24860
 Project: HBase
  Issue Type: Task
  Components: thirdparty
Reporter: Josh Elser
Assignee: Josh Elser






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24860) Bump copyright year in NOTICE



 [ 
https://issues.apache.org/jira/browse/HBASE-24860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-24860:
---
Description: year in NOTICE is 2018, should be 2020

> Bump copyright year in NOTICE
> -
>
> Key: HBASE-24860
> URL: https://issues.apache.org/jira/browse/HBASE-24860
> Project: HBase
>  Issue Type: Task
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Trivial
>
> year in NOTICE is 2018, should be 2020



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24860) Bump copyright year in NOTICE

Josh Elser created HBASE-24860:
--

 Summary: Bump copyright year in NOTICE
 Key: HBASE-24860
 URL: https://issues.apache.org/jira/browse/HBASE-24860
 Project: HBase
  Issue Type: Task
  Components: thirdparty
Reporter: Josh Elser
Assignee: Josh Elser






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty



 [ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-24858 started by Josh Elser.
--
> Builds without a `clean` phase fail on hbase-shaded-jetty
> -
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty



[ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175799#comment-17175799
 ] 

Josh Elser commented on HBASE-24858:


Yup. That totally makes sense. On the first build (or a build after a clean), 
we have the empty jar for hbase-shaded-jetty. But, after the first build, we 
have the maven-shade-plugin replacing that artifact with the shaded one we've 
built. I think I know how to fix this.

> Builds without a `clean` phase fail on hbase-shaded-jetty
> -
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty



[ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175798#comment-17175798
 ] 

Josh Elser commented on HBASE-24858:


{noformat}
[WARNING] hbase-shaded-jetty-3.4.0.jar, jetty-http-9.4.31.v20200723.jar, 
jetty-io-9.4.31.v20200723.jar, jetty-jmx-9.4.31.v20200723.jar, 
jetty-security-9.4.31.v20200723.jar, jetty-server-9.4.31.v20200723.jar, 
jetty-servlet-9.4.31.v20200723.jar, jetty-util-9.4.31.v20200723.jar, 
jetty-util-ajax-9.4.31.v20200723.jar, jetty-webapp-9.4.31.v20200723.jar, 
jetty-xml-9.4.31.v20200723.jar define 1 overlapping resource: {noformat}
Seems like the problem is that, on the N+1th build, we will pick up the jar we 
built in the Nth build, and try to include that during shading. The 
ServicesResourceTransformer just happens to choke on this, whereas the shade 
plugin seems smart enough to not re-add the same classes again.

> Builds without a `clean` phase fail on hbase-shaded-jetty
> -
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty



[ 
https://issues.apache.org/jira/browse/HBASE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175789#comment-17175789
 ] 

Josh Elser commented on HBASE-24858:


Appears that the previous run will have both the unrelocated and the relocated 
services file in the sources jar
{noformat}
[DEBUG] Processing JAR 
/Users/jelser/projects/hbase-thirdparty.git/hbase-shaded-jetty/target/hbase-shaded-jetty-3.4.0-sources.jar
[DEBUG] Transforming 
META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
 using org.apache.maven.plugins.shade.resource.ServicesResourceTransformer
[DEBUG] Transforming 
META-INF/services/org.eclipse.jetty.http.HttpFieldPreEncoder using 
org.apache.maven.plugins.shade.resource.ServicesResourceTransformer
{noformat}

That would explain why a {{clean}} or a fresh-checkout don't suffer from this. 

> Builds without a `clean` phase fail on hbase-shaded-jetty
> -
>
> Key: HBASE-24858
> URL: https://issues.apache.org/jira/browse/HBASE-24858
> Project: HBase
>  Issue Type: Bug
>  Components: thirdparty
>    Reporter: Josh Elser
>    Assignee: Josh Elser
>Priority: Minor
>
> In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
> failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
> command.
> {noformat}
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  3.604 s
> [INFO] Finished at: 2020-08-11T14:38:45-04:00
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
> hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
> META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
>  -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {noformat}
> Poking around, it appears that this is actually to do with the creating of 
> the shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty

Josh Elser created HBASE-24858:
--

 Summary: Builds without a `clean` phase fail on hbase-shaded-jetty
 Key: HBASE-24858
 URL: https://issues.apache.org/jira/browse/HBASE-24858
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser


In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
command.

{noformat}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  3.604 s
[INFO] Finished at: 2020-08-11T14:38:45-04:00
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{noformat}

Poking around, it appears that this is actually to do with the creating of the 
shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty

Josh Elser created HBASE-24858:
--

 Summary: Builds without a `clean` phase fail on hbase-shaded-jetty
 Key: HBASE-24858
 URL: https://issues.apache.org/jira/browse/HBASE-24858
 Project: HBase
  Issue Type: Bug
Reporter: Josh Elser
Assignee: Josh Elser


In 3.4.0-rc1 that [~zhangduo] created, I noticed that builds of the project 
failed on hbase-shaded-jetty when I did not include {{clean}} in my build 
command.

{noformat}
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  3.604 s
[INFO] Finished at: 2020-08-11T14:38:45-04:00
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (default) on project 
hbase-shaded-jetty: Error creating shaded jar: duplicate entry: 
META-INF/services/org.apache.hbase.thirdparty.org.eclipse.jetty.http.HttpFieldPreEncoder
 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{noformat}

Poking around, it appears that this is actually to do with the creating of the 
shaded source jar.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HBASE-24858) Builds without a `clean` phase fail on hbase-shaded-jetty