Re: No builds on Phoenix-Master jenkins since Feb 19th?

[Istvan Toth]

Created https://issues.apache.org/jira/browse/PHOENIX-5843 to track this.

On Fri, Apr 10, 2020 at 2:47 AM la...@apache.org  wrote:

> +1 on removing test that are no longer relevant... As long as we keep test
> coverage.
>
>
>
>
>
>
> On Wednesday, April 8, 2020, 10:49:16 PM PDT, Istvan Toth <
> st...@apache.org> wrote:
>
>
>
>
>
> Yes, the tests are (and have been for some time) in a horrible shape for
> all branches and profiles.
>
> I've spent a few days on it, and now at least we are at the point where the
> test runs do not hang indefinitely, or get randomly killed, but
> there are still a lot of flaky tests, and unexplained random failures.
>
> Been too optimistic, I've just seen a recent 4.x run test hang. Fortunately
> it's a spurious test that I'm about to move to the PQS repo in a few
> minutes.
>
> We do get a few successful test runs on each branch / profile combination,
> but they are few and far between.
>
> We do have a lot of old and no longer relevant jobs in Apache. I have
> deactivated a few myself. I support removing them to reduce confusion.
> I think that the only relevant ones are:
>
>
> https://builds.apache.org/view/M-R/view/Phoenix/job/PreCommit-PHOENIX-Build/
> https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-4.x-matrix/
> https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master-matrix/
> and possibly
> https://builds.apache.org/view/M-R/view/Phoenix/job/PhoenixFindTestTimeout/
> though I haven't looked into the last one.
>
>
> István
>
> On Thu, Apr 9, 2020 at 1:30 AM la...@apache.org  wrote:
>
> > Thanks Josh.
> >
> > unfortunately
> >
> >
> https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master-matrix/
> >
> > has not passed a single time (at least for the past 15 runs)
> >
> > -- Lars
> >
> >
> > On Wednesday, April 8, 2020, 2:14:47 PM PDT, Josh Elser <
> els...@apache.org>
> > wrote:
> >
> >
> >
> >
> > It looks like the Phoenix-Master[1] build is now defunct after Istvan's
> > hbase profile work. New job over in [2] which is a matrix job for all of
> > HBase 2.0, 2.1, and 2.2.
> >
> > I think the question is whether or not we want to keep the old job
> > around? I think the answer is "no".
> >
> > [1] https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master/
> > [2]
> >
> https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master-matrix/
> >
> > On 4/8/20 2:00 PM, la...@apache.org wrote:
> > > Just looking at the Phoenix Jenkins jobs I noticed that was no build on
> > master since for 3 weeks.
> > > Is that in purpose? There were clearly changes on the master branch
> > since then.
> > >
> > > Cheers.
> > >
> > > -- Lars
> > >
> >
>


-- 
*István Tóth* | Staff Software Engineer
st...@cloudera.com 
[image: Cloudera] 
[image: Cloudera on Twitter]  [image:
Cloudera on Facebook]  [image: Cloudera
on LinkedIn] 

--

[jira] [Created] (PHOENIX-5843) Remove unused Phoenix jobs from ASF Jenkins

[Istvan Toth (Jira)]

Istvan Toth created PHOENIX-5843:


 Summary: Remove unused Phoenix jobs from ASF Jenkins
 Key: PHOENIX-5843
 URL: https://issues.apache.org/jira/browse/PHOENIX-5843
 Project: Phoenix
  Issue Type: Task
Reporter: Istvan Toth
Assignee: Istvan Toth


As discussed on the dev list, we have a lot of Jenkins jobs that are no longer 
relevant.
Remove them to reduce clutter.

Keep only the following jobs:

https://builds.apache.org/view/M-R/view/Phoenix/job/PreCommit-PHOENIX-Build/
https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-4.x-matrix/
https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master-matrix/
https://builds.apache.org/view/M-R/view/Phoenix/job/PhoenixFindTestTimeout/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (PHOENIX-5842) Code Coverage tool for Phoenix

[Sandeep Guggilam (Jira)]

Sandeep Guggilam created PHOENIX-5842:
-

 Summary: Code Coverage tool for Phoenix
 Key: PHOENIX-5842
 URL: https://issues.apache.org/jira/browse/PHOENIX-5842
 Project: Phoenix
  Issue Type: Improvement
Reporter: Sandeep Guggilam


Currently we don't have any code coverage tool for Phoenix. This is required 
for us to measure the test coverage and helps us in improving the test suite 
further till we reach may be 80% coverage

The test coverage results can also be integrated into the Hadoop QA run of the 
precommit build such that the reviewers could take a look at the report and see 
if the added code doesn't have enough coverage



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5841) When data columns get TTLed, we need inline index validation to publish a metric for this

[Gokcen Iskender (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-5841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gokcen Iskender updated PHOENIX-5841:
-
Description: 
We do index writes as full row writes. This means all columns keep get 
re-written to index and they have a current timestamp.

However, if there is a column that did not get updated for a long time (like 
Created_By type of columns that don't change) index inline validation marks 
these as "Index has extra columns". We need to publish an extra metric to 
distinguish these cases since they are expected to be not matching rows.

  was:
We do index writes as full row writes. This means all columns keep get 
re-written to index and they have a current timestamp.

However, if there is a column that did not get updated for a long time (like 
Created_By type of columns that don't change) the regular scan that 
IndexScrutiny uses doesn't return this column even though we might see it in 
the raw scan.

IndexScrutiny mark these rows as invalid because data table had TTL columns and 
index table have it.

Index inline validation also marks these as "Index has extra columns" if the 
column got TTLed.


> When data columns get TTLed, we need inline index validation to publish a 
> metric for this
> -
>
> Key: PHOENIX-5841
> URL: https://issues.apache.org/jira/browse/PHOENIX-5841
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Gokcen Iskender
>Priority: Major
>
> We do index writes as full row writes. This means all columns keep get 
> re-written to index and they have a current timestamp.
> However, if there is a column that did not get updated for a long time (like 
> Created_By type of columns that don't change) index inline validation marks 
> these as "Index has extra columns". We need to publish an extra metric to 
> distinguish these cases since they are expected to be not matching rows.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (PHOENIX-5841) When data columns get TTLed, we need inline index validation to publish a metric for this

[Gokcen Iskender (Jira)]

Gokcen Iskender created PHOENIX-5841:


 Summary: When data columns get TTLed, we need inline index 
validation to publish a metric for this
 Key: PHOENIX-5841
 URL: https://issues.apache.org/jira/browse/PHOENIX-5841
 Project: Phoenix
  Issue Type: Improvement
Reporter: Gokcen Iskender


We do index writes as full row writes. This means all columns keep get 
re-written to index and they have a current timestamp.

However, if there is a column that did not get updated for a long time (like 
Created_By type of columns that don't change) the regular scan that 
IndexScrutiny uses doesn't return this column even though we might see it in 
the raw scan.

IndexScrutiny mark these rows as invalid because data table had TTL columns and 
index table have it.

Index inline validation also marks these as "Index has extra columns" if the 
column got TTLed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5543) Implement show schemas / show tables SQL commands

[Bharath Vissapragada (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharath Vissapragada updated PHOENIX-5543:
--
Attachment: PHOENIX-5543.master.v2.patch

> Implement show schemas / show tables SQL commands
> -
>
> Key: PHOENIX-5543
> URL: https://issues.apache.org/jira/browse/PHOENIX-5543
> Project: Phoenix
>  Issue Type: Improvement
>Affects Versions: 4.15.0, 5.1.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Minor
> Attachments: PHOENIX-5543.master.v1.patch, 
> PHOENIX-5543.master.v2.patch
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently users rely on {{!tables}} and {{!schemas}} commands provided by 
> sqlline which pulls the information using the standard JDBC metadata calls 
> like {{getTables()}} and {{getSchemas()}}.
> Most other databases (like mysql[1,2]) implement these as first class SQL 
> commands that gives the user more flexibility in querying by adding necessary 
> filters and looking up for table information in specific schemas. The ask 
> here is to implement the following SQL commands.
> {noformat}
> SHOW SCHEMAS [LIKE '']
> SHOW TABLES [IN ] [LIKE '']
> {noformat}
> [1] https://dev.mysql.com/doc/refman/8.0/en/show-tables.html
> [2] https://dev.mysql.com/doc/refman/8.0/en/show-databases.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5804) Implement strong verification with -v ONLY option for old design of secondary indexes

[Swaroopa Kadam (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swaroopa Kadam updated PHOENIX-5804:

Attachment: (was: PHOENIX-5804.4.x.v4.patch)

> Implement strong verification with -v ONLY option for old design of secondary 
> indexes
> -
>
> Key: PHOENIX-5804
> URL: https://issues.apache.org/jira/browse/PHOENIX-5804
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Swaroopa Kadam
>Priority: Major
> Attachments: PHOENIX-5804.4.x.v1.patch, PHOENIX-5804.4.x.v3.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, with -v ONLY option we get weak verification i.e. we find out if 
> index row is present or not. It does not provide information on if the values 
> have mismatch when executed with old design.
> We attempt to provide scrutiny like implementation but way faster. The 
> verification will be done only one the latest version of the row.  
> This will help us in quantifying the success of new self-consistent secondary 
> indexes design. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (PHOENIX-5840) IndexTool inline verification should not fail with -v ONLY option

[Swaroopa Kadam (Jira)]

Swaroopa Kadam created PHOENIX-5840:
---

 Summary: IndexTool inline verification should not fail with -v 
ONLY option
 Key: PHOENIX-5840
 URL: https://issues.apache.org/jira/browse/PHOENIX-5840
 Project: Phoenix
  Issue Type: Improvement
Reporter: Swaroopa Kadam
Assignee: Swaroopa Kadam


IndexTool inline verification should not fail with -v ONLY option even if 
verification fails. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5789) try to standardize on a JSON library

[Richard Antal (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Richard Antal updated PHOENIX-5789:
---
Attachment: PHOENIX-5789.4.x.v1.patch

> try to standardize on a JSON library
> 
>
> Key: PHOENIX-5789
> URL: https://issues.apache.org/jira/browse/PHOENIX-5789
> Project: Phoenix
>  Issue Type: Improvement
>  Components: core
>Reporter: Istvan Toth
>Assignee: Richard Antal
>Priority: Minor
> Attachments: PHOENIX-5789.4.x.v1.patch, PHOENIX-5789.master.v1.patch, 
> PHOENIX-5789.master.v2.patch, PHOENIX-5789.master.v3.patch, 
> PHOENIX-5789.master.v4.patch
>
>
> Phoenix uses at least the following JSON libraries:
>  * gson
>  * jackson
>  * jettison
> Of these, only the jackson usage is performance critical, as it is used 
> during bulk loading. 
> Try to standardize on a single one to reduce dependency hell.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (PHOENIX-5837) Table Level Phoenix Metrics Implementation.

[vikas meka (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vikas meka updated PHOENIX-5837:

Description: currently GlobalClient Metrics provide aggregated information 
about the usage of the application in-terms of total no. of reads, total no. of 
writes, total number of failures, total no.of failures etc. In some scenarios 
these counters do not provide much useful insights into Phoenix usage pattern. 
The task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics 
which would be helpful in analyzing the reads/writes/deletes on a table.  (was: 
currently GlobalClient Metrics provide aggregated information about the usage 
of the application in-terms of total no. of reads, total no. of writes, total 
number of failures, total no.of failures etc. In some scenarios these counters 
do not provide much useful insights into Phoenix usage pattern.
 The task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics 
which would be helpful in analyzing the reads/writes/deletes on a table.)

> Table Level Phoenix Metrics Implementation.
> ---
>
> Key: PHOENIX-5837
> URL: https://issues.apache.org/jira/browse/PHOENIX-5837
> Project: Phoenix
>  Issue Type: Task
>Reporter: vikas meka
>Priority: Major
>  Labels: metric-collector, metrics
>
> currently GlobalClient Metrics provide aggregated information about the usage 
> of the application in-terms of total no. of reads, total no. of writes, total 
> number of failures, total no.of failures etc. In some scenarios these 
> counters do not provide much useful insights into Phoenix usage pattern. The 
> task is to add TableLevel Phoenix metrics on top of Global Phoenix metrics 
> which would be helpful in analyzing the reads/writes/deletes on a table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)