[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909736#comment-16909736
]
Quanlong Huang commented on IMPALA-8124:
Thanks, [~anuragmantri]! Can we close this JIRA?
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909478#comment-16909478
]
ASF subversion and git services commented on IMPALA-8124:
-
Commit 3948eda50bcae20c1080475571f95a1080b0820d in impala's branch
refs/heads/master from Anurag Mantripragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3948eda ]
IMPALA-8124: Modify TestWebPage::test_catalog to avoid flakiness.
This test scrapes the /catalog webpage for metrics. This can
occasionally run into a race condition when a table lock is
being held for functional*.* tables and the test tries to get
the metrics. This result in a failed assert for the metrics.
This change rewrites the test to create new tables with
unique_database fixture to avoid flakiness.
Also, seperated out an assert with AND into two different asserts
so it's easy to track which one failed.
Testing:
Ran multiple runs of this test locally while refreshing the table.
Change-Id: I341bf25baf8d9316a21a9eff860de84b33afd12f
Reviewed-on: http://gerrit.cloudera.org:8080/14075
Reviewed-by: Bharath Vissapragada
Tested-by: Impala Public Jenkins
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903416#comment-16903416
]
Anurag Mantripragada commented on IMPALA-8124:
--
Thanks [~Xiaomeng Zhang] for letting us know, I will take a look at it.
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903360#comment-16903360
]
Xiaomeng Zhang commented on IMPALA-8124:
Hit this issue again in
[webserver.test_web_pages.TestWebPage.test_catalog|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-data-load/712/testReport/junit/webserver.test_web_pages/TestWebPage/test_catalog/]
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823662#comment-16823662
]
bharath v commented on IMPALA-8124:
---
Yes, looks like it is racing with another metadata operation on the same table.
I can think of two ways to fix this.
1. Make the test serial with {{@pytest.mark.execute_serially}}
2. Add a few retries until it succeeds.
I'd prefer (1) to avoid flakiness in (2). Thoughts?
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823457#comment-16823457
]
Tim Armstrong commented on IMPALA-8124:
---
I actually hit this on a dockerised run and didn't link it back to this JIRA. I
did add some more output to the assert afterwards:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/53/
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
[jira] [Commented] (IMPALA-8124) Build failure: webserver/test_web_pages.py
[
https://issues.apache.org/jira/browse/IMPALA-8124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823371#comment-16823371
]
Michael Ho commented on IMPALA-8124:
Hit this again recently. I think the key here seems to be the following
statement in the message above:
{noformat}
lltypesare not available because the table is currently modified by another
operation.
{noformat}
Another instance shows the following:
{noformat}
Metrics for table functional.alltypesare not available because the table
is currently modified by another operation.
{noformat}
In essence, the test is trying to load the table metrics via the debug webpage
of the Catalogd but it can fail every now and then due to race with other
concurrent meta-data operations such as refreshes or invalidate metadata
{noformat}
-- 2019-04-22 05:13:54,125 INFO MainThread: Started query
2d4f1c8298f3dcce:3c822bc0
-- 2019-04-22 05:13:54,260 INFO MainThread: Starting new HTTP connection
(1): localhost
-- 2019-04-22 05:13:54,286 INFO MainThread: Starting new HTTP connection
(1): localhost
-- executing against localhost:21000
refresh functional.alltypes;
-- 2019-04-22 05:13:54,448 INFO MainThread: Started query
3c42f862fc08e584:25c0cc3c
-- 2019-04-22 05:13:54,456 INFO MainThread: Starting new HTTP connection
(1): localhost
-- executing against localhost:21000
refresh functional.alltypes;
-- 2019-04-22 05:13:54,597 INFO MainThread: Started query
574e64579d27d8ab:67d684e7
-- 2019-04-22 05:13:54,604 INFO MainThread: Starting new HTTP connection
(1): localhost
{noformat}
Anyhow, this seems to be a flaky test issue. Can some of the folks more
familiar with metadata please take a look and see how the test can be fixed ?
cc'ing [~bharathv], [~vihangk1]
> Build failure: webserver/test_web_pages.py
> --
>
> Key: IMPALA-8124
> URL: https://issues.apache.org/jira/browse/IMPALA-8124
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0, Impala 3.2.0
>Reporter: Paul Rogers
>Assignee: Lenisha Gandhi
>Priority: Major
> Labels: build-failure, flaky
> Fix For: Impala 3.3.0
>
>
> Build log output:
> {noformat}
> 04:20:18 === FAILURES
> ===
> 04:20:18 ___ TestWebPage.test_catalog
> ___
> 04:20:18 [gw4] linux2 -- Python 2.7.5
> /data/jenkins/workspace/impala-cdh6.x-core/repos/Impala/bin/../infra/python/env/bin/python
> 04:20:18 webserver/test_web_pages.py:265: in test_catalog
> 04:20:18 self.__test_table_metrics("functional", "alltypes",
> "total-file-size-bytes")
> 04:20:18 webserver/test_web_pages.py:283: in __test_table_metrics
> 04:20:18 "?name=%s.%s" % (db_name, tbl_name), metric,
> ports_to_test=self.CATALOG_TEST_PORT)
> 04:20:18 webserver/test_web_pages.py:144: in get_and_check_status
> 04:20:18 assert response.status_code == requests.codes.ok\
> 04:20:18 E AssertionError: Offending url:
> http://localhost:25020/table_metrics?name=functional.alltypes
> 04:20:18 E assert (200 == 200 and 'total-file-size-bytes' in
>
