Hi Amos, I tried to see if IMPALA-3643 <https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible for me. In the process I also ran into the "NoClassDefFoundError" error you saw and found out, that this happens if my local hadoop services (./testdata/bin/run-all.sh) isn't running.
Once I have the local hadoop services running I'm able to repro the issue in IMPALA-3643. The only thing I remember doing differently from the recommended ways of dev-machine setup is using Java 8. The Java 8 docs on LinkedHashSet <https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html> don't read like they'd change the ordering, however it doesn't say that the ordering has to be stable *between different versions of Java*. I will try to confirm that this issue does not repro with Java 7 when I have time. Can you check the Java version you're running. Mine is this: $ $JAVA_HOME/bin/java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Alex, could a difference in the implementation between Java 7 and 8 be a reasonable explanation for this? Cheers, Lars On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <amosb...@gmail.com> wrote: > > Thank you for helping me out :D > > > Thanks for keeping this discussion going. > > > > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <amosb...@gmail.com> wrote: > > > >> > >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <amosb...@gmail.com> > wrote: > >> > > >> >> > >> >> Hi there, > >> >> > >> >> I followed the wiki > >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+ > >> >> to+load+and+run+Impala+tests > >> >> carefully but still have some problems in my local env. > >> >> > >> >> 1. I need to manually execute "hdfs dfs -mkdir > >> /test-warehouse/emptytable" > >> >> to get rid of some fe test error. > >> >> > >> >> > >> > Ideally, you should not have to do this. Could you tell me what errors > >> you > >> > encountered? Sounds like there may be a test or data loading bug we > >> should > >> > fix. > >> > >> The error is : > >> > >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time > >> elapsed: 0.033 sec <<< FAILURE! > >> java.lang.AssertionError: got error: > >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does > >> not exist. > >> expected: > >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' > >> contains no visible files. > >> at org.junit.Assert.fail(Assert.java:88) > >> at org.junit.Assert.assertTrue(Assert.java:41) > >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError( > >> FrontendTestBase.java:312) > >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError( > >> FrontendTestBase.java:292) > >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData( > >> AnalyzeStmtsTest.java:2860) > >> > >> > > Do you have a table functional.emptytable? If yes, then what location is > > reported in "show create table"? > Query: show create table functional.emptytable > +-------------------------------------------------------------+ > | result | > +-------------------------------------------------------------+ > | CREATE EXTERNAL TABLE functional.emptytable ( | > | field STRING | > | ) | > | PARTITIONED BY ( | > | f2 INT | > | ) | > | STORED AS TEXTFILE | > | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' | > | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') | > +-------------------------------------------------------------+ > Fetched 1 row(s) in 5.51s > > > Does the directory exist in HDFS? > No. > > > > > You could try to manually reload the table and see if the directory is > > created: > > bin/load-data.py -f -w functional-query --table_names=emptytable > > --table_formats=text/none > After executing this command the directory appears. > > > > >>> > >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization > >> errors. > >> >> > >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time > >> >> elapsed: 0.333 sec <<< FAILURE! > >> >> java.lang.AssertionError: got error: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> default.nodb > >> >> expected: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> nodb.alltypes > >> >> at org.junit.Assert.fail(Assert.java:88) > >> >> at org.junit.Assert.assertTrue(Assert.java:41) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2220) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2203) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2197) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect( > >> >> AuthorizationTest.java:512) > >> >> > >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time > >> >> elapsed: 0.324 sec <<< FAILURE! > >> >> java.lang.AssertionError: got error: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> default.nodb > >> >> expected: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> nodb.alltypes > >> >> at org.junit.Assert.fail(Assert.java:88) > >> >> at org.junit.Assert.assertTrue(Assert.java:41) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2220) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2203) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> AuthorizationTest.java:2197) > >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect( > >> >> AuthorizationTest.java:512) > >> >> > >> >> > >> >> Results : > >> >> > >> >> Failed tests: > >> >> AuthorizationTest.TestSelect:512->AuthzError:2197-> > >> >> AuthzError:2203->AuthzError:2220 got error: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> default.nodb > >> >> expected: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> nodb.alltypes > >> >> AuthorizationTest.TestSelect:512->AuthzError:2197-> > >> >> AuthzError:2203->AuthzError:2220 got error: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> default.nodb > >> >> expected: > >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> nodb.alltypes > >> >> > >> >> > >> >> > >> > Strange. In this test, we register two authorization requests, and it > >> seems > >> > like those are not checked in the expected order. However, that should > >> not > >> > be possible because we store them in a LinkedHashSet. > >> > Could you dig into this a little further to see if you can figure out > why > >> > the order is wrong? > >> > > >> > This is where we register the authorization requests: > >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/ > >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544 > >> > > >> > This is where we check the authorization requests: > >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/ > >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391 > >> > > >> > > >> > >> I tried directly executing "select 1 from nodb.alltypes" in > >> impala-shell, leading to this error: > >> ERROR: AnalysisException: Could not resolve table reference: > >> 'nodb.alltypes' > >> > >> How can I reproduce the authorization tests in impala-shell so I can > >> debug it? > >> > >> > >> > > FYI, this is actually a known issue and may have something to do with the > > JRE version you are running: https://issues.cloudera.org/ > browse/IMPALA-3643 > > As far as I can tell the bug should be "impossible" because we use a > > LinkedHashSet, but maybe certain JREs do not properly honor the > guarantees. > > > > The AuthorizationTests in particular require a non-trivial setup, so I'd > > not recommend trying to debug via the Impala shell. > > > > I'd recommend debugging in one of these ways: > > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from the > FE > > directory. Attach debugger and break in TestSelect(). > > - Run the JUnit test from an IDE such as Eclipse and then debug the test. > > I'm afraid there is no easy way to just run that single query in our > > current test setup. You will need to run the whole suite, but you can > break > > TestSelect() or hack the code in various places to set useful > breakpoints. > > > > Hope that helps. > > I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest > fails with ClassNoDefine. > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec > <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest > initializationError(com.cloudera.impala.analysis.AuthorizationTest) Time > elapsed: 0.006 sec <<< ERROR! > java.lang.NoClassDefFoundError: Could not initialize class > com.cloudera.impala.analysis.AuthorizationTest > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > FrameworkMethod.java:50) > at org.junit.internal.runners.model.ReflectiveCallable.run( > ReflectiveCallable.java:12) > at org.junit.runners.model.FrameworkMethod.invokeExplosively( > FrameworkMethod.java:47) > at org.junit.runners.Parameterized.allParameters(Parameterized.java:280) > at org.junit.runners.Parameterized.<init>(Parameterized.java:248) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:62) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.junit.internal.builders.AnnotatedBuilder.buildRunner( > AnnotatedBuilder.java:104) > at org.junit.internal.builders.AnnotatedBuilder.runnerForClass( > AnnotatedBuilder.java:86) > at org.junit.runners.model.RunnerBuilder.safeRunnerForClass( > RunnerBuilder.java:59) > at org.junit.internal.builders.AllDefaultPossibilitiesBuilder > .runnerForClass(AllDefaultPossibilitiesBuilder.java:26) > at org.junit.runners.model.RunnerBuilder.safeRunnerForClass( > RunnerBuilder.java:59) > at org.junit.internal.requests.ClassRequest.getRunner( > ClassRequest.java:33) > at org.apache.maven.surefire.junit4.JUnit4Provider.execute( > JUnit4Provider.java:283) > at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun( > JUnit4Provider.java:173) > at org.apache.maven.surefire.junit4.JUnit4Provider. > executeTestSet(JUnit4Provider.java:153) > at org.apache.maven.surefire.junit4.JUnit4Provider.invoke( > JUnit4Provider.java:128) > at org.apache.maven.surefire.booter.ForkedBooter. > invokeProviderInSameClassLoader(ForkedBooter.java:203) > at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess( > ForkedBooter.java:155) > at org.apache.maven.surefire.booter.ForkedBooter.main( > ForkedBooter.java:103) > > > > >> > >> >> > >> >> > >> >> 3. For end-to-end tests, I encountered two kinds of errors > >> >> > >> >> a) connection refused. > >> >> > >> >> SET sync_ddl=False; > >> >> -- executing against localhost:21000 > >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE; > >> >> > >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_ > >> option: > >> >> {'disable_codegen': False, 'abort_on_error': 1, > 'exec_single_node_rows_ > >> threshold': > >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] > >> >> ___________________ > >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/ > >> >> bin/../infra/python/env/bin/python > >> >> metadata/test_load.py:77: in setup_method > >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i)) > >> >> util/hdfs_util.py:122: in copy > >> >> data = self.read_file(src) > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> pywebhdfs/webhdfs.py:183: > >> >> in read_file > >> >> response = requests.get(uri, allow_redirects=True) > >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: > in > >> get > >> >> return request('get', url, params=params, **kwargs) > >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: > in > >> >> request > >> >> response = session.request(method=method, url=url, **kwargs) > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> requests/sessions.py:465: > >> >> in request > >> >> resp = self.send(prep, **send_kwargs) > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> requests/sessions.py:594: > >> >> in send > >> >> history = [resp for resp in gen] if allow_redirects else [] > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> requests/sessions.py:196: > >> >> in resolve_redirects > >> >> **adapter_kwargs > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> requests/sessions.py:573: > >> >> in send > >> >> r = adapter.send(request, **kwargs) > >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> requests/adapters.py:415: > >> >> in send > >> >> raise ConnectionError(err, request=request) > >> >> E ConnectionError: ('Connection aborted.', error(111, 'Connection > >> >> refused')) > >> >> > >> >> > >> > The connection refused issue is very bizarre. One thing that I > noticed is > >> > that your Python does not seem to match what we use (Python 2.7.3). > >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if you > get > >> > the expected version into infra/python/env/local/bin? > >> > > >> > Alternatively, maybe there's a problem with your /etc/hosts? You can > try > >> > searching online for WebHdfs and /etc/hosts > >> > > >> > >> well, I find this 'find_py26.py' file under deps. Is it normal? > >> > > > > Yes, that's normal. That file looks Python 2.6 on your system but should > > not be relevant for running tests because we use the Python from our > > virtualenv and not the one on your system. > > > > What's your output when you run "impala-python --version". You should get > > 'Python 2.7.3". > > [amos@t450s tests]$ impala-python --version > Python 2.6.6 > > > Also, what's the Python version on your system? Our virtualenv will use > > Python 2.6 if your system has a Python < 2.6. > > You could try to upgrade your system Python and then > > re-run infra/python/bootstrap_virtualenv.py > > My system has python 2.6.6. > > > > > Still, theoretically Python 2.6 in the virtual env should work. I think > > it's more likely you are having a connection problem due to a > misconfigured > > /etc/hosts. > > > > Are you running the test from a shell that has bin/impala-config.sh and > > bin/set-classpath.sh sourced? > Yes. > > > > > To further debug this you could try to specify your namenode address when > > running the test to see whether it is somehow picking up a wrong address: > > cd tests > > ./run-tests.py metadata/test_load.py --namenode_http_address= > localhost:50070 > > > > And see if that works. > > Unfortunately, no. > > > > > > > > >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/ > >> download_requirements find_py26.py pip_download.py requirements.txt > >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_ > >> requirements > >> #!/bin/bash > >> > >> # Licensed to the Apache Software Foundation (ASF) under one > >> # or more contributor license agreements. See the NOTICE file > >> # distributed with this work for additional information > >> # regarding copyright ownership. The ASF licenses this file > >> # to you under the Apache License, Version 2.0 (the > >> # "License"); you may not use this file except in compliance > >> # with the License. You may obtain a copy of the License at > >> # > >> # http://www.apache.org/licenses/LICENSE-2.0 > >> # > >> # Unless required by applicable law or agreed to in writing, > >> # software distributed under the License is distributed on an > >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY > >> # KIND, either express or implied. See the License for the > >> # specific language governing permissions and limitations > >> # under the License. > >> > >> set -euo pipefail > >> > >> DIR="$(dirname "$0")" > >> > >> pushd "$DIR" > >> PY26="$(./find_py26.py)" > >> # Directly download packages listed in requirements.txt, but don't > install > >> them. > >> "$PY26" pip_download.py > >> # For virtualenv, other scripts rely on the .tar.gz package (not a .whl > >> package). > >> "$PY26" pip_download.py virtualenv 13.1.0 > >> # kudu-python is downloaded separately because pip install attempts to > >> execute a > >> # setup.py subcommand for kudu-python that can fail even if the download > >> succeeds. > >> "$PY26" pip_download.py kudu-python 0.1.1 > >> popd > >> > >> > >> > >> > b) stats not match > >> >> > >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/ > >> >> bin/../infra/python/env/bin/python > >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats > >> >> self.run_test_case('QueryTest/show-stats', vector, "functional") > >> >> common/impala_test_suite.py:342: in run_test_case > >> >> self.__verify_results_and_errors(vector, test_section, result, > >> use_db) > >> >> common/impala_test_suite.py:234: in __verify_results_and_errors > >> >> replace_filenames_with_placeholder) > >> >> common/test_result_verifier.py:398: in verify_raw_results > >> >> VERIFIER_MAP[verifier](expected, actual) > >> >> common/test_result_verifier.py:231: in verify_query_result_is_equal > >> >> assert expected_results == actual_results > >> >> > >> >> ... > >> >> > >> >> -- executing against localhost:21000 > >> >> show column stats alltypes_clone; > >> >> > >> >> MainThread: Comparing QueryTestResults (expected vs actual): > >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8 > >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1 > >> >> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING', > >> >> 736,-1,8,8 > >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8 > >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4 > >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4 > >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4 > >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4 > >> >> 'smallint_col','SMALLINT',10,-1,2,2 == > 'smallint_col','SMALLINT',10,- > >> 1,2,2 > >> >> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,- > 1,-1 > >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 != > >> 'timestamp_col','TIMESTAMP', > >> >> 7552,-1,16,16 > >> >> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1, > 1,1 > >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4 > >> >> > >> >> > >> >> Very strange. Can you do a compute stats on functional.alltypes and > >> > confirm that the NDV for timestamp_col are 7552 in your setup? > >> > >> Yes. > >> > > > > I'll need to ask around for help. I have no idea why this is happening. > > Thanks :) > > > > >> > >> > > >> > > >> > > >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help > is > >> >> much appreciated! > >> > > >> > > >> > > >> > > >> >> > >> >> Best regards, > >> >> Amos > >> >> > >> > >> > >
