[impala] branch master updated: IMPALA-8091: incremental improvements to NTP sync
This is an automated email from the ASF dual-hosted git repository.
mikeb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 6c0ec34 IMPALA-8091: incremental improvements to NTP sync
6c0ec34 is described below
commit 6c0ec348a7ed04857a4931e7670fc81aa819b2e1
Author: Michael Brown
AuthorDate: Tue Jan 22 14:59:14 2019 -0800
IMPALA-8091: incremental improvements to NTP sync
- Warn when ntp-wait is not available.
- Install ntp-perl for Redhat-based systems, which provides ntp-wait.
Debian-based systems already have ntp-wait as provided by ntp.
Change-Id: Ifc8cacf81765d132dd574993f8149231bdbb7bc6
Reviewed-on: http://gerrit.cloudera.org:8080/12253
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
bin/bootstrap_system.sh | 2 +-
testdata/cluster/admin | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/bin/bootstrap_system.sh b/bin/bootstrap_system.sh
index b9c41e2..28df210 100755
--- a/bin/bootstrap_system.sh
+++ b/bin/bootstrap_system.sh
@@ -148,7 +148,7 @@ fi
redhat sudo yum install -y curl gcc gcc-c++ git krb5-devel krb5-server
krb5-workstation \
-libevent-devel libffi-devel make ntp ntpdate openssl-devel cyrus-sasl \
+libevent-devel libffi-devel make ntp ntpdate ntp-perl openssl-devel
cyrus-sasl \
cyrus-sasl-gssapi cyrus-sasl-devel cyrus-sasl-plain \
python-devel python-setuptools postgresql postgresql-server \
wget vim-common nscd cmake lzo-devel fuse-devel snappy-devel
zlib-devel \
diff --git a/testdata/cluster/admin b/testdata/cluster/admin
index ef00b4e..22e7ab3 100755
--- a/testdata/cluster/admin
+++ b/testdata/cluster/admin
@@ -335,7 +335,9 @@ function start_cluster {
NTP_CANARY=pool.ntp.org
if ! ping -c 1 -w 5 "${NTP_CANARY}" >/dev/null 2>/dev/null; then
echo "WARNING: cannot reach ${NTP_CANARY}; ntp sync recommended for
Kudu"
-elif ntp-wait --help >/dev/null 2>/dev/null && ! ntp-wait -v; then
+elif ! ntp-wait --help >/dev/null 2>/dev/null; then
+ echo "WARNING: ntp-wait not installed; ntp sync recommended for Kudu"
+elif ! ntp-wait -v; then
echo "ntp-wait failed; cannot start kudu"
return 1
fi
[impala] branch asf-site updated: Remove stale meetup link
This is an automated email from the ASF dual-hosted git repository. jbapple pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/impala.git The following commit(s) were added to refs/heads/asf-site by this push: new 019a770 Remove stale meetup link 019a770 is described below commit 019a770a5c246cf0bfb62da11bf5de46d2e89e62 Author: stiga-huang AuthorDate: Wed Jan 23 06:34:49 2019 -0800 Remove stale meetup link Change-Id: Ia109b11e0cb65315b1bf3fa1e07540fd48c89ca3 Reviewed-on: http://gerrit.cloudera.org:8080/12256 Reviewed-by: Fredy Wijaya Tested-by: Jim Apple --- community.html | 1 - 1 file changed, 1 deletion(-) diff --git a/community.html b/community.html index 291caf1..33095ae 100644 --- a/community.html +++ b/community.html @@ -151,7 +151,6 @@ Issues: https://issues.apache.org/jira/browse/IMPALA";>https://issues.apache.org/jira/browse/IMPALA -Meetup: http://www.meetup.com/Bay-Area-Impala-Users-Group/";>http://www.meetup.com/Bay-Area-Impala-Users-Group/ Twitter: https://twitter.com/apacheimpala";>@ApacheImpala Stack Overflow: http://stackoverflow.com/search?q=impala";>http://stackoverflow.com/search?q=impala Quora:
[impala] 04/04: IMPALA-7979: Enhance decoders to support value-skipping
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 969dea84f41cfccf863b14904958a5d2be91983f
Author: Zoltan Borok-Nagy
AuthorDate: Mon Jan 7 16:50:36 2019 +0100
IMPALA-7979: Enhance decoders to support value-skipping
This commit adds value-skipping functionality to the decoders.
Value-skipping is useful when we know that we won't need the
following N values, so instead of decoding and dropping them, we
can just "jump" through them.
This feature is a prerequisite for Parquet page skipping (IMPALA-5843).
I added backend tests for all the decoders. Backed tests related to
bitpacking are moved to the newly created bit-stream-utils-test.
Change-Id: Ib848f1bd71735fe84e8064daf700417b32589f57
Reviewed-on: http://gerrit.cloudera.org:8080/12172
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
be/src/exec/parquet/CMakeLists.txt | 1 +
be/src/exec/parquet/parquet-bool-decoder-test.cc | 108
be/src/exec/parquet/parquet-bool-decoder.cc | 28 ++
be/src/exec/parquet/parquet-bool-decoder.h | 4 +
be/src/util/CMakeLists.txt | 1 +
be/src/util/bit-stream-utils-test.cc | 207 ++
be/src/util/bit-stream-utils.h | 5 +
be/src/util/bit-stream-utils.inline.h| 12 +
be/src/util/dict-encoding.h | 20 ++
be/src/util/dict-test.cc | 92 ++-
be/src/util/rle-encoding.h | 87 ++
be/src/util/rle-test.cc | 330 ++-
12 files changed, 765 insertions(+), 130 deletions(-)
diff --git a/be/src/exec/parquet/CMakeLists.txt
b/be/src/exec/parquet/CMakeLists.txt
index a3ac2de..7bf24e2 100644
--- a/be/src/exec/parquet/CMakeLists.txt
+++ b/be/src/exec/parquet/CMakeLists.txt
@@ -38,6 +38,7 @@ add_library(Parquet
add_dependencies(Parquet gen-deps)
+ADD_BE_LSAN_TEST(parquet-bool-decoder-test)
ADD_BE_LSAN_TEST(parquet-plain-test)
ADD_BE_LSAN_TEST(parquet-version-test)
ADD_BE_LSAN_TEST(hdfs-parquet-scanner-test)
diff --git a/be/src/exec/parquet/parquet-bool-decoder-test.cc
b/be/src/exec/parquet/parquet-bool-decoder-test.cc
new file mode 100644
index 000..a414546
--- /dev/null
+++ b/be/src/exec/parquet/parquet-bool-decoder-test.cc
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/parquet/parquet-bool-decoder.h"
+#include "testutil/gtest-util.h"
+
+#include
+
+#include "common/names.h"
+
+namespace impala {
+
+void EncodeData(const vector& data, parquet::Encoding::type encoding,
+uint8_t* buffer, int buffer_len) {
+ if (encoding == parquet::Encoding::PLAIN) {
+BitWriter writer(buffer, buffer_len);
+for (int b : data) {
+ ASSERT_TRUE(writer.PutValue(b, 1));
+}
+writer.Flush();
+ } else {
+DCHECK(encoding == parquet::Encoding::RLE);
+// We need to pass 'buffer + 4' because the ParquetBoolDecoder ignores the
first 4
+// bytes (in Parquet RLE, the first 4 bytes are used to encode the data
size).
+RleEncoder encoder(buffer + 4, buffer_len - 4, 1, 8);
+for (int b : data) {
+ ASSERT_TRUE(encoder.Put(b));
+}
+encoder.Flush();
+ }
+}
+
+void TestSkipping(parquet::Encoding::type encoding, uint8_t* encoded_data,
+int encoded_data_len, const vector& expected_data, int skip_at,
+int skip_count) {
+ using namespace parquet;
+ ParquetBoolDecoder decoder;
+ decoder.SetData(encoding, encoded_data, encoded_data_len);
+
+ auto Decode = [encoding, &decoder]() {
+bool b;
+if (encoding == Encoding::PLAIN) {
+ EXPECT_TRUE(decoder.DecodeValue(&b));
+} else {
+ EXPECT_TRUE(decoder.DecodeValue(&b));
+}
+return b;
+ };
+
+ for (int i = 0; i < skip_at && i < expected_data.size(); ++i) {
+EXPECT_EQ(Decode(), expected_data[i]) << i;
+ }
+ decoder.SkipValues(skip_count);
+ for (int i = skip_at + skip_count; i < expected_data.size(); ++i) {
+EXPECT_EQ(Decode(), expected_data[i]) << i;
+ }
+}
+
+TEST(ParquetBoolD
[impala] branch master updated (6fa3547 -> 969dea8)
This is an automated email from the ASF dual-hosted git repository. tarmstrong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/impala.git. from 6fa3547 IMPALA-5847: Fix incorrect use of SET in .test files new 561158b IMPALA-3323: Fix unrecognizable shell option when --config_file is specified new 9105c5c IMPALA-8073: Fix FE tests that leak HMS connections new d1f2c1a IMPALA-7941: part 1: detect cgroups memory limit new 969dea8 IMPALA-7979: Enhance decoders to support value-skipping The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/common/init.cc | 2 + be/src/exec/parquet/CMakeLists.txt | 1 + be/src/exec/parquet/parquet-bool-decoder-test.cc | 108 +++ be/src/exec/parquet/parquet-bool-decoder.cc| 28 ++ be/src/exec/parquet/parquet-bool-decoder.h | 4 + be/src/util/CMakeLists.txt | 2 + be/src/util/bit-stream-utils-test.cc | 207 + be/src/util/bit-stream-utils.h | 5 + be/src/util/bit-stream-utils.inline.h | 12 + be/src/util/cgroup-util.cc | 177 +++ be/src/util/cgroup-util.h | 62 be/src/util/default-path-handlers.cc | 27 +- be/src/util/dict-encoding.h| 20 ++ be/src/util/dict-test.cc | 92 +- be/src/util/proc-info-test.cc | 21 ++ be/src/util/process-state-info.cc | 34 +-- be/src/util/process-state-info.h | 13 +- be/src/util/rle-encoding.h | 87 ++ be/src/util/rle-test.cc| 330 + .../java/org/apache/impala/catalog/Catalog.java| 4 +- .../org/apache/impala/analysis/AuditingTest.java | 19 +- .../apache/impala/analysis/AuthorizationTest.java | 149 +- .../impala/analysis/StmtMetadataLoaderTest.java| 26 +- .../catalog/CatalogObjectToFromThriftTest.java | 1 - .../org/apache/impala/catalog/CatalogTest.java | 4 + .../catalog/CatalogdTableInvalidatorTest.java | 6 +- .../impala/catalog/PartialCatalogInfoTest.java | 4 + .../org/apache/impala/common/FrontendTestBase.java | 1 + .../apache/impala/testutil/BlockIdGenerator.java | 53 ++-- .../apache/impala/testutil/ImpaladTestCatalog.java | 6 + .../org/apache/impala/util/SentryProxyTest.java| 131 shell/impala_shell.py | 3 +- shell/option_parser.py | 30 +- tests/shell/good_impalarc | 1 + tests/shell/test_shell_commandline.py | 1 + www/root.tmpl | 4 + 36 files changed, 1292 insertions(+), 383 deletions(-) create mode 100644 be/src/exec/parquet/parquet-bool-decoder-test.cc create mode 100644 be/src/util/bit-stream-utils-test.cc create mode 100644 be/src/util/cgroup-util.cc create mode 100644 be/src/util/cgroup-util.h
[impala] 02/04: IMPALA-8073: Fix FE tests that leak HMS connections
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 9105c5c5427dd0479caf280b5754b31845df5fc2
Author: Fredy Wijaya
AuthorDate: Fri Jan 18 09:38:23 2019 -0800
IMPALA-8073: Fix FE tests that leak HMS connections
This patch fixes FE tests that leak the HMS connections. I was to
reproduce the issue by in SentryTestProxy by looping the test multiple
times.
Testing:
- Ran SentryProxyTest in a loop without any issue.
- Ran all FE tests
Change-Id: I56930bc5f7a7bda4db11d4447461f907b1017b91
Reviewed-on: http://gerrit.cloudera.org:8080/12241
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
.../java/org/apache/impala/catalog/Catalog.java| 4 +-
.../org/apache/impala/analysis/AuditingTest.java | 19 +--
.../apache/impala/analysis/AuthorizationTest.java | 149 +++--
.../impala/analysis/StmtMetadataLoaderTest.java| 26 ++--
.../catalog/CatalogObjectToFromThriftTest.java | 1 -
.../org/apache/impala/catalog/CatalogTest.java | 4 +
.../catalog/CatalogdTableInvalidatorTest.java | 6 +-
.../impala/catalog/PartialCatalogInfoTest.java | 4 +
.../org/apache/impala/common/FrontendTestBase.java | 1 +
.../apache/impala/testutil/BlockIdGenerator.java | 53
.../apache/impala/testutil/ImpaladTestCatalog.java | 6 +
.../org/apache/impala/util/SentryProxyTest.java| 131 +-
12 files changed, 215 insertions(+), 189 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/Catalog.java
b/fe/src/main/java/org/apache/impala/catalog/Catalog.java
index 1f46882..7a3aa93 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Catalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Catalog.java
@@ -57,7 +57,7 @@ import com.google.common.base.Preconditions;
* database that the user cannot modify.
* Builtins are populated on startup in initBuiltins().
*/
-public abstract class Catalog {
+public abstract class Catalog implements AutoCloseable {
// Initial catalog version and ID.
public final static long INITIAL_CATALOG_VERSION = 0L;
public static final TUniqueId INITIAL_CATALOG_SERVICE_ID = new TUniqueId(0L,
0L);
@@ -322,9 +322,9 @@ public abstract class Catalog {
* Release the Hive Meta Store Client resources. Can be called multiple times
* (additional calls will be no-ops).
*/
+ @Override
public void close() { metaStoreClientPool_.close(); }
-
/**
* Returns a managed meta store client from the client connection pool.
*/
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
b/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
index 836020b..484ae8e 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuditingTest.java
@@ -370,15 +370,16 @@ public class AuditingTest extends FrontendTestBase {
// an AuthorizationError
AuthorizationConfig config =
AuthorizationConfig.createHadoopGroupAuthConfig(
"server1", "/does/not/exist", "");
-ImpaladCatalog catalog = new ImpaladTestCatalog(config);
-Frontend fe = new Frontend(config, catalog);
-AnalysisContext analysisCtx = createAnalysisCtx(config);
-// We should get an audit event even when an authorization failure occurs.
-try {
- parseAndAnalyze("create table foo_does_not_exist(i int)", analysisCtx,
fe);
- Assert.fail("Expected AuthorizationException");
-} catch (AuthorizationException e) {
- Assert.assertEquals(1,
analysisCtx.getAnalyzer().getAccessEvents().size());
+try (ImpaladCatalog catalog = new ImpaladTestCatalog(config)) {
+ Frontend fe = new Frontend(config, catalog);
+ AnalysisContext analysisCtx = createAnalysisCtx(config);
+ // We should get an audit event even when an authorization failure
occurs.
+ try {
+parseAndAnalyze("create table foo_does_not_exist(i int)", analysisCtx,
fe);
+Assert.fail("Expected AuthorizationException");
+ } catch (AuthorizationException e) {
+Assert.assertEquals(1,
analysisCtx.getAnalyzer().getAccessEvents().size());
+ }
}
}
diff --git a/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
b/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
index 7e1214e..9de90d6 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
@@ -842,58 +842,59 @@ public class AuthorizationTest extends FrontendTestBase {
AuthorizationConfig authzConfig = new AuthorizationConfig("server1",
AUTHZ_POLICY_FILE, "",
LocalGroupResourceAuthorizationProvider.class.getName());
-ImpaladCatalog catalog = new ImpaladTestCatalog(authzConfig);
-
-// Thi
[impala] 03/04: IMPALA-7941: part 1: detect cgroups memory limit
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit d1f2c1a2269604efb9a498c0a0137bb4ea4eff07
Author: Tim Armstrong
AuthorDate: Fri Dec 21 07:30:43 2018 -0800
IMPALA-7941: part 1: detect cgroups memory limit
This adds the logic to detect the cgroups memory limit,
but does not use it to set the process memory limit yet.
The information obtained is logged at startup and accessible
through the Web UI.
The patch boils down to reading from the appropriate
places in the filesystem. The main complication is that paths need to be
translated to point to the right cgroup node inside the container.
This deletes some useless cgroup logic from ProcessStateInfo that
printed whatever happened to be the last cgroup in a file.
Testing:
Added a unit test to check that the code could parse the cgroups
hierarchy ok and return a positive value.
Tested on CentOS6 and CentOS7.
Change-Id: Ic0f5ed0e94511361bf5605822c0874c16c45ffa9
Reviewed-on: http://gerrit.cloudera.org:8080/12120
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
be/src/common/init.cc| 2 +
be/src/util/CMakeLists.txt | 1 +
be/src/util/cgroup-util.cc | 177 +++
be/src/util/cgroup-util.h| 62
be/src/util/default-path-handlers.cc | 27 +++---
be/src/util/proc-info-test.cc| 21 +
be/src/util/process-state-info.cc| 34 ++-
be/src/util/process-state-info.h | 13 +--
www/root.tmpl| 4 +
9 files changed, 289 insertions(+), 52 deletions(-)
diff --git a/be/src/common/init.cc b/be/src/common/init.cc
index fcd6971..a9d00ce 100644
--- a/be/src/common/init.cc
+++ b/be/src/common/init.cc
@@ -38,6 +38,7 @@
#include "runtime/hdfs-fs-cache.h"
#include "runtime/lib-cache.h"
#include "runtime/mem-tracker.h"
+#include "util/cgroup-util.h"
#include "util/cpu-info.h"
#include "util/debug-util.h"
#include "util/decimal-util.h"
@@ -282,6 +283,7 @@ void impala::InitCommonRuntime(int argc, char** argv, bool
init_jvm,
LOG(INFO) << DiskInfo::DebugString();
LOG(INFO) << MemInfo::DebugString();
LOG(INFO) << OsInfo::DebugString();
+ LOG(INFO) << CGroupUtil::DebugString();
LOG(INFO) << "Process ID: " << getpid();
LOG(INFO) << "Default AES cipher mode for spill-to-disk: "
<<
EncryptionKey::ModeToString(EncryptionKey::GetSupportedDefaultMode());
diff --git a/be/src/util/CMakeLists.txt b/be/src/util/CMakeLists.txt
index 62148f4..1405dd8 100644
--- a/be/src/util/CMakeLists.txt
+++ b/be/src/util/CMakeLists.txt
@@ -35,6 +35,7 @@ add_library(Util
bit-util.cc
bloom-filter.cc
bloom-filter-ir.cc
+ cgroup-util.cc
coding-util.cc
codec.cc
common-metrics.cc
diff --git a/be/src/util/cgroup-util.cc b/be/src/util/cgroup-util.cc
new file mode 100644
index 000..412ec55
--- /dev/null
+++ b/be/src/util/cgroup-util.cc
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "util/cgroup-util.h"
+
+#include
+#include
+#include
+#include
+
+#include
+
+#include "gutil/strings/escaping.h"
+#include "gutil/strings/substitute.h"
+#include "util/error-util.h"
+#include "util/string-parser.h"
+
+#include "common/names.h"
+
+using boost::algorithm::is_any_of;
+using boost::algorithm::split;
+using boost::algorithm::token_compress_on;
+using strings::CUnescape;
+using std::pair;
+
+namespace impala {
+
+Status CGroupUtil::FindGlobalCGroup(const string& subsystem, string* path) {
+ ifstream proc_cgroups("/proc/self/cgroup", ios::in);
+ string line;
+ while (true) {
+if (proc_cgroups.fail() || proc_cgroups.bad()) {
+ return Status(Substitute("Error reading /proc/self/cgroup: $0",
GetStrErrMsg()));
+} else if (proc_cgroups.eof()) {
+ return Status(
+ Substitute("Could not find subsystem $0 in /proc/self/cgroup",
subsystem));
+}
+// The line format looks like this:
+// 4:memory:/user.slice
+// 9:cpu,cpuacct:/user.slice
+getline(proc_cgro
[impala] 01/04: IMPALA-3323: Fix unrecognizable shell option when --config_file is specified
This is an automated email from the ASF dual-hosted git repository.
tarmstrong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 561158b30656b97ccb6ec83052b62c476e0873ed
Author: Fredy Wijaya
AuthorDate: Fri Jan 18 12:57:37 2019 -0800
IMPALA-3323: Fix unrecognizable shell option when --config_file is specified
Impala shell defines a dictionary of default values for some shell
options. Before this patch, the logic for --config_file checks if
a shell option exists by using the default value dictionary, which
does not contain the exhaustive list of shell options. This causes
a valid option in the Impala shell config file to be treated as
unrecognizable shell option due to the option not having a default
value. The patch fixes the issue by changing the logic that checks
for the existence of an option using the option list from optparse.
The patch also fixes the missing dest parameter for ldap_password_cmd
option.
Testing:
- Updated test_shell_commandline::test_config_file
- Ran all shell tests
Change-Id: Iff371d038fa77ba659e9b7c7a4ed5b374237f2ea
Reviewed-on: http://gerrit.cloudera.org:8080/12245
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
shell/impala_shell.py | 3 ++-
shell/option_parser.py| 30 +++---
tests/shell/good_impalarc | 1 +
tests/shell/test_shell_commandline.py | 1 +
4 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/shell/impala_shell.py b/shell/impala_shell.py
index 7471c2a..5fb2e6c 100755
--- a/shell/impala_shell.py
+++ b/shell/impala_shell.py
@@ -1629,7 +1629,8 @@ if __name__ == "__main__":
# default shell options loaded in from impala_shell_config_defaults.py
# options defaults overwritten by those in config file
try:
-loaded_shell_options, query_options = get_config_from_file(config_to_load)
+loaded_shell_options, query_options = get_config_from_file(config_to_load,
+
parser.option_list)
impala_shell_defaults.update(loaded_shell_options)
except Exception, e:
print_to_stderr(e)
diff --git a/shell/option_parser.py b/shell/option_parser.py
index ccae53b..04d69d4 100755
--- a/shell/option_parser.py
+++ b/shell/option_parser.py
@@ -32,14 +32,17 @@ import sys
from impala_shell_config_defaults import impala_shell_defaults
from optparse import OptionParser
+
class ConfigFileFormatError(Exception):
"""Raised when the config file cannot be read by ConfigParser."""
pass
+
class InvalidOptionValueError(Exception):
"""Raised when an option contains an invalid value."""
pass
+
def parse_bool_option(value):
"""Returns True for '1' and 'True', and False for '0' and 'False'.
Throws ValueError for other values.
@@ -49,21 +52,24 @@ def parse_bool_option(value):
elif value.lower() in ["false", "0"]:
return False
else:
-raise InvalidOptionValueError("Unexpected value in configuration file. '"
+ value \
+raise InvalidOptionValueError("Unexpected value in configuration file. '"
+ value
+ "' is not a valid value for a boolean option.")
-def parse_shell_options(options, defaults):
+
+def parse_shell_options(options, defaults, option_list):
"""Filters unknown options and converts some values from string to their
corresponding
- python types (booleans and None).
+ python types (booleans and None). 'option_list' contains the list of
valid options,
+ and 'defaults' is used to deduce the type of some options (only bool at
the moment).
Returns a dictionary with option names as keys and option values as
values.
"""
result = {}
+ option_dests = set(opt.dest for opt in option_list)
for option, value in options:
-if option not in defaults:
- print >> sys.stderr, "WARNING: Unable to read configuration file
correctly. " \
-"Ignoring unrecognized config option: '%s'\n" % option
-elif isinstance(defaults[option], bool):
+if option not in option_dests:
+print >> sys.stderr, "WARNING: Unable to read configuration file
correctly. " \
+ "Ignoring unrecognized config option: '%s'\n" % option
+elif isinstance(defaults.get(option), bool):
result[option] = parse_bool_option(value)
elif value.lower() == "none":
result[option] = None
@@ -71,7 +77,8 @@ def parse_shell_options(options, defaults):
result[option] = value
return result
-def get_config_from_file(config_filename):
+
+def get_config_from_file(config_filename, option_list):
"""Reads contents of configuration file
Two config sections are supported:
@@ -90,7 +97,6 @@ def get_config_from_file(config_filename):
Returns a pair of dictionaries (shell_options, query_options), with option
names
as keys and option values as values.
"""
-
config = Conf
