[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user interma closed the pull request at: https://github.com/apache/incubator-hawq/pull/1212 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1212#discussion_r111329344
--- Diff: src/test/feature/Ranger/rangerpolicy.py ---
@@ -96,12 +100,22 @@ def delete_policy(delete_policy_name, rangerhelper):
delete_policy_name = options.deletedpolicyname
#init rangerresthelper
-helper = RangerRestHelper(host, port, rangeruser, rangerpasswd);
+helper = RangerRestHelper(host, port, rangeruser, rangerpasswd)
if new_policy_json_file_name != "":
-policyname = create_policy(new_policy_json_file_name, helper)
-print "policy {} created".format(policyname)
+policyname, is_success = create_policy(new_policy_json_file_name,
helper)
+if is_success:
+print "policy {} created".format(policyname)
+else:
+print "policy {} create failed".format(policyname)
+sys.exit(-1)
--- End diff --
If add policy failed in ranger, the py script will return -1 instead of
always return 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1212#discussion_r110828640 --- Diff: src/test/feature/sanity_tests.txt --- @@ -2,5 +2,5 @@ #SERIAL=* are the serial tests to run, optional but should not be empty #you can have several PARALLEL or SRRIAL -PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected -SERIAL=TestHawqRanger.*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* +PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected:TestHawqRanger.FallbackTest:TestHawqRanger.PXF* +SERIAL=TestHawqRanger.BasicTest:TestHawqRanger.Allow*:TestHawqRanger.Resource*:TestHawqRanger.Deny*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* --- End diff -- I'm not sure how the syntax is parsed in the script, but maybe you could check. If it is really complex to do maybe you could file a JIRA. I think this will be an annoying issue given there will be more test cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user interma commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1212#discussion_r110825436 --- Diff: src/test/feature/sanity_tests.txt --- @@ -2,5 +2,5 @@ #SERIAL=* are the serial tests to run, optional but should not be empty #you can have several PARALLEL or SRRIAL -PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected -SERIAL=TestHawqRanger.*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* +PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected:TestHawqRanger.FallbackTest:TestHawqRanger.PXF* +SERIAL=TestHawqRanger.BasicTest:TestHawqRanger.Allow*:TestHawqRanger.Resource*:TestHawqRanger.Deny*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* --- End diff -- Thanks paul. I have tested, the schedule file does not support *+=* now. We can support the human-readable syntax (also diff easily) in future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1212#discussion_r110563213
--- Diff: src/test/feature/Ranger/test_policyhelper.cpp ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "policy_helper.h"
+#include "gtest/gtest.h"
+#include "lib/sql_util.h"
+
+using namespace std;
+
+class TestRangerPolicyHelper : public ::testing::Test {
+public:
+ TestRangerPolicyHelper()
+ {
+ _rangerHost = RANGER_HOST;
+ }
+ string _rangerHost;
+};
+
+TEST_F(TestRangerPolicyHelper, BasicTest) {
+ hawq::test::SQLUtility util;
+ PolicyHelper helper(util.getTestRootPath(), _rangerHost);
+ int ret = 0;
+ string suffix = "PolicyHelper_";
+ string user = suffix+"TestUser"; //Note: must create this user on
Ranger Admin first
+ string database = suffix+"Testdb";
+ string table = suffix+"TestTable";
+ string schema = "public";
+ string protocol= "protocol";
+ std::vector accesses;
+
+ ret = helper.AddSchemaPolicy(suffix+"Policy1", user, database, schema,
{"usage-schema", "create-schema"});
--- End diff --
We can seed a policy like this now, seems more convenience than write a
json file (especially on mutation).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1212#discussion_r110351041 --- Diff: src/test/feature/sanity_tests.txt --- @@ -2,5 +2,5 @@ #SERIAL=* are the serial tests to run, optional but should not be empty #you can have several PARALLEL or SRRIAL -PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected -SERIAL=TestHawqRanger.*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* +PARALLEL=TestErrorTable.*:TestPreparedStatement.*:TestUDF.*:TestAOSnappy.*:TestAlterOwner.*:TestAlterTable.*:TestCreateTable.*:TestGuc.*:TestType.*:TestDatabase.*:TestParquet.*:TestPartition.*:TestSubplan.*:TestAggregate.*:TestCreateTypeComposite.*:TestGpDistRandom.*:TestInformationSchema.*:TestQueryInsert.*:TestQueryNestedCaseNull.*:TestQueryPolymorphism.*:TestQueryPortal.*:TestQueryPrepare.*:TestQuerySequence.*:TestCommonLib.*:TestToast.*:TestTransaction.*:TestCommand.*:TestCopy.*:TestHawqRegister.TestPartitionTableMultilevel:TestHawqRegister.TestUsage1ExpectSuccessDifferentSchema:TestHawqRegister.TestUsage1ExpectSuccess:TestHawqRegister.TestUsage1SingleHawqFile:TestHawqRegister.TestUsage1SingleHiveFile:TestHawqRegister.TestDataTypes:TestHawqRegister.TestUsage1EofSuccess:TestHawqRegister.TestUsage2Case1Expected:TestHawqRegister.TestUsage2Case2Expected:TestHawqRanger.FallbackTest:TestHawqRanger.PXF* +SERIAL=TestHawqRanger.BasicTest:TestHawqRanger.Allow*:TestHawqRanger.Resource*:TestHawqRanger.Deny*:TestExternalOid.TestExternalOidAll:TestExternalTable.TestExternalTableAll:TestTemp.BasicTest:TestRowTypes.* --- End diff -- I'm wondering whether we could use some more human-readable syntax, e.g. SERIAL=TestHawqRanger.a:TestHawqRanger.b SERIAL+=TestFeatureB.c --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1212: HAWQ-1396. Add more test cases for queryi...
GitHub user interma opened a pull request: https://github.com/apache/incubator-hawq/pull/1212 HAWQ-1396. Add more test cases for querying external table via PXF wi⦠â¦th Ranger enable. include: * add HDFS/Hive/HBase test cases. * add a policy_helper class for add/delete policy conveniently, don't need write policy json file manually now. * some ranger tests can be run parallelly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/interma/interma-hawq hawq-1396 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1212.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1212 commit ea52b008847d0bfa9653eb3f5d62c266c738437d Author: interma Date: 2017-04-05T05:47:40Z HAWQ-1396. Add more test cases for querying external table via PXF with Ranger enable. include: * add HDFS/Hive/HBase test cases. * add a policy_helper class for add/delete policy conveniently, donn't need write policy json file manually now. * some ranger tests can be run parallelly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
